From patchwork Wed Sep 11 20:29:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13801063 Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B84113B582 for ; Wed, 11 Sep 2024 20:30:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086629; cv=none; b=HRTry+gPF7/PD5g4aFzOvzefL+NsYjVfvTkJ7WYWOh/CuekkSD4kPHiSd2G/pzG5k14csZrVbcTthl6zuHH+Rl9eivvdfL7sKkjmVclVg6/+k7MgwTi4zE3aud8PrTkkxs/55Hip2ug9c+3oViSQbln7onLA08QS5+pEMe6V/Jg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086629; c=relaxed/simple; bh=KeJ+1XAHXqnaZmYxhDYpSKKmPD++L8hs4mdmozoofeY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O3nK4yMTX3qcRYvrlWA+HNG7c9G1zsfma9lZoDtlXst2sP8TdJzd9GF0KVtUz7cswyMJxB4rj6e8vbv6WKYjMVCk4RXwfjZdoy144Ipe+znFFEkcxecLNyImNcw3yfB1eL9KztL+QDmHNzCUThbn9R7b6ziTF1zjKEGd8qwxjIw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=cNYeSQPq; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="cNYeSQPq" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-3a0220c2c6bso1054835ab.0 for ; Wed, 11 Sep 2024 13:30:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1726086625; x=1726691425; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gnzcgKOPDhAIhk/fMldP5pkSJeyy04T5uncSe/6MQ6c=; b=cNYeSQPqp+g9iKswYphbKPgxfLzS8CUzdgDdxvRPuvFbIpoqxTaMhBOKGe4lYGt5v3 NPFuHCadD1VtmNO+qZtkdN71FutpX32+quZWAVTRzY9CGowWrMTnR0yIx7LrshqGDvxF NRDWW+DaBRsLR18LKFpmLInhFwSlnbiLKH/T0vCTeDQQk4l+CgbZvK51jzU7+3taXany A2qfrBiVt5JqS/8hiFk5t102EYaxFoNJOxWG5wgmPu1aZBk93mGKxSf6i5mYj1VGNZFs H0Sid97EFDnbd9QEWCK9WWBI63peWK4/hNZAtkNm2ICxzyM5o4WoC8cDSAq3BwsSfqYj cDaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726086625; x=1726691425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gnzcgKOPDhAIhk/fMldP5pkSJeyy04T5uncSe/6MQ6c=; b=joWZSCyaFD7yPuUzznJqQLT7GA2WPc4CvnBQg4YI2EPZBwUffBwzZpx4AeOWgouSKY uY5SBkUszMpWg0D38I6Zj6TMcwQONol+5NVxRKPI1cC0zkwvCrLPRWVVN2pMVF8VbZUF rSbHglHkRn0iwCLHYUEFAu/99MuXyd9zwWAsh4LlfPN5lveD3riGxqY87IEmL+BtiXhX wFzfXiajFEgmRxRysr5U6/8b2fH0aj0aqWmAfh3pkC5Sz4wtPsZIPU4noNS3zPwNWiN8 8KNCCxlpAdLVIySbT+gbEICCWISabx9Wf8Q7hTGNkvJb3rZNZXwKfR80iCAsjfnXRHIZ Wkqw== X-Gm-Message-State: AOJu0YxXU3Wb91kr6i/sAxKdfjtLfJs5OXtLJqOKOPYpb7+SxHXqGWIs fDehZcGI+KxpQxa/czlHdT++7uCjYF8qmMvgkQhyYbHsBjPZRc+sP/QsjQ5vWeKmfOFhphsB7PM 1Xw8= X-Google-Smtp-Source: AGHT+IH53hq6SM9ev+g3+wFmH2hB/HQRyHZKO28RqcPCUIhQhHHjrBwMzCV5G+3Mtn9J6tCYcvJ+lg== X-Received: by 2002:a05:6e02:2192:b0:3a0:4ba6:e45b with SMTP id e9e14a558f8ab-3a0848ac72emr5874265ab.1.1726086625015; Wed, 11 Sep 2024 13:30:25 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4d35f433d60sm185173173.26.2024.09.11.13.30.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 13:30:24 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] io_uring/rsrc: clear 'slot' entry upfront Date: Wed, 11 Sep 2024 14:29:39 -0600 Message-ID: <20240911203021.416244-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240911203021.416244-1-axboe@kernel.dk> References: <20240911203021.416244-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, but clearing the slot pointer earlier will be required by a later change. Signed-off-by: Jens Axboe --- io_uring/rsrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7d639a996f28..d42114845fac 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -114,6 +114,7 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo struct io_mapped_ubuf *imu = *slot; unsigned int i; + *slot = NULL; if (imu != &dummy_ubuf) { for (i = 0; i < imu->nr_bvecs; i++) unpin_user_page(imu->bvec[i].bv_page); @@ -121,7 +122,6 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo io_unaccount_mem(ctx, imu->acct_pages); kvfree(imu); } - *slot = NULL; } static void io_rsrc_put_work(struct io_rsrc_node *node) From patchwork Wed Sep 11 20:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13801064 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A7DB14386D for ; Wed, 11 Sep 2024 20:30:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086630; cv=none; b=kEzv9HT9NDgz/XIvc4Bv+9SlIMZnTVdgPnfiUznDRsfTCZjMpfG696F3H8Hj0YumqMW/OqPKlixxwney9H4mUoBE+LG8yAcpv0lYr1o2L7t/TxqxtXz725ePOmmxDAb+Sa8fzbgXL/ttoUPrJ57JIaZZk0xzTJkCi6qZn3nt/V4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086630; c=relaxed/simple; bh=GM5rez7i8AcYityEiDjC4PZI1ccMmlLlITNVJtUHxTM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QgigX1msdEsW5fTIA/rrJN2iG+EdJbphVRiUf0MMkXE3S2dFAItE3WzGV+M/AkCqZDx4CHft8/I02E7HILRPMYsD7CThIlJgpGl4S/G23Og6mYczNNMiSkO9wzT7P8oKhyW46Pu4ibS0Zts0TLylyLn+cxs1tAKgovnmQppuzn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=PgW6H4/q; arc=none smtp.client-ip=209.85.166.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="PgW6H4/q" Received: by mail-io1-f52.google.com with SMTP id ca18e2360f4ac-829f7911eecso9070339f.1 for ; Wed, 11 Sep 2024 13:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1726086626; x=1726691426; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pbZg1ed04SkE9PDBN3N3iel9ESiPexbtZx/Ls3JLSeo=; b=PgW6H4/qfR+fa9xRZpxB10Iqm1WYklhlFnH1/QJsvq8qFc1eefkO5dxv4Cn4Q5I9UB BOogPXJRwg3rR2CEAPcIVQdiTNj8ZOyz6PePDkHF5RAeF5DqPXuoG7cmczdn4vdsJDU6 4/wKzg0HLntMBossAa8pkmDP9xpK72ROkgsdmnDnvwZ/zqpqg7/GuLMnN0hNPCSUn8VU AmH2PvE5a0lXdTyX/mWy4bbTla5ncaFUFHvPrIqaceQfNreNiRmmnvBsKiusPXob5X01 50L3IKPyss14T11WR+K4oTxDnkQxKfmvx1+am6miNpoQiRR6NirNd9WsfVdccEBXCSI9 LAOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726086626; x=1726691426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pbZg1ed04SkE9PDBN3N3iel9ESiPexbtZx/Ls3JLSeo=; b=tIdaoK2sFGviMIcFpJx8I/wBmChIeZi3i2HiUOP4JAnb6jriDGdXz8yWMfm25+CDwJ ZR+q1r3MIk6IGjUn/8IaI5W+wNv8qcfbdolX6Ot+LfqsOweSnPo94GrdEkEW2InoURh6 pndFTnacrCFGK9kS8GgS4Ck/ogPA1nPQvejW/yyigcDnVy1e1k5xFxO8gy7Ww6KhnK4f 9lt1AxSL0w84+8g7J0ZIX+4qug/9ngVhTvvx3Ci1Yh2skkiY0PQXo4layHuw9RH5DT16 r/aGOxKSfq2bUV/VxNm2Ml0CEIb6Cop5kR9bvHl4mbqSkITNtoLq/UoFA/Pv55KAaFUb xwHQ== X-Gm-Message-State: AOJu0YyhIvuZsw0TzWMutaoMZ6rJYxG9HDF/FJWXi717kGNnS2fS7OhF Gfg0HL/FSi6eft7IUWqDO047mnAePpyC5b5xeVgE6S/24vQMVu4/HnLtppNEYMOS0ljX9SWUIA+ BLRI= X-Google-Smtp-Source: AGHT+IE8QSi7rRuxJXP5MFAxMTfbnQXLmVdlh+6QaifvAXACmGm3dL5U8hEkzzPEc/nh4bKy5p5qtw== X-Received: by 2002:a05:6e02:1a88:b0:3a0:80c6:367 with SMTP id e9e14a558f8ab-3a084931e1emr4426075ab.19.1726086626610; Wed, 11 Sep 2024 13:30:26 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4d35f433d60sm185173173.26.2024.09.11.13.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 13:30:25 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] io_uring/rsrc: add reference count to struct io_mapped_ubuf Date: Wed, 11 Sep 2024 14:29:40 -0600 Message-ID: <20240911203021.416244-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240911203021.416244-1-axboe@kernel.dk> References: <20240911203021.416244-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently there's a single ring owner of a mapped buffer, and hence the reference count will always be 1 when it's torn down and freed. However, in preparation for being able to link io_mapped_ubuf to different spots, add a reference count to manage the lifetime of it. Signed-off-by: Jens Axboe --- io_uring/rsrc.c | 3 +++ io_uring/rsrc.h | 1 + 2 files changed, 4 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d42114845fac..28f98de3c304 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -116,6 +116,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; if (imu != &dummy_ubuf) { + if (!refcount_dec_and_test(&imu->refs)) + return; for (i = 0; i < imu->nr_bvecs; i++) unpin_user_page(imu->bvec[i].bv_page); if (imu->acct_pages) @@ -990,6 +992,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->folio_shift = data.folio_shift; imu->folio_mask = ~((1UL << data.folio_shift) - 1); } + refcount_set(&imu->refs, 1); off = (unsigned long) iov->iov_base & ~imu->folio_mask; *pimu = imu; ret = 0; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 3d0dda3556e6..98a253172c27 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -47,6 +47,7 @@ struct io_mapped_ubuf { unsigned int folio_shift; unsigned long acct_pages; unsigned long folio_mask; + refcount_t refs; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Wed Sep 11 20:29:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13801065 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8414D7DA9C for ; Wed, 11 Sep 2024 20:30:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086631; cv=none; b=hL/Z8fcQ3v7H57S/AKNwCsh7jd25EmFmKipIDEGSpZKRjD5zzLvdkilyrAbNHPO4EOKCf5i/lfe3g1JrctoXCcQ/PeUDlfuilNcN7bH7m3IMEdYzLywD3JnK6Vq+Q+ec2kM89HXdxBV42gQ+1PuLoWFmRS1hChEpTHzaj5cx1Ss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726086631; c=relaxed/simple; bh=J0gjB2X7/k+nBqFg6DA6HRhFcTFOnlZDK8jzIkKCBsg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TwpSGyQQl2tkP0iGHqH4BCUnBbuDhelzwlIT8NFNMXrBHKxYz5SsAold9o6dxYWlnPjSQTnBqF9lUBGX9f/WmxqCCe9E7zwq9aPK2wrPQ9OZAHuNaqUbNrp1PRMEip8zcvUurTx2/sJWJNzr2Jf7q69EPmn8+Pxsmea7blgXbEI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZueaWWoW; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZueaWWoW" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-82aa3f65864so8790739f.2 for ; Wed, 11 Sep 2024 13:30:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1726086628; x=1726691428; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SBbgqLjwU1cfqThy8NlSSBjOLmfIDQJ7BLoWNPWdLms=; b=ZueaWWoW54yK/zzBxYGbMXFWnF1Jm+1VszL/fA2RNe20nWb0+3a7a/m4WFLJ8TYJlG sNTZph+wQfr8znwCEjGH/LkfU9XxN0HGZxR00zdQUEvrCEs8X11CZ+gp6jMobW1UzuwO t65Lnnd59W314tsBxHXdbvHZD1Rkv1vMjuQyEJErNUGjCr34Y4paiMcOhJWDctG52qGS jw8zL6GR8MLMib9PNxlhevfbCSUMXLixy6oj+8MaA+RdYdlzfCPM8gFbt8RNDs0ZCI51 NFxshOSDHUeYfPv7i56a54WFo5fw3vjiiDS8MeTMcX2NquFT/uA0Hl6IIFQPcM2yi9Qx 5hCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726086628; x=1726691428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SBbgqLjwU1cfqThy8NlSSBjOLmfIDQJ7BLoWNPWdLms=; b=PtOwwQ4NUmvadZOad6YaCsdRbqUNwKFUMiPKg+zoU2hyC5Cuu39hRr42Ox+4+HNgwa GdBbecW2wGVS0OtQe8iNCSN2jfzGuk9k1Y46Xpt5LwNjkAURZGfEIi2MOTk6fzqviSe9 s9REj7OR7LSzkFviOm8G/GHVRRnkC6Wy247/0hQs/whcKAZSg0OTBJJBZeyrhyEi3S5L smcgEVYk2yK2Go4AsYm62DT1d77IPFq4BUNMtASM9AEpIFZm4pwYZx9Visyd0MFBBYeK lRgFH0GcbCrPCmbzMVTEDTl/hil4V2wLzvMV9Ub/CumfZeeDzJqZD6muEgz2nCee+QPI w1ag== X-Gm-Message-State: AOJu0YyMKm28MCcdpEDjK5ZqTLvRBWXAq5Qy8DpjEqEI921tx7SshjOJ TuUvaP9eJeNS0RT6mFASIobRji5oF06bUE70V71n/mecdT76Jm+nPwBkcissRrUHaKSIjU1vCjg Gb+g= X-Google-Smtp-Source: AGHT+IGULWhy/fc7PHkx/GD50PpGWAIgN9lusGKymfd92uT46AgW7C5fSTm3RaePDONUQZKopGUa4w== X-Received: by 2002:a05:6602:1344:b0:825:bdf:aecf with SMTP id ca18e2360f4ac-82d1f93f6femr96095739f.11.1726086628198; Wed, 11 Sep 2024 13:30:28 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4d35f433d60sm185173173.26.2024.09.11.13.30.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 13:30:27 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] io_uring: add IORING_REGISTER_COPY_BUFFERS method Date: Wed, 11 Sep 2024 14:29:41 -0600 Message-ID: <20240911203021.416244-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240911203021.416244-1-axboe@kernel.dk> References: <20240911203021.416244-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Buffers can get registered with io_uring, which allows to skip the repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This reduces the overhead of O_DIRECT IO. However, registrering buffers can take some time. Normally this isn't an issue as it's done at initialization time (and hence less critical), but for cases where rings can be created and destroyed as part of an IO thread pool, registering the same buffers for multiple rings become a more time sensitive proposition. As an example, let's say an application has an IO memory pool of 500G. Initial registration takes: Got 500 huge pages (each 1024MB) Registered 500 pages in 409 msec or about 0.4 seconds. If we go higher to 900 1GB huge pages being registered: Registered 900 pages in 738 msec which is, as expected, a fully linear scaling. Rather than have each ring pin/map/register the same buffer pool, provide an io_uring_register(2) opcode to simply duplicate the buffers that are registered with another ring. Adding the same 900GB of registered buffers to the target ring can then be accomplished in: Copied 900 pages in 17 usec While timing differs a bit, this provides around a 25,000-40,000x speedup for this use case. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 8 ++++ io_uring/register.c | 6 +++ io_uring/rsrc.c | 90 +++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 1 + 4 files changed, 105 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index a275f91d2ac0..7b15216a3d7f 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -609,6 +609,9 @@ enum io_uring_register_op { IORING_REGISTER_CLOCK = 29, + /* copy registered buffers from source ring to current ring */ + IORING_REGISTER_COPY_BUFFERS = 30, + /* this goes last */ IORING_REGISTER_LAST, @@ -694,6 +697,11 @@ struct io_uring_clock_register { __u32 __resv[3]; }; +struct io_uring_copy_buffers { + __u32 src_fd; + __u32 pad[7]; +}; + struct io_uring_buf { __u64 addr; __u32 len; diff --git a/io_uring/register.c b/io_uring/register.c index 57cb85c42526..c8670de33343 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -542,6 +542,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_clock(ctx, arg); break; + case IORING_REGISTER_COPY_BUFFERS: + ret = -EINVAL; + if (!arg || nr_args != 1) + break; + ret = io_register_copy_buffers(ctx, arg); + break; default: ret = -EINVAL; break; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 28f98de3c304..e94b6ab6e749 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1137,3 +1137,93 @@ int io_import_fixed(int ddir, struct iov_iter *iter, return 0; } + +static int io_copy_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx) +{ + struct io_mapped_ubuf **user_bufs; + struct io_rsrc_data *data; + int i, ret, nbufs; + + /* + * Drop our own lock here. We'll setup the data we need and reference + * the source buffers, then re-grab, check, and assign at the end. + */ + mutex_unlock(&ctx->uring_lock); + + mutex_lock(&src_ctx->uring_lock); + ret = -ENXIO; + nbufs = src_ctx->nr_user_bufs; + if (!nbufs) + goto out_unlock; + ret = io_rsrc_data_alloc(ctx, IORING_RSRC_BUFFER, NULL, nbufs, &data); + if (ret) + goto out_unlock; + + ret = -ENOMEM; + user_bufs = kcalloc(nbufs, sizeof(*ctx->user_bufs), GFP_KERNEL); + if (!user_bufs) + goto out_free_data; + + for (i = 0; i < nbufs; i++) { + struct io_mapped_ubuf *src = src_ctx->user_bufs[i]; + + refcount_inc(&src->refs); + user_bufs[i] = src; + } + + /* Have a ref on the bufs now, drop src lock and re-grab our own lock */ + mutex_unlock(&src_ctx->uring_lock); + mutex_lock(&ctx->uring_lock); + if (!ctx->user_bufs) { + ctx->user_bufs = user_bufs; + ctx->buf_data = data; + ctx->nr_user_bufs = nbufs; + return 0; + } + + /* someone raced setting up buffers, dump ours */ + for (i = 0; i < nbufs; i++) + io_buffer_unmap(ctx, &user_bufs[i]); + io_rsrc_data_free(data); + kfree(user_bufs); + return -EBUSY; +out_free_data: + io_rsrc_data_free(data); +out_unlock: + mutex_unlock(&src_ctx->uring_lock); + mutex_lock(&ctx->uring_lock); + return ret; +} + +/* + * Copy the registered buffers from the source ring whose file descriptor + * is given in the src_fd to the current ring. This is identical to registering + * the buffers with ctx, except faster as mappings already exist. + * + * Since the memory is already accounted once, don't account it again. + */ +int io_register_copy_buffers(struct io_ring_ctx *ctx, void __user *arg) +{ + struct io_uring_copy_buffers buf; + struct fd f; + int ret; + + if (ctx->user_bufs || ctx->nr_user_bufs) + return -EBUSY; + if (copy_from_user(&buf, arg, sizeof(buf))) + return -EFAULT; + if (memchr_inv(buf.pad, 0, sizeof(buf.pad))) + return -EINVAL; + + f = fdget(buf.src_fd); + if (!f.file) + return -EBADF; + if (!io_is_uring_fops(f.file)) { + fdput(f); + return -EBADF; + } + + ret = io_copy_buffers(ctx, f.file->private_data); + fdput(f); + return ret; +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 98a253172c27..93546ab337a6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -68,6 +68,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, u64 buf_addr, size_t len); +int io_register_copy_buffers(struct io_ring_ctx *ctx, void __user *arg); void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,