From patchwork Wed Feb 22 14:36:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13149243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D017EC61DA4 for ; Wed, 22 Feb 2023 14:39:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231852AbjBVOjV (ORCPT ); Wed, 22 Feb 2023 09:39:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231351AbjBVOjU (ORCPT ); Wed, 22 Feb 2023 09:39:20 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E05837544; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id 6so7551409wrb.11; Wed, 22 Feb 2023 06:39:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PvupsfzCcOKXDRykNFlbD3M522JsoKtEmFaAw+ESIZ4=; b=kjEvxcRC92xwc7L6WYECbkxta2Ix9i8Ccqeo2j6WclYyqyqGOnOmAGeq5fFozV7/GA zjFFZf8Mn+2FHNLk7wvq8agSoXbjw0XLLp/fHEqmgLCYXbY/KworMugVrnXT+y1x8zGn +k5FVGgjdT83dqVY9yFf1e2soQ5uMgWwFNFUS9lfejLuIQh7ok2/BwZjI1oesGZlpyKs U4hDz3rOTWp9NzsAb6xaRYXIlpQBajp0MaCYo1yXMXHQw592KfYkYD7q2j9tU6IVGKV3 C8XBZvVXSs9xSwhrACo0XNVBbASbSld+Xoaldlp7TAOYSqV8daMnKO78611VDsw0q2ga /big== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PvupsfzCcOKXDRykNFlbD3M522JsoKtEmFaAw+ESIZ4=; b=6bFoaIiVBI8Qrwy5jxJtqB76UwiUSP5VqgYxtt+cqPwU1Q5QO8XYykkh89t/XKraCx pbti8QygJWjMKwYZb+h3XrmLbihzP7PsffNd8EmKTNml8JNHv35GfTd+bVDr4ynYE/1a qacvab0dBLuFMt4KymN3HLt7BbgageS2SkWQ8OU8wCTw2ebssyVS55ms/z07qXUXx/oB BeOybzlhVqRrfP3tLTL/7gkOWFyntWbvm6Yg6axq9BFRCpIz4J9IyCN6C3Hs4pOBJI7/ 16X4iDVvQGZMAgpVaji1z9HwjKjN5qNEWeHFbEWBWZ/QB/0X4Aco4qh0VeEJGTty5bXn dPQQ== X-Gm-Message-State: AO0yUKXX8NKjXdClANq3TApbydFdamEg6pRcYnJ4w4rhVKpy3mv9HuOn 2iMgXnmfUl7fdYKf5T6p6dICwHGW9H8= X-Google-Smtp-Source: AK7set99govZ0ojdmAK14T61Q3W/ygyD/Bw9jtguWJZIasm4lcoV1YJ28LmxxupjOe9cn6ofITqyjw== X-Received: by 2002:adf:dcc5:0:b0:2c7:ce2:6479 with SMTP id x5-20020adfdcc5000000b002c70ce26479mr1780754wrm.40.1677076757339; Wed, 22 Feb 2023 06:39:17 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:17 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 1/4] io_uring/rsrc: disallow multi-source reg buffers Date: Wed, 22 Feb 2023 14:36:48 +0000 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If two or more mappings go back to back to each other they can be passed into io_uring to be registered as a single registered buffer. That would even work if mappings came from different sources, e.g. it's possible to mix in this way anon pages and pages from shmem or hugetlb. That is not a problem but it'd rather be less prone if we forbid such mixing. Cc: Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index a59fc02de598..8d7eb1548a04 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1162,14 +1162,17 @@ struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages) pret = pin_user_pages(ubuf, nr_pages, FOLL_WRITE | FOLL_LONGTERM, pages, vmas); if (pret == nr_pages) { + struct file *file = vmas[0]->vm_file; + /* don't support file backed memory */ for (i = 0; i < nr_pages; i++) { - struct vm_area_struct *vma = vmas[i]; - - if (vma_is_shmem(vma)) + if (vmas[i]->vm_file != file) { + ret = -EINVAL; + break; + } + if (!file) continue; - if (vma->vm_file && - !is_file_hugepages(vma->vm_file)) { + if (!vma_is_shmem(vmas[i]) && !is_file_hugepages(file)) { ret = -EOPNOTSUPP; break; } From patchwork Wed Feb 22 14:36:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13149244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2BBBC64EC4 for ; Wed, 22 Feb 2023 14:39:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231831AbjBVOjW (ORCPT ); Wed, 22 Feb 2023 09:39:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231658AbjBVOjU (ORCPT ); Wed, 22 Feb 2023 09:39:20 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A060F37B65; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id j19-20020a05600c1c1300b003e9b564fae9so664333wms.2; Wed, 22 Feb 2023 06:39:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aLlCzWw1AssfjgBXmFlk8NMc2CPaZuoh5Bi4Ozm/gs0=; b=l/ttigjxsKa3YvoIQjINVBOoNPeK2/2BaOf/7PSu4DdWgzHhBKsJ+2ouhN4H2YXXqX LF475KPpC6+x1VOtBOjoUhvEsX5sOpizb4nSzJKBP1DK4Rq/ZQSaQ8Gio2IkjZD3svXP kzdzDQlDePQKDlDzrsyY0EwMiZKmconuJ464dYSKlomNQmuGfwCtLgM2y1jhP4CAvejq cGuk5kKw10CVAXzevFhF9wMm6ykyxMuXacliu6YH4Emh9K7PNmbntQNkMwGchde7Ymyf vmJMvN0FUNqzAqBe2rbeS7mt04v+uXAXBcYtBDVk2kcTmj2V0walKtKGtiZopb98lX1f GBHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aLlCzWw1AssfjgBXmFlk8NMc2CPaZuoh5Bi4Ozm/gs0=; b=65piNaVGrBTes1XJDzoRz7hPvIeW9a6t7cXbcFs4q4cH0u+ibfoLyDVy9srDkRn4fI gJRDfCICKTyUOWbGlymLKK94Va7oGkt9ohGWS324Uie7bhblU+HR+DwMwB1G5vtsok7N zhCk6HiUyzidHsLBhQAYwkta6En65dIqOG1dL8vJsqC7yD2B6iWmK+jagiJL8cSdyoSl TbBIh6xmuEEyerbfmbh/AZFiVIVcCRtPDtG/g1JAf2dCJX8G2EYTXZqHNgRZYLSC+SsZ UBF9OSu7GqFKPsxmj2Uluq0Wfl6+uddlysZh2QFjhQDpl+v8AUSpE7Jj5pg286cED+v+ ntZg== X-Gm-Message-State: AO0yUKVDTOxtQcAEdcFyjZvmn9y59Mb8bJ52aMYXmJdKc/GN7xQgINbB n832jbIa7HOgqpafpqnufIUok88cLd0= X-Google-Smtp-Source: AK7set/klMhFR2GKixgOumpNP1VKA96VGfmzhrDYK+ETWl3gEHalFbrv6ebC9dthswmQBpa7aIx8mA== X-Received: by 2002:a05:600c:3d8c:b0:3e0:c5e:ad78 with SMTP id bi12-20020a05600c3d8c00b003e00c5ead78mr6031015wmb.7.1677076758017; Wed, 22 Feb 2023 06:39:18 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:17 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 2/4] io_uring/rsrc: fix a comment in io_import_fixed() Date: Wed, 22 Feb 2023 14:36:49 +0000 Message-Id: <1445461129902f506c46645d8659f78bd5b35008.1677041932.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_import_fixed() supports offsets, but "may not" means the opposite. Replace it with "might not" so the comments rather speaks about possible cases. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 8d7eb1548a04..53845e496881 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1338,7 +1338,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter, return -EFAULT; /* - * May not be a start of buffer, set size appropriately + * Might not be a start of buffer, set size appropriately * and advance us to the beginning. */ offset = buf_addr - imu->ubuf; From patchwork Wed Feb 22 14:36:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13149245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A312CC677F1 for ; Wed, 22 Feb 2023 14:39:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231939AbjBVOjX (ORCPT ); Wed, 22 Feb 2023 09:39:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231862AbjBVOjV (ORCPT ); Wed, 22 Feb 2023 09:39:21 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93CE838030; Wed, 22 Feb 2023 06:39:20 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id t15so8067841wrz.7; Wed, 22 Feb 2023 06:39:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M22oHJnuucBcT6iKB9lxW+hMSBsBNFHqZcHQOuzIIKc=; b=ZykuQEKlBFq6ujC0ObdSGHbCSm/snEq4aV9KxuboP9EFXzEeY1U79ozyKgjFHHthmf 0hFMSfi6beUjiDTigg8SpQXuJDjkABt+JdL4wJEEh9eQIHVkcc7S47IJNmsM7P+Pg4n4 QDnxFwrpNzwPIKIEdFmw2BPWED5431FYF9tnIKllCgamReeOon8C+8hqImeyB5Of9dGK dM2Fu67Wb6NRVJGP3dLaWscF+0tEdqS0eebfY2uHxLgEdUahj9V19t+7P759MWEYrAqs ApjBt7YA37iJGalEGm2OI25QtjLMrC/rDcSSANiRzXnR4ku0lCFODiqWfW+u+udurx8+ GjAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M22oHJnuucBcT6iKB9lxW+hMSBsBNFHqZcHQOuzIIKc=; b=x56e1uctdnxWxLNmlu4ChzRbT/OugZLTyiXrVvPHGK8gF8Y1SDUPGi0/qY7ZYIEyoQ IPaLwIoCHDGqlsvbF4I/Xe+Gpg5jfeJ+Qz3LPXLAgX0b13KThzWhkD2w5jf3UUqJCaPl zj6GsXa7jcYUiQ+E8K4bs0Og1wVMJYj6EkYIKJJp2H4JSZVfuDUBCMMC/WgB0Im+M28r Q1fGn6sv5UVM4oTARBV6NavGKrwFCQvL/x3Xt+sCG+RwKfbIkW/gpJWNw+2XVISXGYxM i4mqwYOOf/YXVeTyxqzB0lrizKJLckB6yfc/vgpT8epetSAPNUkQ0ia5Xd4+QBei8ppP RBaw== X-Gm-Message-State: AO0yUKUiJTOlVz/4T4C5c6keldw5IkhWuXavLzC9yy3IWJN8Y5JSqC8c OD+yLskNd4GFmoU+QDtJ3F4Ip9bQkSI= X-Google-Smtp-Source: AK7set+k5mgTDKfGeD1kAvandRiZgXbSjsP+KNmZB3LcQrQ94qDCi9l/6JGlG12oiZ0mFXG/9F8sxQ== X-Received: by 2002:a5d:45cb:0:b0:2c5:4f04:c50d with SMTP id b11-20020a5d45cb000000b002c54f04c50dmr7554750wrs.48.1677076758850; Wed, 22 Feb 2023 06:39:18 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:18 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 3/4] io_uring/rsrc: optimise single entry advance Date: Wed, 22 Feb 2023 14:36:50 +0000 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Iterating within the first bvec entry should be essentially free, but we use iov_iter_advance() for that, which shows up in benchmark profiles taking up to 0.5% of CPU. Replace it with a hand coded version. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 53845e496881..ebbd2cea7582 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1364,7 +1364,10 @@ int io_import_fixed(int ddir, struct iov_iter *iter, const struct bio_vec *bvec = imu->bvec; if (offset <= bvec->bv_len) { - iov_iter_advance(iter, offset); + iter->bvec = bvec; + iter->nr_segs = bvec->bv_len; + iter->count -= offset; + iter->iov_offset = offset; } else { unsigned long seg_skip; From patchwork Wed Feb 22 14:36:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13149246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F743C61DA4 for ; Wed, 22 Feb 2023 14:39:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231916AbjBVOjY (ORCPT ); Wed, 22 Feb 2023 09:39:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231600AbjBVOjW (ORCPT ); Wed, 22 Feb 2023 09:39:22 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C8183803E; Wed, 22 Feb 2023 06:39:21 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id r7so7797051wrz.6; Wed, 22 Feb 2023 06:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Yqisb0p/1TgnFfChFg4hQVIMzPMIgaeBy+4V1ZQduJ4=; b=Fg0LA5G3eErF0j0+tGhZ9F2ZwjIou9MWYHnsZZbtNv/lNRl+rkgFrA2Tc9infoD9e5 +15HKDFakYlv9lZiQCsj1qmClc8lWeS/HkiT5ZaWJnpgBN/HqJyC2j3WAi5d5EA4oVrc sAMI1q/GZVfTdR5YXHfHqkz7V1q5gfwU4mxUKz/mUa3GQvQlm9Pwjs2FMXDtvE2Pp4p+ L0IXcCLFNCFa2sAEx1Mr5ig9Q+JXxDYbj03y2FfRDfKUKLN/vjvnRWFeBfbQOT9ePSKt o80GCrstl32fZhtCKotgPXlldos+JaKLyEHqAlTiUfmLXIh3QNhNzcS6bi8Tyxop7CtO 9now== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Yqisb0p/1TgnFfChFg4hQVIMzPMIgaeBy+4V1ZQduJ4=; b=htMdd9gq2chFCkHcTkv9trMK/jX8D+wOaazY1Pn2VSz6CLUiqZ+5XZ7YEIw4Ke/O3J hn2aK8uw4XNt0W/2rT/0SyNLRjhjsjulXQeSHgKDmokdnT8/OghMdEjhctqzwHc/2Bq9 NEEMxVv0Tm9HLb7JclJjtK7ajCzOpGfkJBbbmnvN6zJdgAdjRli43JRwAe0UEur5YaQz PLyPlzwT2qKCZtlTp9ACx7C9gr3SQMbUWQOCzsn5XeAZAZsmjLE0YGZolxe5clhmgQai ApXKsaUF0tJ3DO1l4dvAR2ShckizUy3Kx0l0UbDbfUGuDwNt4YoUzsF2PXwbjOeQU/ON 6xMQ== X-Gm-Message-State: AO0yUKXwlkaU/JPomul/h6MnlFGR8RNe+UFMYEHLdPP4kBLnh8cCXJCY Hd+byopJi6AlAlMowFFflYoa7UE1CQo= X-Google-Smtp-Source: AK7set/y6zFHJXaafEe6jphQJOgrhJdwRhp4l4WKN2gjsCRmWRGW28GWxZPNkzLOY3euHphGIKQYkQ== X-Received: by 2002:a5d:6190:0:b0:2c7:a0b:e8d2 with SMTP id j16-20020a5d6190000000b002c70a0be8d2mr2569920wru.19.1677076759581; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:19 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 4/4] io_uring/rsrc: optimise registered huge pages Date: Wed, 22 Feb 2023 14:36:51 +0000 Message-Id: <757a0c399774f23df5dbfe2e0cc79f7ea432b04c.1677041932.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org When registering huge pages, internally io_uring will split them into many PAGE_SIZE bvec entries. That's bad for performance as drivers need to eventually dma-map the data and will do it individually for each bvec entry. Coalesce huge pages into one large bvec. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index ebbd2cea7582..aab1bc6883e9 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1210,6 +1210,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; + struct folio *folio; *pimu = ctx->dummy_ubuf; if (!iov->iov_base) @@ -1224,6 +1225,21 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } + /* If it's a huge page, try to coalesce them into a single bvec entry */ + if (nr_pages > 1) { + folio = page_folio(pages[0]); + for (i = 1; i < nr_pages; i++) { + if (page_folio(pages[i]) != folio) { + folio = NULL; + break; + } + } + if (folio) { + folio_put_refs(folio, nr_pages - 1); + nr_pages = 1; + } + } + imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) goto done; @@ -1236,6 +1252,17 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, off = (unsigned long) iov->iov_base & ~PAGE_MASK; size = iov->iov_len; + /* store original address for later verification */ + imu->ubuf = (unsigned long) iov->iov_base; + imu->ubuf_end = imu->ubuf + iov->iov_len; + imu->nr_bvecs = nr_pages; + *pimu = imu; + ret = 0; + + if (folio) { + bvec_set_page(&imu->bvec[0], pages[0], size, off); + goto done; + } for (i = 0; i < nr_pages; i++) { size_t vec_len; @@ -1244,12 +1271,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, off = 0; size -= vec_len; } - /* store original address for later verification */ - imu->ubuf = (unsigned long) iov->iov_base; - imu->ubuf_end = imu->ubuf + iov->iov_len; - imu->nr_bvecs = nr_pages; - *pimu = imu; - ret = 0; done: if (ret) kvfree(imu); @@ -1364,6 +1385,11 @@ int io_import_fixed(int ddir, struct iov_iter *iter, const struct bio_vec *bvec = imu->bvec; if (offset <= bvec->bv_len) { + /* + * Note, huge pages buffers consists of one large + * bvec entry and should always go this way. The other + * branch doesn't expect non PAGE_SIZE'd chunks. + */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset;