From patchwork Mon Nov 26 16:45:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69EF415A8 for ; Mon, 26 Nov 2018 16:45:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B90E2A083 for ; Mon, 26 Nov 2018 16:45:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 502C72A088; Mon, 26 Nov 2018 16:45:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5875D2A087 for ; Mon, 26 Nov 2018 16:45:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726282AbeK0Dka (ORCPT ); Mon, 26 Nov 2018 22:40:30 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:44736 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726542AbeK0Dk3 (ORCPT ); Mon, 26 Nov 2018 22:40:29 -0500 Received: by mail-io1-f68.google.com with SMTP id r200so14451904iod.11 for ; Mon, 26 Nov 2018 08:45:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uXUPTEErriSDBDqj30UJ7EFKAesBtDa0/H0fvvZRr1Y=; b=PdAr8Mh5pSv9yS/FtEHQciRH9nioWconVE1xeU06K388kUaDINPQDZM2bC8MYoUZPk cADijkXboEKoP97mhm7rmUdgBqCg9zwHXgo+u7FRhuaFNKbBikrw/Sa+0x8BFVFIXNtl PFvp3eWlhkNr9XSSSrqml519jqy6drw8TcX7bzM9/WeUXyHda3ojj6iAsAhWbTs+096M OyC0bgaM3mPUqpWBfRbJqd3/TrkPy00s4CmJUlSAmtuW1OKjqujlYoAOeLmdn613MoAA EH/jWPcC0mAnXBEW52lr0f5GN383ezv3T4R+pnO5z8qU8oKV/hM3m12RsOugKM5heJig JEOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uXUPTEErriSDBDqj30UJ7EFKAesBtDa0/H0fvvZRr1Y=; b=dix1GGAL9/JU9UnUDq1aiOuqFXimDbUs0gI9Vtj3XgVTlo+E/s+oXDQ7iW3YqIkSv9 hu15mEk7s4/pL08DNwJMIFXmy+qfpokzzxq1G/9SvEuTElay/KwwvQJ3PbFbfd5vUhCu sgC1rmxGT7YWJKYl7ePZYnYBZbaSe2Dy1EJnR9Fph2YLVSCEmXpJn8o7h3F9M6GR6FcF xJVigpxkNp1O8GOHVjIFJDnZV2L3j5v6JT0lb+tXbGYZTRSIWCixc2IIVmwqrSrgKZ8h zKVEZqXWP05e57464ainCwg/fi0Ui88Fx1lYBJt0DCpOLTKj9R50A4LbwtJCnrq6Brpp +LUQ== X-Gm-Message-State: AA+aEWbAYKE3NcyvqUOCPnSJbi914zAMb5hFJSuJfeleTeVUc3ytaUb9 OVwkflewv/Vm1KUoN7aFKU1DpHIpalg= X-Google-Smtp-Source: AFSGD/WjSR+dAatWBussqO1RZbe8Dy24Pf+/goQkghIvtZu04nyJojsMaeUw6MUT73GFlyFXV67WVg== X-Received: by 2002:a5d:8b8d:: with SMTP id p13mr14268158iol.223.1543250749785; Mon, 26 Nov 2018 08:45:49 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:48 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/20] aio: fix failure to put the file pointer Date: Mon, 26 Nov 2018 09:45:25 -0700 Message-Id: <20181126164544.5699-2-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If the ioprio capability check fails, we return without putting the file pointer. Fixes: d9a08a9e616b ("fs: Add aio iopriority support") Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/aio.c b/fs/aio.c index b984918be4b7..205390c0c1bb 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1436,6 +1436,7 @@ static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) ret = ioprio_check_cap(iocb->aio_reqprio); if (ret) { pr_debug("aio ioprio check cap error: %d\n", ret); + fput(req->ki_filp); return ret; } From patchwork Mon Nov 26 16:45:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698653 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87F4F15A8 for ; Mon, 26 Nov 2018 16:45:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 734042A087 for ; Mon, 26 Nov 2018 16:45:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 678172A08C; Mon, 26 Nov 2018 16:45:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18EE22A087 for ; Mon, 26 Nov 2018 16:45:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726441AbeK0Dkb (ORCPT ); Mon, 26 Nov 2018 22:40:31 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:45314 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726542AbeK0Dkb (ORCPT ); Mon, 26 Nov 2018 22:40:31 -0500 Received: by mail-io1-f67.google.com with SMTP id w7so14459865iom.12 for ; Mon, 26 Nov 2018 08:45:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NbaVNkMyxvwaepZlvQFQJrgWkKKIKk5aWR0maYoyXOs=; b=z3FyYJ5q8+IqGRiv1IFmR7COSEdIAkh0a/45cbNar4425z8lglyvqTL8jLTTDCsx9k 76ygE/wUOaHNFHSPBnzoK8onwPvjwCg9PCcHpF21PQAHTK+yFbVVhgScw8hi2Iex2AD8 RVGBQc6s6UY+bmBCmukFhQLMP/B/7DLet7STVsXttUQbTyQS2PLbEmfN9CrfyPK0OHRT MF/FLPh/KFUOL8dyX/WezJ7mVEgDIXJlkFX9Ed0r+JlqSblTVMcoDAJKKjwiVNDsdjc/ cCo0sCncRXUSCjfsQiUqC3HLIUTWX07oGTWm2lv7/Gbz2/zjXC8vsq1P8fgdsL5YCtgL oXGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NbaVNkMyxvwaepZlvQFQJrgWkKKIKk5aWR0maYoyXOs=; b=Jm/ECHAVbpSxKlJHCEWigruYTyLYIoYGnCVsa7URLnQEkFGCijoGkoxUN0IOVBvrJx NbiGAS4sQPZ6mlUnu+G+6Peb8CjtIb1KH4kEEzjzzVqKVJSDNXFnaDt7Woq9/uC4ON25 KcM9X7HpqN1zYiupIhIKVGZkq8Qh0Phd775KJpQd7bsUbLHZOllYPUHOITwe12X0kOMz T5orj056HI9N4XJzbTvBWgC6YmbzLINIYfgqJC3v56g2Nhsb4LvfjCkMYC8YhnsvkM7M 3HXdqH2SFtN8wtQQZEZvWYbM9s2qFelTbCLtV6o5wc3ZeK8XqVjrTROLXbCo6NGh5hQl bvZQ== X-Gm-Message-State: AA+aEWZFqw/Kik07y/Rm6XuhaBZYElMWKFBQf125BWqiUnU3auXIqlbA HbsTMLGtDvRX7maEevW8qBU4QUAQ+XY= X-Google-Smtp-Source: AFSGD/XebPi3O7CTyBzvm/N/uzGpQkCtOKAaWJ/jPn78/jX5dFwaVoc9uC+SzYsWFKOKnnba/yvzpg== X-Received: by 2002:a6b:5d01:: with SMTP id r1mr20644103iob.170.1543250751799; Mon, 26 Nov 2018 08:45:51 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:50 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Christoph Hellwig , Jens Axboe Subject: [PATCH 02/20] aio: clear IOCB_HIPRI Date: Mon, 26 Nov 2018 09:45:26 -0700 Message-Id: <20181126164544.5699-3-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christoph Hellwig No one is going to poll for aio (yet), so we must clear the HIPRI flag, as we would otherwise send it down the poll queues, where no one will be polling for completions. Signed-off-by: Christoph Hellwig IOCB_HIPRI, not RWF_HIPRI. Signed-off-by: Jens Axboe Reviewed-by: Johannes Thumshirn --- fs/aio.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 205390c0c1bb..05647d352bf3 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1436,8 +1436,7 @@ static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) ret = ioprio_check_cap(iocb->aio_reqprio); if (ret) { pr_debug("aio ioprio check cap error: %d\n", ret); - fput(req->ki_filp); - return ret; + goto out_fput; } req->ki_ioprio = iocb->aio_reqprio; @@ -1446,7 +1445,13 @@ static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) ret = kiocb_set_rw_flags(req, iocb->aio_rw_flags); if (unlikely(ret)) - fput(req->ki_filp); + goto out_fput; + + req->ki_flags &= ~IOCB_HIPRI; /* no one is going to poll for this I/O */ + return 0; + +out_fput: + fput(req->ki_filp); return ret; } From patchwork Mon Nov 26 16:45:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CDF6217D5 for ; Mon, 26 Nov 2018 16:45:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BFDF12A087 for ; Mon, 26 Nov 2018 16:45:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B41752A08C; Mon, 26 Nov 2018 16:45:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D2C32A087 for ; Mon, 26 Nov 2018 16:45:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726542AbeK0Dke (ORCPT ); Mon, 26 Nov 2018 22:40:34 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:34672 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727028AbeK0Dkd (ORCPT ); Mon, 26 Nov 2018 22:40:33 -0500 Received: by mail-it1-f193.google.com with SMTP id x124so24405315itd.1 for ; Mon, 26 Nov 2018 08:45:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2auvEeZY6eQcIorX3a41A16F0lYlc9wyO+6UPHFanyg=; b=w/SNo+dMYxjWJHZ1zwoKIzE2g4TUUNz4RIMN31QgV0av0LR6idhPATbySxGRI/GCdF UNLMDXYVwfE6CgFjORlV7kmVtSBaWChfo2yD5cfsoyqCFgDX7iuuVSz2B+yx4WkhtxSX n04FQszoOxo9m1Wqh8TFuFauIu3gkykriE2QkqC7BaMQvd7RZg8v/8ZEZQjXXjNeJjVj 3C8OdrZePp96fzLBtbbMKZti9ybCWUm9jpXFLmAm8PtD4hB2BB4L6U6wSvxoXk5SXer5 +3ncmVBsi9eWx4ZQFBbxZP2pg0ZyNMbDDo7HjQSBew2yjPJROGDLRX2fcXFaoGpYrn2b ouBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2auvEeZY6eQcIorX3a41A16F0lYlc9wyO+6UPHFanyg=; b=jSEdb5hcOc2JdmatEHcrK99nVQaO5DR55o/GkwtBpDtiN/d24kj5wyj4iWzlCGMIFR Vpebs3RRnL+QLdIyyeFCJYbVEP+7MJ9WEWx9hirSfGst5WQlMaOfTg6NFJIwuYmJacNF /n1MooOO9Wtkq5slJpisXAZ4mRW/Zo4AcGhIi8R9gvp8AOLeiSJlogL+dedhxlEaeF9s bUSmnwDm5eiNnhD3bPNPY0KRfb6QlEwiuh6wqMeEQ9vKa+F/+Kfo01GUFq1SbgNuLjxn 63fe1yO+sGYCO2z8Ed99VK73Wlpqlwgxg5VG68CYs99vlsl6LzAt0un8MUAlPbO18DYI y7GA== X-Gm-Message-State: AA+aEWYmIQ3hKnvxjvRj5cb3tQZb595qnzmmhtfkUnqQ5jXFS3QBxNK5 I7AwBe1MeYPzUfodoIAphI+EWwD85g8= X-Google-Smtp-Source: AFSGD/U8icyOsqlt7qbnnKpgZLt1jeRy1Nmfvp72LILDfeBOCPEH8yjdod0uOLxbL5XwZLep/qTtqg== X-Received: by 2002:a24:c40b:: with SMTP id v11mr1249763itf.73.1543250754148; Mon, 26 Nov 2018 08:45:54 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:53 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Christoph Hellwig , Jens Axboe Subject: [PATCH 03/20] fs: add an iopoll method to struct file_operations Date: Mon, 26 Nov 2018 09:45:27 -0700 Message-Id: <20181126164544.5699-4-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christoph Hellwig This new methods is used to explicitly poll for I/O completion for an iocb. It must be called for any iocb submitted asynchronously (that is with a non-null ki_complete) which has the IOCB_HIPRI flag set. The method is assisted by a new ki_cookie field in struct iocb to store the polling cookie. TODO: we can probably union ki_cookie with the existing hint and I/O priority fields to avoid struct kiocb growth. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe Reviewed-by: Johannes Thumshirn --- Documentation/filesystems/vfs.txt | 3 +++ include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+) diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt index 5f71a252e2e0..d9dc5e4d82b9 100644 --- a/Documentation/filesystems/vfs.txt +++ b/Documentation/filesystems/vfs.txt @@ -857,6 +857,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *); @@ -902,6 +903,8 @@ otherwise noted. write_iter: possibly asynchronous write with iov_iter as source + iopoll: called when aio wants to poll for completions on HIPRI iocbs + iterate: called when the VFS needs to read the directory contents iterate_shared: called when the VFS needs to read the directory contents diff --git a/include/linux/fs.h b/include/linux/fs.h index a1ab233e6469..6a5f71f8ae06 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -310,6 +310,7 @@ struct kiocb { int ki_flags; u16 ki_hint; u16 ki_ioprio; /* See linux/ioprio.h */ + unsigned int ki_cookie; /* for ->iopoll */ } __randomize_layout; static inline bool is_sync_kiocb(struct kiocb *kiocb) @@ -1781,6 +1782,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *); From patchwork Mon Nov 26 16:45:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698661 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8ED3515A8 for ; Mon, 26 Nov 2018 16:45:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 816AA2A087 for ; Mon, 26 Nov 2018 16:45:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7572C2A08C; Mon, 26 Nov 2018 16:45:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DBC52A087 for ; Mon, 26 Nov 2018 16:45:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726719AbeK0Dkf (ORCPT ); Mon, 26 Nov 2018 22:40:35 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:52734 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726425AbeK0Dkf (ORCPT ); Mon, 26 Nov 2018 22:40:35 -0500 Received: by mail-it1-f195.google.com with SMTP id i7so29328339iti.2 for ; Mon, 26 Nov 2018 08:45:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jl3/dynblu/agc3BZMjXxwYt9clu/5wSo7rh5aN5YNA=; b=TvGFxnMstmEfJMIOLMXNiOzTVykZyMUQqT16CwADuhfVYscHH+JyxL0s7oyCTiJotD 8CTHzg3TPD5pKHlCHWxIo9zfZbiIJo+dG/ykf+imY+tT6byWhUwQRvo3Hf5qTKlsjgEM EGQmGYhuu1YSZj3OpKbA9K8/s1PGulibE8lHZxDZ9vniaoJh4g/id+oEed9XvMdy2GxJ mlSn+NRCCQLKcixy9SldCeaKVXQUhTG0crow7X70mFDeeB6VEA8QLQKsdqyTa4SARsp+ JW9S/vIB8vBR3GswWalaOvWRAi3/oi9XHdyugCdHSrF6xXZM/HpHOB2lDvCxinCkKTZR a25A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jl3/dynblu/agc3BZMjXxwYt9clu/5wSo7rh5aN5YNA=; b=MpsX40hQve/iNhFXqLYUOszm9OizOy6i2RzDb6icLnWSmWz6xlYLImV3U45wTm357L dk9pzBNyvlrP8Kn0tB5q/UJ66sS7Jxfu/d9wgQd/jO45c1mggfqSi9vOtNVdKEtHYH0/ zyX0WMaiOqx+lF6LZP4YrQEmH6POAHhiMnJeqoGv0ZVFIPMcC55S8U44dNXrDQxlqvI4 PPGbA/XrMLGNhHBa9U64k9qJQyT8ukvaTXpV0cUEA7h6NRiP8DvdB5D/0aumDNycAGOH C2kXluH+GcZhKzLcVECTqxZh6Pnxx6n6fBn/ua37OYngMIWVm5RDJLetV44hi8V6nN44 8wKA== X-Gm-Message-State: AA+aEWbPHfZJn2qcw+rcUTshkiYCHWREU5wLcRSyME88EIq1BeGAA5jn 71kgFQ6oygbb+jGV08mt3YR1Kh97ZsQ= X-Google-Smtp-Source: AFSGD/VAAgn1G80svAGS4/lcIM732CDMZ2l6nLLhCGz38+VmSG+zuSk90tu7NS5mds4SlFQCveLtqw== X-Received: by 2002:a24:130a:: with SMTP id 10mr12425542itz.58.1543250755800; Mon, 26 Nov 2018 08:45:55 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:54 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Christoph Hellwig , Jens Axboe Subject: [PATCH 04/20] block: wire up block device iopoll method Date: Mon, 26 Nov 2018 09:45:28 -0700 Message-Id: <20181126164544.5699-5-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christoph Hellwig Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe Reviewed-by: Johannes Thumshirn --- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fs/block_dev.c b/fs/block_dev.c index d233a59ea364..9acb26f7c9ec 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -281,6 +281,14 @@ struct blkdev_dio { static struct bio_set blkdev_dio_pool; +static int blkdev_iopoll(struct kiocb *kiocb, bool wait) +{ + struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host); + struct request_queue *q = bdev_get_queue(bdev); + + return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait); +} + static void blkdev_bio_end_io(struct bio *bio) { struct blkdev_dio *dio = bio->bi_private; @@ -396,6 +404,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) bio->bi_opf |= REQ_HIPRI; qc = submit_bio(bio); + WRITE_ONCE(iocb->ki_cookie, qc); break; } @@ -2061,6 +2070,7 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read_iter = blkdev_read_iter, .write_iter = blkdev_write_iter, + .iopoll = blkdev_iopoll, .mmap = generic_file_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl, From patchwork Mon Nov 26 16:45:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698663 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71E7F15A8 for ; Mon, 26 Nov 2018 16:46:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63DE82A083 for ; Mon, 26 Nov 2018 16:46:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 582192A08C; Mon, 26 Nov 2018 16:46:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 152472A083 for ; Mon, 26 Nov 2018 16:46:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727043AbeK0Dkh (ORCPT ); Mon, 26 Nov 2018 22:40:37 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:54531 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727028AbeK0Dkh (ORCPT ); Mon, 26 Nov 2018 22:40:37 -0500 Received: by mail-it1-f194.google.com with SMTP id m123-v6so5068995ita.4 for ; Mon, 26 Nov 2018 08:45:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8/ImYfz/riZa32isBNBlGI10dC+68boqA2fqt5yslkE=; b=lP+rqi+JEJxGvbLvurZwrNc7lzsn7lfTmgbo22Ffx3Hm/lCKBuj28CPk18v+YZiCG7 SNlPcunHTj6D6mbcUi+nkUe0Gvm8ulewvC2UEI/yMx9dAQVZG/7iE9WiI9yfRyNELVu1 hR8kZbAvjSsSsEy9uSAIT4+NrnkACftnKKfcBedtWypDMkPFUVhR/Fmb4FF/jBJyShu5 WxK31gXgu5x6Z2Dq6Rz+HQGVGtnjR3QeltZeQSsQJ34IdPXPaOb1IioBFdd9GuqGN0/p oYCyJ+Vx2cuc/9vzeKZD0j9LOxTazHmc0sAc/h/jM8JcY5q7Ip52EUtKCY7V6+uExgIl 0ndQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8/ImYfz/riZa32isBNBlGI10dC+68boqA2fqt5yslkE=; b=khVippFL4fTMm5DeACxyd2NXYjUcrLVLrKTf1eLxsKVJyovpJFTEm12iqtlqseO6qj kVqtXg0HXEHAEm8d6Miexo34QfMqE3B6V1zoOpAfXhi6hapt44rb3N9c82NvElgiRqp0 aSNj8dUNZWc1lR2hJe3Ej34jf2+MqE57IfU0YWnEl/ukqXryjCl/B1+C0PbY9imb+3C4 LFQeJd8buI0V6kUKdR5VNUEo7u0TInY2LUSnxanLmbkZd0FwheCsgLd9MTyxBRZhk21B cv4GGZmASKcV87WrTPluVNuTBtmoQaEDv/8ECFBkKjvFZ6cQ8zoFMLZ9wy48BboRtZ2S IJVA== X-Gm-Message-State: AGRZ1gIYG3pwAr64Kc1r8WQkcCn/+plJNaueV4BoiZt0CbCCWlS9inso bEWV6JKl1U3iN72UrhRSjUXypuy9pTE= X-Google-Smtp-Source: AFSGD/VMqOH8mkK44M6+hC82Uh2kkuHxB7MLTm3cr6UrpfPjnAIVkILTJ4iAsOkxVfDX0TS1Ww37VA== X-Received: by 2002:a05:660c:fd2:: with SMTP id m18mr24372303itn.1.1543250757685; Mon, 26 Nov 2018 08:45:57 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:56 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/20] block: ensure that async polled IO is marked REQ_NOWAIT Date: Mon, 26 Nov 2018 09:45:29 -0700 Message-Id: <20181126164544.5699-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can't wait for polled events to complete, as they may require active polling from whoever submitted it. If that is the same task that is submitting new IO, we could deadlock waiting for IO to complete that this task is supposed to be completing itself. Signed-off-by: Jens Axboe --- fs/block_dev.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 9acb26f7c9ec..9580ffe0cf24 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -400,8 +400,16 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES); if (!nr_pages) { - if (iocb->ki_flags & IOCB_HIPRI) + if (iocb->ki_flags & IOCB_HIPRI) { bio->bi_opf |= REQ_HIPRI; + /* + * For async polled IO, we can't wait for + * requests to complete, as they may also be + * polled and require active reaping. + */ + if (!is_sync) + bio->bi_opf |= REQ_NOWAIT; + } qc = submit_bio(bio); WRITE_ONCE(iocb->ki_cookie, qc); From patchwork Mon Nov 26 16:45:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698669 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 072FC17D5 for ; Mon, 26 Nov 2018 16:46:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED1B12A087 for ; Mon, 26 Nov 2018 16:46:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E19342A08C; Mon, 26 Nov 2018 16:46:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77F202A088 for ; Mon, 26 Nov 2018 16:46:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727062AbeK0Dkk (ORCPT ); Mon, 26 Nov 2018 22:40:40 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:39061 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726425AbeK0Dkk (ORCPT ); Mon, 26 Nov 2018 22:40:40 -0500 Received: by mail-it1-f195.google.com with SMTP id m15so28746374itl.4 for ; Mon, 26 Nov 2018 08:46:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=p6XwCf9lidTSvsA0jLObhvA2jqsQMgsk5MoVc4+K+Yo=; b=xqeDTr9ENdt3CXTkMQZFU6k9mZWgQJ7H8w8CwmrE37KRxFXUUANz0deVdrdpfIDQz8 ttapRJx1mIaReODymvsLFP3sdTcRHmjxuMuD+vjuiOV3oXmCiFpMDkAR1qWmUbBEdl60 TgnlqUwPVcrQcFxB3IA4qHMd/KMc3v4U8RVIils0Ja5LkpHZcGtumaywtEJLdkzXBbuM JI3zKT9AxCv+pWyqX+JZb55HZgQ+p5IwY1JwyfsZAvTzoC7rUB19zsHIwltAOOGiTYZU VsnGOVtf6IehyxhULL9d/fv122c1RN0TPx2je8O4PvHXhoBjJumFLGZpX8E+fcQ8nRAN WfBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=p6XwCf9lidTSvsA0jLObhvA2jqsQMgsk5MoVc4+K+Yo=; b=fUjGv4D4QIqhSTSsXuaI7MdT1HPRfAhpfsrlfvpM1U52CcxJVj+0Io+sQTVh9MULq9 lvaFDZ3tivDuMiXKnzp7UPc27eU5kOfgBbqxcvJ58bOc+CUqHDF1Nb2FJ25I26L/Rkvu vIdzIGFkmQ1AikNrPFDpYIMMYCCcB9xDr7slfRKGeVYHz1K8Wkry3sl56FuRd7v0e/gI bP070Ds78SpaaI79ibDlTQ0pFngDpoaL7F2IyPR5VaLFiBaFtdofeRFQIsWOhNesxHkP nL0APtj9JoVkZGMhgS6nXnQfaq9gh6BfOzfO6D7p7ApFn8ftaeiveygkKubrWQ1jL6Wk Yxng== X-Gm-Message-State: AA+aEWZ529R3BV9bg2/DPU28BXtenM8vKI1dDxXsQ21kIEaqPLsJX2oY IKDTTu4XnyjVggztTBBuFgmTE5AuFho= X-Google-Smtp-Source: AJdET5cJvhDt+lu0mbHys5Iyl8ob6nIpxDc2nrgj0sLCgK62FIQ2oUqoSZuVE7y4T0A+5nqZ2vwzdg== X-Received: by 2002:a24:f982:: with SMTP id l124mr23640459ith.59.1543250759925; Mon, 26 Nov 2018 08:45:59 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.45.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:45:58 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Christoph Hellwig , Jens Axboe Subject: [PATCH 06/20] iomap: wire up the iopoll method Date: Mon, 26 Nov 2018 09:45:30 -0700 Message-Id: <20181126164544.5699-7-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christoph Hellwig Store the request queue the last bio was submitted to in the iocb private data in addition to the cookie so that we find the right block device. Also refactor the common direct I/O bio submission code into a nice little helper. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/gfs2/file.c | 2 ++ fs/iomap.c | 43 ++++++++++++++++++++++++++++--------------- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 32 insertions(+), 15 deletions(-) diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c index 45a17b770d97..358157efc5b7 100644 --- a/fs/gfs2/file.c +++ b/fs/gfs2/file.c @@ -1280,6 +1280,7 @@ const struct file_operations gfs2_file_fops = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open, @@ -1310,6 +1311,7 @@ const struct file_operations gfs2_file_fops_nolock = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open, diff --git a/fs/iomap.c b/fs/iomap.c index 74c1f37f0fd6..16787b3b09fd 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1436,6 +1436,28 @@ struct iomap_dio { }; }; +int iomap_dio_iopoll(struct kiocb *kiocb, bool spin) +{ + struct request_queue *q = READ_ONCE(kiocb->private); + + if (!q) + return 0; + return blk_poll(q, READ_ONCE(kiocb->ki_cookie), spin); +} +EXPORT_SYMBOL_GPL(iomap_dio_iopoll); + +static void iomap_dio_submit_bio(struct iomap_dio *dio, struct iomap *iomap, + struct bio *bio) +{ + atomic_inc(&dio->ref); + + if (dio->iocb->ki_flags & IOCB_HIPRI) + bio->bi_opf |= REQ_HIPRI; + + dio->submit.last_queue = bdev_get_queue(iomap->bdev); + dio->submit.cookie = submit_bio(bio); +} + static ssize_t iomap_dio_complete(struct iomap_dio *dio) { struct kiocb *iocb = dio->iocb; @@ -1548,7 +1570,7 @@ static void iomap_dio_bio_end_io(struct bio *bio) } } -static blk_qc_t +static void iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos, unsigned len) { @@ -1562,15 +1584,10 @@ iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - if (dio->iocb->ki_flags & IOCB_HIPRI) - flags |= REQ_HIPRI; - get_page(page); __bio_add_page(bio, page, len, 0); bio_set_op_attrs(bio, REQ_OP_WRITE, flags); - - atomic_inc(&dio->ref); - return submit_bio(bio); + iomap_dio_submit_bio(dio, iomap, bio); } static loff_t @@ -1666,9 +1683,6 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, bio_set_pages_dirty(bio); } - if (dio->iocb->ki_flags & IOCB_HIPRI) - bio->bi_opf |= REQ_HIPRI; - iov_iter_advance(dio->submit.iter, n); dio->size += n; @@ -1676,11 +1690,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, copied += n; nr_pages = iov_iter_npages(&iter, BIO_MAX_PAGES); - - atomic_inc(&dio->ref); - - dio->submit.last_queue = bdev_get_queue(iomap->bdev); - dio->submit.cookie = submit_bio(bio); + iomap_dio_submit_bio(dio, iomap, bio); } while (nr_pages); if (need_zeroout) { @@ -1883,6 +1893,9 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, if (dio->flags & IOMAP_DIO_WRITE_FUA) dio->flags &= ~IOMAP_DIO_NEED_SYNC; + WRITE_ONCE(iocb->ki_cookie, dio->submit.cookie); + WRITE_ONCE(iocb->private, dio->submit.last_queue); + if (!atomic_dec_and_test(&dio->ref)) { if (!dio->wait_for_completion) return -EIOCBQUEUED; diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 53c9ab8fb777..603e705781a4 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1203,6 +1203,7 @@ const struct file_operations xfs_file_operations = { .write_iter = xfs_file_write_iter, .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = xfs_file_ioctl, #ifdef CONFIG_COMPAT .compat_ioctl = xfs_file_compat_ioctl, diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 9a4258154b25..0fefb5455bda 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -162,6 +162,7 @@ typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret, unsigned flags); ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, const struct iomap_ops *ops, iomap_dio_end_io_t end_io); +int iomap_dio_iopoll(struct kiocb *kiocb, bool spin); #ifdef CONFIG_SWAP struct file; From patchwork Mon Nov 26 16:45:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698673 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7871D15A8 for ; Mon, 26 Nov 2018 16:46:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69F8B2A083 for ; Mon, 26 Nov 2018 16:46:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E2D22A087; Mon, 26 Nov 2018 16:46:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 094BD2A08C for ; Mon, 26 Nov 2018 16:46:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726425AbeK0Dkl (ORCPT ); Mon, 26 Nov 2018 22:40:41 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:33126 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727066AbeK0Dkl (ORCPT ); Mon, 26 Nov 2018 22:40:41 -0500 Received: by mail-it1-f194.google.com with SMTP id m8so7482672itk.0 for ; Mon, 26 Nov 2018 08:46:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZECYVF1gJ/UytKVuamY/WF200Edf1Xru8F859yvniEc=; b=08LOPMzsRDOo83TfkVDlEi/iJuRLKf2GrSp6f9MjEgiY+PZYAQnsqSR5HLrX1sBgoy 2hgXGpDBOcRia8qBI0Hd131S87KgppoJHaSUpiSa2V9uB70bUHolxWygL4EsM2KjF8Dt kGyPhvLV1H6lf5vlrFGGn9VQO4OdZRojKDJxaxMYhibOtX1HvfWIYP0Gp9UweszraMv4 wEsrWIOHG6/tQxM6WqaaRoovtInllQ9JlOWpILBJAeUZh3V5BJX2UEorxgkX17dg6L0p Ra0UoXyGUhynREvEmNu6gp+7ev7BMpaRSGidas7xTC+5cX89Bw52uUfI2PTXpHPE9RNV iG8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZECYVF1gJ/UytKVuamY/WF200Edf1Xru8F859yvniEc=; b=g8x7RCyNBlLSy/y8Nvi0dJFYGjLCFgY4e4Si6U7sZDftosdlMblxW4l34hyJ8Q5h/i TaZkAm/Rh4wjppzC8d3userDcbRe5rzylwXy7NEIhVmpM2wnzIURudHlhG7W3ThbdXlH J2wFTBvyJa1gUkV58KK4inm6rSaz56R2U9dxEbmgXpfNv8Q4UxiKBvx0b+YgcHAlSyhp Coq6X0DZouwcLkO3ZKaauo9TDowSEDQE+FMUDSlh9CtRpWi9awx4w2rXRx7ED+Wnl3SV 1k4W3L/USdvp2mcKEgoCibiPNxZpw2Mzm3yejgBZO1CBE+bGe9w+SUfk99AV5f/Y0BpM 2uJA== X-Gm-Message-State: AA+aEWZ6wv016tAwvGCEsOfl0T6tYFY4q6ClwUsoHysCKygSXv4qdRFa UVS6SV6crdDrUQGXxAMTd5j72QzstHI= X-Google-Smtp-Source: AJdET5eHoC1ioxXSqVwH0514iId5+vubvjql+khkf/Y9BTi1ZrNv6tbrhsvUyT49skUGkvyDGlbtWQ== X-Received: by 2002:a24:da06:: with SMTP id z6-v6mr23077915itg.178.1543250761799; Mon, 26 Nov 2018 08:46:01 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:00 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/20] iomap: ensure that async polled IO is marked REQ_NOWAIT Date: Mon, 26 Nov 2018 09:45:31 -0700 Message-Id: <20181126164544.5699-8-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can't wait for polled events to complete, as they may require active polling from whoever submitted it. If that is the same task that is submitting new IO, we could deadlock waiting for IO to complete that this task is supposed to be completing itself. Signed-off-by: Jens Axboe --- fs/iomap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/fs/iomap.c b/fs/iomap.c index 16787b3b09fd..96d60b9b2bea 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1451,8 +1451,16 @@ static void iomap_dio_submit_bio(struct iomap_dio *dio, struct iomap *iomap, { atomic_inc(&dio->ref); - if (dio->iocb->ki_flags & IOCB_HIPRI) + if (dio->iocb->ki_flags & IOCB_HIPRI) { bio->bi_opf |= REQ_HIPRI; + /* + * For async polled IO, we can't wait for requests to + * complete, as they may also be polled and require active + * reaping. + */ + if (!dio->wait_for_completion) + bio->bi_opf |= REQ_NOWAIT; + } dio->submit.last_queue = bdev_get_queue(iomap->bdev); dio->submit.cookie = submit_bio(bio); From patchwork Mon Nov 26 16:45:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698677 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15C1014E2 for ; Mon, 26 Nov 2018 16:46:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 052A12A083 for ; Mon, 26 Nov 2018 16:46:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EDB382A088; Mon, 26 Nov 2018 16:46:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAAB62A083 for ; Mon, 26 Nov 2018 16:46:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727066AbeK0Dkn (ORCPT ); Mon, 26 Nov 2018 22:40:43 -0500 Received: from mail-io1-f65.google.com ([209.85.166.65]:42259 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726410AbeK0Dkn (ORCPT ); Mon, 26 Nov 2018 22:40:43 -0500 Received: by mail-io1-f65.google.com with SMTP id x6so14475591ioa.9 for ; Mon, 26 Nov 2018 08:46:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=er4XNTBrDyS5t2gvWJtJjpkVFpAPOC78sIXCy9gEMJ4=; b=X7yj5lXzUhsMWy+CnAJoPhk33lrLTk7/RULRQTYaLro+GeoZwGIMOVH8bq5zHYf1sr /F6SsVasfH96v6P+hY8Y68obhZrHdB+zvqECwbKGnYIoNzfHCR57tRoV0A+zQj12EqjR 5Fx78m1ba/mNAnpUdVXEGRQgwmiDjM9cr9Qytw/XsQsa6hBpbRzDlzllq/le4IXliqC/ TV6C+Y8aqDwKRe0G/nlEHbxEMr/DXsAh3tdDinBqlp+yGd4pnnvlQorfoOh2tjvMpgb3 IrWknMWXZJYI5VgiftefMQPq7+6QfVecz6RSO2tAmpOG8GLPi4RBbVWWoYB1+kTTrqAY p8zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=er4XNTBrDyS5t2gvWJtJjpkVFpAPOC78sIXCy9gEMJ4=; b=VuBYjbQp2vstEfgDgYEnOGLsLGxZwyZy9Upqa6T1pZuA3FwxeJR8/6s9M1F9uWUr39 WugBTQuZUtOmx3zDC8n37o8XyjPzdZsLY7SQke0LWwVGkxwneWOrlEyslfQD/zSk4l4D gY8mmoJtSswlah8ZMbjoOlBQU5Ur6//GSr5LYlZ9vx9JMhFqCnec1Lvi1a1vbNWSrRxs 8su8rriMJzhiQ8H7e8PKt+S5pLbr2MFwA3NIkZunYvjj7UhmX+66dAcD7+oWrD57S/KM YS9VDM0/V0QtYiv0ICs1yDqod1lQceSY7m1anB2TBbL7HTDtJ2AVTCmEeo2bFTL7AbAf h5kA== X-Gm-Message-State: AA+aEWba0hgm3SL8YL1HW6ugr6WO9wErTfYYP5UoQHo0dPjN5hntU8vF H5KIYH63az4h2AMRJGTXvpw8HzMGctY= X-Google-Smtp-Source: AFSGD/VyjPChe8c8fQzhIk+QLnEMJOTZTCBhCuZroLZiWlymSlrug7H25N+HcWRNpd3Jx2b1lIhMIA== X-Received: by 2002:a5e:d702:: with SMTP id v2mr20014535iom.17.1543250763394; Mon, 26 Nov 2018 08:46:03 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:02 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/20] aio: use assigned completion handler Date: Mon, 26 Nov 2018 09:45:32 -0700 Message-Id: <20181126164544.5699-9-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We know this is a read/write request, but in preparation for having different kinds of those, ensure that we call the assigned handler instead of assuming it's aio_complete_rq(). Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/aio.c b/fs/aio.c index 05647d352bf3..cf0de61743e8 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1490,7 +1490,7 @@ static inline void aio_rw_done(struct kiocb *req, ssize_t ret) ret = -EINTR; /*FALLTHRU*/ default: - aio_complete_rw(req, ret, 0); + req->ki_complete(req, ret, 0); } } From patchwork Mon Nov 26 16:45:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698679 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7083917D5 for ; Mon, 26 Nov 2018 16:46:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 61F1F2A087 for ; Mon, 26 Nov 2018 16:46:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 568BE2A08C; Mon, 26 Nov 2018 16:46:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0537A2A087 for ; Mon, 26 Nov 2018 16:46:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726410AbeK0Dkp (ORCPT ); Mon, 26 Nov 2018 22:40:45 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:54570 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727082AbeK0Dkp (ORCPT ); Mon, 26 Nov 2018 22:40:45 -0500 Received: by mail-it1-f193.google.com with SMTP id m123-v6so5069799ita.4 for ; Mon, 26 Nov 2018 08:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sgBueyD41WcD6YGB1PXXB7qHQZZJctD1Yvvmpvxoomc=; b=fzMXQnJBDQ12o1SCpVJ0wTAHz6XhLyu3SvL2bffUfqUh1V7cFkmYKtx+O8Nie0UUH8 yEv39/xjpjR+UU0IqFdAE3I0TGU426cz8a/dWippznmaCnIUD8DRCyPUS8L7RiWd0PzL K9NNoPMG9glOb2KD97yBgYdQS6PHgPTYt7Fj26L/H6J3Lt4mtqtETTWcEpU2M7fnCgwi AVwi71pt1Mydwy+d6goLkZKrueJiZqOG2L8d94RcKKPAdhEx3qPAAUy2UvMMaOeFn0sn 46u1ah82QLykjCaIo4B5Hjmjf2UjVB1PE8LSePGJmbWd2UtWq4LDqYMouG46GFAGfwRK JH3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sgBueyD41WcD6YGB1PXXB7qHQZZJctD1Yvvmpvxoomc=; b=W+ClUFsCBUoLTWFp2q7+USKkN4GDRJFM1DOdkiG2lOwJ+TFOvtpNLZc2mndrijI+35 E+Nu9WglCIGTvad3f7f1KnsXDormKR+sVyCRxG2LLlF6K/qZdICqXFMlnqmJh1FAvU78 p81LI50QpasZzwOz+AeCsbwra537p1/gszuq0sKNnxnHy8gguRh8hTMA1fJiwl0lLfLw 18Wuv227HEL1/vFbDSIyefGovkB+OOD7b8Eg+FXQBAdyVzlQ7s/Uc6R5FlbGeWRU7iYh wC/NWy0oEybTyuM1ne/EbyBeVcTeEuxqqrNXx1+v6jAPc63FRF4+SaGHJTjVTEcitklL wlrA== X-Gm-Message-State: AA+aEWY91bVcURVXPH4EvZN/0++l+n2vJrkaKOP3A1wvQfg67jaqb7bM tNC3nIqx1qK8SXSDdSRz/kFyk9mc638= X-Google-Smtp-Source: AFSGD/U0UjEjF6tNASkpLnca4/HGYyUBxqCAafb4XiHHQ1nvARl6VBgbi58+SEbIaCWGKvYffdVZ5A== X-Received: by 2002:a24:89:: with SMTP id 131mr4630058ita.105.1543250765420; Mon, 26 Nov 2018 08:46:05 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:04 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/20] aio: separate out ring reservation from req allocation Date: Mon, 26 Nov 2018 09:45:33 -0700 Message-Id: <20181126164544.5699-10-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is in preparation for certain types of IO not needing a ring reserveration. Signed-off-by: Jens Axboe --- fs/aio.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index cf0de61743e8..eaceb40e6cf5 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -901,7 +901,7 @@ static void put_reqs_available(struct kioctx *ctx, unsigned nr) local_irq_restore(flags); } -static bool get_reqs_available(struct kioctx *ctx) +static bool __get_reqs_available(struct kioctx *ctx) { struct kioctx_cpu *kcpu; bool ret = false; @@ -993,6 +993,14 @@ static void user_refill_reqs_available(struct kioctx *ctx) spin_unlock_irq(&ctx->completion_lock); } +static bool get_reqs_available(struct kioctx *ctx) +{ + if (__get_reqs_available(ctx)) + return true; + user_refill_reqs_available(ctx); + return __get_reqs_available(ctx); +} + /* aio_get_req * Allocate a slot for an aio request. * Returns NULL if no requests are free. @@ -1001,24 +1009,15 @@ static inline struct aio_kiocb *aio_get_req(struct kioctx *ctx) { struct aio_kiocb *req; - if (!get_reqs_available(ctx)) { - user_refill_reqs_available(ctx); - if (!get_reqs_available(ctx)) - return NULL; - } - req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL|__GFP_ZERO); if (unlikely(!req)) - goto out_put; + return NULL; percpu_ref_get(&ctx->reqs); INIT_LIST_HEAD(&req->ki_list); refcount_set(&req->ki_refcnt, 0); req->ki_ctx = ctx; return req; -out_put: - put_reqs_available(ctx, 1); - return NULL; } static struct kioctx *lookup_ioctx(unsigned long ctx_id) @@ -1805,9 +1804,13 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, return -EINVAL; } + if (!get_reqs_available(ctx)) + return -EAGAIN; + + ret = -EAGAIN; req = aio_get_req(ctx); if (unlikely(!req)) - return -EAGAIN; + goto out_put_reqs_available; if (iocb.aio_flags & IOCB_FLAG_RESFD) { /* @@ -1870,11 +1873,12 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, goto out_put_req; return 0; out_put_req: - put_reqs_available(ctx, 1); percpu_ref_put(&ctx->reqs); if (req->ki_eventfd) eventfd_ctx_put(req->ki_eventfd); kmem_cache_free(kiocb_cachep, req); +out_put_reqs_available: + put_reqs_available(ctx, 1); return ret; } From patchwork Mon Nov 26 16:45:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698683 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7270015A8 for ; Mon, 26 Nov 2018 16:46:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63E0E2A087 for ; Mon, 26 Nov 2018 16:46:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 588C32A08C; Mon, 26 Nov 2018 16:46:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 121032A087 for ; Mon, 26 Nov 2018 16:46:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727104AbeK0Dkr (ORCPT ); Mon, 26 Nov 2018 22:40:47 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:54576 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727100AbeK0Dkq (ORCPT ); Mon, 26 Nov 2018 22:40:46 -0500 Received: by mail-it1-f195.google.com with SMTP id m123-v6so5069964ita.4 for ; Mon, 26 Nov 2018 08:46:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mh+baagw6TRyG7ZRVJ1uWljPK7oJpHidng0AtT9sC8g=; b=ZI/Nj0KWRWNNOzDBr+AA6s1u+beQA0V+xoTW/OIX0kXbANq3qRThh3xyTyZakAX35n VBjMeULPcxYgS6Gp5wq3P1DvPmyFJ8vt3WeJtsZCPAGyFkrDzJR7LA5gtl7iN+a4y7Q8 SwONi7qLWdYZmF7uPfIrrVBb4tSUQqCCcRFzVCVsxJWcPHCzHtEKL8LU059ci/Glur7G XL4Kk384BfccHtj61hgZtmPiAP2hMYaYCk2uvMCOxZEEp+IPp+dpjSerXpddY8RukvqI M9/bnez19KCGlyR7Ip2bnPhO+hpv4tUKRJfYKNCUWbpSMgrsRMfjf6a9axR6zp8bE0C/ fPBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mh+baagw6TRyG7ZRVJ1uWljPK7oJpHidng0AtT9sC8g=; b=ZfY34FIPm3wdtffQfiiP3BCLbx+m8WI8j6CJxclHPLWAvMhtweycEgpFXr10zC3k/y 22a5TUG+0TJlMsKYDNPW+8iR6HfrKEI/WoFa9YZalnFabhBmpVcWORTD8cFB/viuElqJ n/8vdXr77j9bGNQiQe5cRlfey8G0H0K50wNdba+rXAge9vrKNKuFJy/C7375d/xtWlVN 6mn48S1jhLLb4jgc65/PJ8Gfee0wGTFJKhp1DXxLpW/G9azPax7OYitNDExL7BU/mN61 PMER48XRbzbfy2EJIuBP7ogr19TFcznTHin02vZq/p5XshVSV4Tx60D3+DnmuLcTV8Ai BkSA== X-Gm-Message-State: AGRZ1gJCZ1Dwd62aLyhNarXUA+YdxMnLzkXPptXBBmWobkeUU/SCLOed 91gJUuS3631dnn6xDlGWyFcPsPDI8WM= X-Google-Smtp-Source: AJdET5eqjJTja24x3SvUDZJy4+CPO+TjcaJpqgrtOzRfwrmRMc+NKBxBHBXGEV19gsrKeQn6F4vFXw== X-Received: by 2002:a24:458c:: with SMTP id c12mr23420072itd.27.1543250767096; Mon, 26 Nov 2018 08:46:07 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:05 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/20] aio: don't zero entire aio_kiocb aio_get_req() Date: Mon, 26 Nov 2018 09:45:34 -0700 Message-Id: <20181126164544.5699-11-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It's 192 bytes, fairly substantial. Most items don't need to be cleared, especially not upfront. Clear the ones we do need to clear, and leave the other ones for setup when the iocb is prepared and submitted. Signed-off-by: Jens Axboe --- fs/aio.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index eaceb40e6cf5..681f2072f81b 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1009,14 +1009,15 @@ static inline struct aio_kiocb *aio_get_req(struct kioctx *ctx) { struct aio_kiocb *req; - req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL|__GFP_ZERO); - if (unlikely(!req)) - return NULL; + req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL); + if (req) { + percpu_ref_get(&ctx->reqs); + req->ki_ctx = ctx; + INIT_LIST_HEAD(&req->ki_list); + refcount_set(&req->ki_refcnt, 0); + req->ki_eventfd = NULL; + } - percpu_ref_get(&ctx->reqs); - INIT_LIST_HEAD(&req->ki_list); - refcount_set(&req->ki_refcnt, 0); - req->ki_ctx = ctx; return req; } @@ -1730,6 +1731,10 @@ static ssize_t aio_poll(struct aio_kiocb *aiocb, struct iocb *iocb) if (unlikely(!req->file)) return -EBADF; + req->head = NULL; + req->woken = false; + req->cancelled = false; + apt.pt._qproc = aio_poll_queue_proc; apt.pt._key = req->events; apt.iocb = aiocb; From patchwork Mon Nov 26 16:45:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698689 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DA42181D for ; Mon, 26 Nov 2018 16:46:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EED22A087 for ; Mon, 26 Nov 2018 16:46:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 03C402A08D; Mon, 26 Nov 2018 16:46:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEB552A087 for ; Mon, 26 Nov 2018 16:46:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727105AbeK0Dkt (ORCPT ); Mon, 26 Nov 2018 22:40:49 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:41208 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726484AbeK0Dkt (ORCPT ); Mon, 26 Nov 2018 22:40:49 -0500 Received: by mail-io1-f68.google.com with SMTP id s22so14461692ioc.8 for ; Mon, 26 Nov 2018 08:46:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8/1Nol3a3uaTr8sM9gFlQE0kinP2KU2cED85d/G/25k=; b=SpG2ITlTENkBMGZK/kJkn/8cxIwNnkar1M8O94LTHwrAlqMcDehZ+x4SX0yv8fyWBk CGnm8gTrSEI6GZJ7+iahYvmKqdqKnwb1byicIDFkgxY+J1i8rpGmQUL7PoEjaEShAJd9 V+8+qCFxWFRrZJCMIm+rD2YLu1cu58mNG0GiWbpU/THKsvzkp3/I0Zvyh7rFLoGqOHag abklHSYlwiWYuV272zC/4aHE8kzp0cdnnZgy7+Cchl+W5Nl5RRTpgGqJHzBlvv2ZX5BM HezJzM365+6tacTIdKEXYJCXN2iEO5aoFZ0p3lsb0KqfXBYu5h9Mo5JOst0NQJBDS/pZ gbyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8/1Nol3a3uaTr8sM9gFlQE0kinP2KU2cED85d/G/25k=; b=dyM9Py2Tc0JM/3YK5Fa1JzROPgGjOVV9/cVkSyWXZZoCEHuM642ZVPkkLQW8SKuUlc skOnkoxkdzww06WwFYQvAfGPJqcxvPeaJwrjLE32GpIcaNiLSjPIEUvwuJuMjMmThT11 5dQeDx5dJhKZsnBySDiHy2+1g9ixPVPBkLe+JUf9N+O4O/Y7S9SES3SdyXFR0JIBpesv TkwWETdYQ3TdfqlxIldPf124fjlq6Zp6sMC9y6Jg9DiqzFCRyvhekBoTvsG8oKk2Z47G NqFC2MO4E49KVVX7VUxFaqTjsm3KpK/S+r1dsYv+ABsWdRVsK66496fj6fSGKHvDQ7vL Q6sA== X-Gm-Message-State: AA+aEWYAvScWSPAgVt3ziZ367vGgPlS8mkmLb6FuN5lfbqtPO7p6rVnU LPMPEFa9ZKB1depr3t4DLGLiDQynqJc= X-Google-Smtp-Source: AFSGD/XMTYi6/A14h4AbBuZrWT5os/7E/avzDu65ty7TuN1Eo8U7jb6gqgbAg39BcxVTBUjmlcnM5A== X-Received: by 2002:a5d:87cd:: with SMTP id q13mr5186661ios.92.1543250769114; Mon, 26 Nov 2018 08:46:09 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:08 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 11/20] aio: only use blk plugs for > 2 depth submissions Date: Mon, 26 Nov 2018 09:45:35 -0700 Message-Id: <20181126164544.5699-12-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Plugging is meant to optimize submission of a string of IOs, if we don't have more than 2 being submitted, don't bother setting up a plug. Signed-off-by: Jens Axboe --- fs/aio.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 681f2072f81b..533cb7b1112f 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1887,6 +1887,12 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, return ret; } +/* + * Plugging is meant to work with larger batches of IOs. If we don't + * have more than the below, then don't bother setting up a plug. + */ +#define AIO_PLUG_THRESHOLD 2 + /* sys_io_submit: * Queue the nr iocbs pointed to by iocbpp for processing. Returns * the number of iocbs queued. May return -EINVAL if the aio_context @@ -1919,7 +1925,8 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, if (nr > ctx->nr_events) nr = ctx->nr_events; - blk_start_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) + blk_start_plug(&plug); for (i = 0; i < nr; i++) { struct iocb __user *user_iocb; @@ -1932,7 +1939,8 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, if (ret) break; } - blk_finish_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) + blk_finish_plug(&plug); percpu_ref_put(&ctx->users); return i ? i : ret; @@ -1959,7 +1967,8 @@ COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, if (nr > ctx->nr_events) nr = ctx->nr_events; - blk_start_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) + blk_start_plug(&plug); for (i = 0; i < nr; i++) { compat_uptr_t user_iocb; @@ -1972,7 +1981,8 @@ COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, if (ret) break; } - blk_finish_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) + blk_finish_plug(&plug); percpu_ref_put(&ctx->users); return i ? i : ret; From patchwork Mon Nov 26 16:45:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698693 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A8AB15A8 for ; Mon, 26 Nov 2018 16:46:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C1BD2A083 for ; Mon, 26 Nov 2018 16:46:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 30B182A087; Mon, 26 Nov 2018 16:46:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BE802A08C for ; Mon, 26 Nov 2018 16:46:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726745AbeK0Dkv (ORCPT ); Mon, 26 Nov 2018 22:40:51 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:36117 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726484AbeK0Dku (ORCPT ); Mon, 26 Nov 2018 22:40:50 -0500 Received: by mail-it1-f196.google.com with SMTP id c9so28762749itj.1 for ; Mon, 26 Nov 2018 08:46:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6hPqsRSp/t/J4Haujgv9+B43f7+1jV9cXsozT1ugKjo=; b=HPouFcMTP40WuMbWDUBWUh6rXs04bRB6nLamLVJxWKLfSTgvZzOaSrUtsIOdytKHrP K4xhORduMw++7Ygd7bphWcbJnZx69stLlKsiKLEkIYw5WPFoAN78+Wshw3Ii7WPMcFYK zw2NqZwHBNBuvOONl8N6Q6lpjivwMws3SzbWdDRXM3q4RoaYaakHRTYjLUtm7rgU0Q/9 Nyz0hXaEfqwjHUsF11AKoddp3HpMxwK+2/oClNMVROrDkr7B/YbWFugvv5jli9OQbOrL nF4lkz6Nwt2361RPtmy7L2MrnPJV7Q4nUb1ok45fDsmcj5zo0nKSK106efp1U4Gzg7cf Nspg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6hPqsRSp/t/J4Haujgv9+B43f7+1jV9cXsozT1ugKjo=; b=Gy3xxBSSGxcd4Nvx7C0Hr2XvrVqnAZOHxBrvvQGThwuOVLDvnOWtZKePU4NEYF3myf tQbwCe5IoRPv3+s4jxAMDrFTQh5Au5RrTTy1UMfc4eOr8NPfgNQYadkIe3VFnpt/8Mu1 7LSjUKCQGNNU/V2mIOAezmHknKAfj/lriTOlBAmcyeF/a+FfdoxtOAk7e66VURxKUJPH GwvCMwQGkif4ElZGyiObJ7JCP241pAx+Kjhf+90KgmpELRGKUdWudD89aH7jAHt/OrwF IEwY2ee3Z5E3CPuZK7MXy8BbFsbAtC8CHA4/cieLeRaqOKKCjeoEqykINIwxi6egc7WP vEZw== X-Gm-Message-State: AGRZ1gI+e9ZF46dpv5V0FgJME/PIo7w0xLmRC/pgM9yYUK8rdrdzyiWX XHbHKTUQ6eyWflANBmYWzIecRx5Nzlk= X-Google-Smtp-Source: AJdET5cGqA2CUJoCQGn3U3Kxw/Tksqpsg3yIg40eoEUBb/6RTwmbgfp/nQH5XW7HD1TwaFA51GEBLA== X-Received: by 2002:a24:3c43:: with SMTP id m64mr23062445ita.22.1543250770741; Mon, 26 Nov 2018 08:46:10 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:09 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 12/20] aio: use iocb_put() instead of open coding it Date: Mon, 26 Nov 2018 09:45:36 -0700 Message-Id: <20181126164544.5699-13-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Replace the percpu_ref_put() + kmem_cache_free() with a call to iocb_put() instead. Signed-off-by: Jens Axboe --- fs/aio.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 533cb7b1112f..e8457f9486e3 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1878,10 +1878,9 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, goto out_put_req; return 0; out_put_req: - percpu_ref_put(&ctx->reqs); if (req->ki_eventfd) eventfd_ctx_put(req->ki_eventfd); - kmem_cache_free(kiocb_cachep, req); + iocb_put(req); out_put_reqs_available: put_reqs_available(ctx, 1); return ret; From patchwork Mon Nov 26 16:45:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698697 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5E53017D5 for ; Mon, 26 Nov 2018 16:46:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 505DE2A083 for ; Mon, 26 Nov 2018 16:46:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 44EEA2A088; Mon, 26 Nov 2018 16:46:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B34EF2A083 for ; Mon, 26 Nov 2018 16:46:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726484AbeK0Dkx (ORCPT ); Mon, 26 Nov 2018 22:40:53 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:39107 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeK0Dkx (ORCPT ); Mon, 26 Nov 2018 22:40:53 -0500 Received: by mail-it1-f195.google.com with SMTP id m15so28747763itl.4 for ; Mon, 26 Nov 2018 08:46:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AZTxn1zH/8CPF8nHBpvAZQPHd/ePx+TIdDehIsUuIw4=; b=twSTElw0XBXHI+TBUuoeQhhzB44s8bN7TWF1ai59+6+lJRxcPmTYBV9AUMPs56RZ5l 8S8ud2EQino2u7l5kp6zxXMOx/xUsl/1AFWLjFrJrh4BrV/kMwJJIjc7mcWg0gY9igD0 9+uX8XOfAHUFWBSjnMGiH0YkfYj2Rk2euOd+tfV2VNmyarI0Al1EVq6X5v/iRJwTLxpf Qo18lWjTUPYxGyIgVQqspm5gpRP4s65nImvS5C8vdrsjBff8TBhDj1HA86rIMPE3udI9 /ixLdvwM4xGgnnsu9NYpEM+9HSOKPB1hhTeraqBczvpD6A4JoFIIgFZsOSszi4a+HMeL AXXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AZTxn1zH/8CPF8nHBpvAZQPHd/ePx+TIdDehIsUuIw4=; b=bJLMRx6EMhBbgR791WFDIMmqvEtukJe0j33aujWWCfHy73dI6h/8iSDIO2XiGNEcur hVZg4UNcIkgxXUMpmQCAJ205gTivrva1Dkl91VnfxeeXw0VA8oQdPgZwhPgbmBEDiMxY mez2RO1Qc49KVv4OvlpXY0SXgk8a7aTV5/lT++j/nVyMwkAsvd52Zy3QE+EeLk77Ym2h JP3biA4S3eEnJqFIjunDV6GUgZQZY1Lm8vuX2M7IaoimUEEWbaPpCnENRxRL6sqna1ea gx1RPqp6+bLU4YLirUUzFcxTQ0Lvkn12oi0V9KOM+Y+H/dhgOaU90M/dZ6DJ/1t9sJ+1 6kHA== X-Gm-Message-State: AA+aEWZJt5LLNF4i+XqQRDfHhuPqQ3nFfcRoP4aYzDUfftnwYtA8YABU zTFhFf8IwgD0tn7fWUSTGnfGn6+gKOo= X-Google-Smtp-Source: AJdET5ei1HolyyBHYDajRdKoGslkIO6WSbeT9qNPxLX+3xtsgM8lesRvT9c11Y3LayTxrmC1cd1GNg== X-Received: by 2002:a24:85c2:: with SMTP id r185-v6mr24905151itd.104.1543250772982; Mon, 26 Nov 2018 08:46:12 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:11 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 13/20] aio: split out iocb copy from io_submit_one() Date: Mon, 26 Nov 2018 09:45:37 -0700 Message-Id: <20181126164544.5699-14-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation of handing in iocbs in a different fashion as well. Also make it clear that the iocb being passed in isn't modified, by marking it const throughout. Signed-off-by: Jens Axboe --- fs/aio.c | 68 +++++++++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index e8457f9486e3..ba5758c854e8 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1414,7 +1414,7 @@ static void aio_complete_rw(struct kiocb *kiocb, long res, long res2) aio_complete(iocb, res, res2); } -static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) +static int aio_prep_rw(struct kiocb *req, const struct iocb *iocb) { int ret; @@ -1455,7 +1455,7 @@ static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) return ret; } -static int aio_setup_rw(int rw, struct iocb *iocb, struct iovec **iovec, +static int aio_setup_rw(int rw, const struct iocb *iocb, struct iovec **iovec, bool vectored, bool compat, struct iov_iter *iter) { void __user *buf = (void __user *)(uintptr_t)iocb->aio_buf; @@ -1494,8 +1494,8 @@ static inline void aio_rw_done(struct kiocb *req, ssize_t ret) } } -static ssize_t aio_read(struct kiocb *req, struct iocb *iocb, bool vectored, - bool compat) +static ssize_t aio_read(struct kiocb *req, const struct iocb *iocb, + bool vectored, bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; struct iov_iter iter; @@ -1527,8 +1527,8 @@ static ssize_t aio_read(struct kiocb *req, struct iocb *iocb, bool vectored, return ret; } -static ssize_t aio_write(struct kiocb *req, struct iocb *iocb, bool vectored, - bool compat) +static ssize_t aio_write(struct kiocb *req, const struct iocb *iocb, + bool vectored, bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; struct iov_iter iter; @@ -1583,7 +1583,8 @@ static void aio_fsync_work(struct work_struct *work) aio_complete(container_of(req, struct aio_kiocb, fsync), ret, 0); } -static int aio_fsync(struct fsync_iocb *req, struct iocb *iocb, bool datasync) +static int aio_fsync(struct fsync_iocb *req, const struct iocb *iocb, + bool datasync) { if (unlikely(iocb->aio_buf || iocb->aio_offset || iocb->aio_nbytes || iocb->aio_rw_flags)) @@ -1711,7 +1712,7 @@ aio_poll_queue_proc(struct file *file, struct wait_queue_head *head, add_wait_queue(head, &pt->iocb->poll.wait); } -static ssize_t aio_poll(struct aio_kiocb *aiocb, struct iocb *iocb) +static ssize_t aio_poll(struct aio_kiocb *aiocb, const struct iocb *iocb) { struct kioctx *ctx = aiocb->ki_ctx; struct poll_iocb *req = &aiocb->poll; @@ -1783,27 +1784,23 @@ static ssize_t aio_poll(struct aio_kiocb *aiocb, struct iocb *iocb) return 0; } -static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, - bool compat) +static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, + struct iocb __user *user_iocb, bool compat) { struct aio_kiocb *req; - struct iocb iocb; ssize_t ret; - if (unlikely(copy_from_user(&iocb, user_iocb, sizeof(iocb)))) - return -EFAULT; - /* enforce forwards compatibility on users */ - if (unlikely(iocb.aio_reserved2)) { + if (unlikely(iocb->aio_reserved2)) { pr_debug("EINVAL: reserve field set\n"); return -EINVAL; } /* prevent overflows */ if (unlikely( - (iocb.aio_buf != (unsigned long)iocb.aio_buf) || - (iocb.aio_nbytes != (size_t)iocb.aio_nbytes) || - ((ssize_t)iocb.aio_nbytes < 0) + (iocb->aio_buf != (unsigned long)iocb->aio_buf) || + (iocb->aio_nbytes != (size_t)iocb->aio_nbytes) || + ((ssize_t)iocb->aio_nbytes < 0) )) { pr_debug("EINVAL: overflow check\n"); return -EINVAL; @@ -1817,14 +1814,14 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, if (unlikely(!req)) goto out_put_reqs_available; - if (iocb.aio_flags & IOCB_FLAG_RESFD) { + if (iocb->aio_flags & IOCB_FLAG_RESFD) { /* * If the IOCB_FLAG_RESFD flag of aio_flags is set, get an * instance of the file* now. The file descriptor must be * an eventfd() fd, and will be signaled for each completed * event using the eventfd_signal() function. */ - req->ki_eventfd = eventfd_ctx_fdget((int) iocb.aio_resfd); + req->ki_eventfd = eventfd_ctx_fdget((int) iocb->aio_resfd); if (IS_ERR(req->ki_eventfd)) { ret = PTR_ERR(req->ki_eventfd); req->ki_eventfd = NULL; @@ -1839,32 +1836,32 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, } req->ki_user_iocb = user_iocb; - req->ki_user_data = iocb.aio_data; + req->ki_user_data = iocb->aio_data; - switch (iocb.aio_lio_opcode) { + switch (iocb->aio_lio_opcode) { case IOCB_CMD_PREAD: - ret = aio_read(&req->rw, &iocb, false, compat); + ret = aio_read(&req->rw, iocb, false, compat); break; case IOCB_CMD_PWRITE: - ret = aio_write(&req->rw, &iocb, false, compat); + ret = aio_write(&req->rw, iocb, false, compat); break; case IOCB_CMD_PREADV: - ret = aio_read(&req->rw, &iocb, true, compat); + ret = aio_read(&req->rw, iocb, true, compat); break; case IOCB_CMD_PWRITEV: - ret = aio_write(&req->rw, &iocb, true, compat); + ret = aio_write(&req->rw, iocb, true, compat); break; case IOCB_CMD_FSYNC: - ret = aio_fsync(&req->fsync, &iocb, false); + ret = aio_fsync(&req->fsync, iocb, false); break; case IOCB_CMD_FDSYNC: - ret = aio_fsync(&req->fsync, &iocb, true); + ret = aio_fsync(&req->fsync, iocb, true); break; case IOCB_CMD_POLL: - ret = aio_poll(req, &iocb); + ret = aio_poll(req, iocb); break; default: - pr_debug("invalid aio operation %d\n", iocb.aio_lio_opcode); + pr_debug("invalid aio operation %d\n", iocb->aio_lio_opcode); ret = -EINVAL; break; } @@ -1886,6 +1883,17 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, return ret; } +static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, + bool compat) +{ + struct iocb iocb; + + if (unlikely(copy_from_user(&iocb, user_iocb, sizeof(iocb)))) + return -EFAULT; + + return __io_submit_one(ctx, &iocb, user_iocb, compat); +} + /* * Plugging is meant to work with larger batches of IOs. If we don't * have more than the below, then don't bother setting up a plug. From patchwork Mon Nov 26 16:45:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698699 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 067CB15A8 for ; Mon, 26 Nov 2018 16:46:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB7EC2A087 for ; Mon, 26 Nov 2018 16:46:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DFA8C2A08C; Mon, 26 Nov 2018 16:46:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9397F2A087 for ; Mon, 26 Nov 2018 16:46:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726549AbeK0Dkz (ORCPT ); Mon, 26 Nov 2018 22:40:55 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:44787 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727114AbeK0Dky (ORCPT ); Mon, 26 Nov 2018 22:40:54 -0500 Received: by mail-io1-f67.google.com with SMTP id r200so14453274iod.11 for ; Mon, 26 Nov 2018 08:46:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RA6SARsil7zOqlPTyhrfo176dQaQeNwiqXCQQq9Z3o8=; b=JsPb2kveJdT1klVCCo3xKbZcMV24M8JQz4D6n/6ZAqpNT0FtEjlFGPROB1JcYGLUbo ESAe/3xykNiR+QDJMS7dCdVCoxAAX58IhggzIE+AMvnIqTYXBKd5ov7V9Jskc0g/89Nl OnDLSKkX3+Gx1GUyEuUE1tqmgOwO3L4txi6PiMAT/NmjbcHBKioA8X7RvluS9VrimcSv 5nCHngBZlwvlK8j684A/xCTIvCV+Iq1L7RwGSCmeum9PAOowv0Lfd4/4HsV8nQ6JAhMZ aHRwdbQewEBBP9grneNRl8z8S4JlpjaOEesIWUjBdJcpuRkKINzKqV1bTpO/fVIyIxkX aJuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RA6SARsil7zOqlPTyhrfo176dQaQeNwiqXCQQq9Z3o8=; b=UrCV6W3m/A0qSCG3OXYFZvcvgx/Gl0dQEhf4vJdkOBAMMCTGTJDl0OAPRX9XUabJR9 o6rfXXU4hgpAQHq1Tf1sAZcd6eog1AiaLfjMI65A27TfZLQUU7cU0rE1G867CPEotUyA e1uTjB/OBPXh73CZWl2JOkslhOw0JJMMmGAcgUlDxJuOYP4+yiQIO1chFSJUCaC04gEt QtkDPmAv/teV2raMmI8gXRade88km863y+MNyWUqxQMBT72GgF8LwPSzf/HoKKlD3X2O wcgdoeusZWyxQpwzVDMc7q5dFbAHfr9iBfFeUxdE8pjXROQfUKszsf7rMsLEffDhomVZ 0M7Q== X-Gm-Message-State: AA+aEWYouTfbon7fx0WQeHx0UOF/en6JuU4wXr/nX9Esi0mUPa1/Ob5D t4KFEaAZ3FP9quSFjidN9I6KTtA/ni0= X-Google-Smtp-Source: AFSGD/UnDX38YVgQHt1nUrA2qsJ6OQ34zWViSVIAQY/+3Tl8XGkemiNtfHbE8f5Wbt647Pli3AL06w== X-Received: by 2002:a5d:970c:: with SMTP id h12mr16911196iol.3.1543250775407; Mon, 26 Nov 2018 08:46:15 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:14 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 14/20] aio: abstract out io_event filler helper Date: Mon, 26 Nov 2018 09:45:38 -0700 Message-Id: <20181126164544.5699-15-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Jens Axboe --- fs/aio.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index ba5758c854e8..12859ea1cb64 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1057,6 +1057,15 @@ static inline void iocb_put(struct aio_kiocb *iocb) } } +static void aio_fill_event(struct io_event *ev, struct aio_kiocb *iocb, + long res, long res2) +{ + ev->obj = (u64)(unsigned long)iocb->ki_user_iocb; + ev->data = iocb->ki_user_data; + ev->res = res; + ev->res2 = res2; +} + /* aio_complete * Called when the io request on the given iocb is complete. */ @@ -1084,10 +1093,7 @@ static void aio_complete(struct aio_kiocb *iocb, long res, long res2) ev_page = kmap_atomic(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]); event = ev_page + pos % AIO_EVENTS_PER_PAGE; - event->obj = (u64)(unsigned long)iocb->ki_user_iocb; - event->data = iocb->ki_user_data; - event->res = res; - event->res2 = res2; + aio_fill_event(event, iocb, res, res2); kunmap_atomic(ev_page); flush_dcache_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]); From patchwork Mon Nov 26 16:45:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698705 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE90017D5 for ; Mon, 26 Nov 2018 16:46:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A02B02A083 for ; Mon, 26 Nov 2018 16:46:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 945482A088; Mon, 26 Nov 2018 16:46:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 107332A087 for ; Mon, 26 Nov 2018 16:46:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727114AbeK0Dk5 (ORCPT ); Mon, 26 Nov 2018 22:40:57 -0500 Received: from mail-io1-f66.google.com ([209.85.166.66]:34302 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727122AbeK0Dk5 (ORCPT ); Mon, 26 Nov 2018 22:40:57 -0500 Received: by mail-io1-f66.google.com with SMTP id f6so14483620iob.1 for ; Mon, 26 Nov 2018 08:46:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zVtp4FzAAmfv0AO0psddh6PYN7Ojhn91p4dzu/5A1xE=; b=gwo+yj4oRA4/npssvRBvLgGVaYZMjXSSthalnvqzG/2EdhQrW18+E8b6Vr+qXT8DHH xXKeMis/livLC0TdG8WpkKSX7BJeoCCBF7t9W7lT3jFoX+C1l9jB+fcZsq3e5v6FepnI WoNXdpglvfzLGBNmzUEN5AZwuYn+9GmRO4jK88xFYVaW5J9N+voN6Qb7sAypb5cSzXqe Oum4tIdJ/2UcABbag2Opzy0aoAiDNqFeaG5t4Rtnjox5wBOuVly3tcX8guT/8XWPfd7+ kf7XBFSf4PvLNBzvuOPUsgEsp5dZP3J00bBGWpC2BwUFX9i3l+Ck5aTrP8xvIw8bYhpT b82A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zVtp4FzAAmfv0AO0psddh6PYN7Ojhn91p4dzu/5A1xE=; b=e7u0mxvsksSzvWmpLdRz6n1Vhs9cgmxHzENhynkO4O9qGNDRNkt6fIP+XgoOGskrMc ZQT/0ARYxOjP1n3BI+rXmAJfn5vz5LEscr53XCHng2PiJRlqq3+n31VBCGZPDBTtSdal +Rr8XUZkSVhWYVEO2ED2RNfBWdZERRl0UUG7DrGVWOvi1hbFUhNd7xV9pcwGDmo6u02n UdtoQDf/Zw6IFBLfiuj9K90Dl0ZEmM8TR/HiljiUQRxZAb2+Piaiql8jQI4RTvvQUnrD T8Zn5UXoMFeLieXZKyQcfs/woVBKjMaK0+dkd+8uLlL4Px+IryOOV4b2p53zLQFrI3dR ntQw== X-Gm-Message-State: AA+aEWYDnkmPv0N/NqsVdao/e5v/ajjNPUsEr45BwnzMzPbuh3qGDAT4 8wqHcoky2oScOXd5LvzJeweols9jVrE= X-Google-Smtp-Source: AFSGD/Xu0ULtfsLuGjSY3jj4ieITl4hs9xB406H1ZsosDyilpKAhI9MSsDxZB598aXESOiZxUTwDVQ== X-Received: by 2002:a6b:8f8d:: with SMTP id r135mr21697089iod.5.1543250777143; Mon, 26 Nov 2018 08:46:17 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:16 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 15/20] aio: add io_setup2() system call Date: Mon, 26 Nov 2018 09:45:39 -0700 Message-Id: <20181126164544.5699-16-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is just like io_setup(), except add a flags argument to let the caller control/define some of the io_context behavior. Outside of that, we pass in an iocb array for future use. Signed-off-by: Jens Axboe --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/aio.c | 70 ++++++++++++++++---------- include/linux/syscalls.h | 2 + include/uapi/asm-generic/unistd.h | 4 +- kernel/sys_ni.c | 1 + 5 files changed, 50 insertions(+), 28 deletions(-) diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index f0b1709a5ffb..67c357225fb0 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -343,6 +343,7 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +335 common io_setup2 __x64_sys_io_setup2 # # x32-specific system call numbers start at 512 to avoid cache impact diff --git a/fs/aio.c b/fs/aio.c index 12859ea1cb64..74831ce2185e 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -94,6 +94,8 @@ struct kioctx { unsigned long user_id; + unsigned int flags; + struct __percpu kioctx_cpu *cpu; /* @@ -680,21 +682,24 @@ static void aio_nr_sub(unsigned nr) spin_unlock(&aio_nr_lock); } -/* ioctx_alloc - * Allocates and initializes an ioctx. Returns an ERR_PTR if it failed. - */ -static struct kioctx *ioctx_alloc(unsigned nr_events) +static struct kioctx *io_setup_flags(unsigned long ctxid, + unsigned int nr_events, unsigned int flags) { struct mm_struct *mm = current->mm; struct kioctx *ctx; int err = -ENOMEM; - /* * Store the original nr_events -- what userspace passed to io_setup(), * for counting against the global limit -- before it changes. */ unsigned int max_reqs = nr_events; + if (unlikely(ctxid || nr_events == 0)) { + pr_debug("EINVAL: ctx %lu nr_events %u\n", + ctxid, nr_events); + return ERR_PTR(-EINVAL); + } + /* * We keep track of the number of available ringbuffer slots, to prevent * overflow (reqs_available), and we also use percpu counters for this. @@ -720,6 +725,7 @@ static struct kioctx *ioctx_alloc(unsigned nr_events) if (!ctx) return ERR_PTR(-ENOMEM); + ctx->flags = flags; ctx->max_reqs = max_reqs; spin_lock_init(&ctx->ctx_lock); @@ -1275,6 +1281,33 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr, return ret; } +SYSCALL_DEFINE4(io_setup2, u32, nr_events, u32, flags, struct iocb * __user, + iocbs, aio_context_t __user *, ctxp) +{ + struct kioctx *ioctx; + unsigned long ctx; + long ret; + + if (flags) + return -EINVAL; + + ret = get_user(ctx, ctxp); + if (unlikely(ret)) + goto out; + + ioctx = io_setup_flags(ctx, nr_events, flags); + ret = PTR_ERR(ioctx); + if (IS_ERR(ioctx)) + goto out; + + ret = put_user(ioctx->user_id, ctxp); + if (ret) + kill_ioctx(current->mm, ioctx, NULL); + percpu_ref_put(&ioctx->users); +out: + return ret; +} + /* sys_io_setup: * Create an aio_context capable of receiving at least nr_events. * ctxp must not point to an aio_context that already exists, and @@ -1290,7 +1323,7 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr, */ SYSCALL_DEFINE2(io_setup, unsigned, nr_events, aio_context_t __user *, ctxp) { - struct kioctx *ioctx = NULL; + struct kioctx *ioctx; unsigned long ctx; long ret; @@ -1298,14 +1331,7 @@ SYSCALL_DEFINE2(io_setup, unsigned, nr_events, aio_context_t __user *, ctxp) if (unlikely(ret)) goto out; - ret = -EINVAL; - if (unlikely(ctx || nr_events == 0)) { - pr_debug("EINVAL: ctx %lu nr_events %u\n", - ctx, nr_events); - goto out; - } - - ioctx = ioctx_alloc(nr_events); + ioctx = io_setup_flags(ctx, nr_events, 0); ret = PTR_ERR(ioctx); if (!IS_ERR(ioctx)) { ret = put_user(ioctx->user_id, ctxp); @@ -1313,7 +1339,6 @@ SYSCALL_DEFINE2(io_setup, unsigned, nr_events, aio_context_t __user *, ctxp) kill_ioctx(current->mm, ioctx, NULL); percpu_ref_put(&ioctx->users); } - out: return ret; } @@ -1321,7 +1346,7 @@ SYSCALL_DEFINE2(io_setup, unsigned, nr_events, aio_context_t __user *, ctxp) #ifdef CONFIG_COMPAT COMPAT_SYSCALL_DEFINE2(io_setup, unsigned, nr_events, u32 __user *, ctx32p) { - struct kioctx *ioctx = NULL; + struct kioctx *ioctx; unsigned long ctx; long ret; @@ -1329,23 +1354,14 @@ COMPAT_SYSCALL_DEFINE2(io_setup, unsigned, nr_events, u32 __user *, ctx32p) if (unlikely(ret)) goto out; - ret = -EINVAL; - if (unlikely(ctx || nr_events == 0)) { - pr_debug("EINVAL: ctx %lu nr_events %u\n", - ctx, nr_events); - goto out; - } - - ioctx = ioctx_alloc(nr_events); + ioctx = io_setup_flags(ctx, nr_events, 0); ret = PTR_ERR(ioctx); if (!IS_ERR(ioctx)) { - /* truncating is ok because it's a user address */ - ret = put_user((u32)ioctx->user_id, ctx32p); + ret = put_user(ioctx->user_id, ctx32p); if (ret) kill_ioctx(current->mm, ioctx, NULL); percpu_ref_put(&ioctx->users); } - out: return ret; } diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index 2ac3d13a915b..b661e78717e6 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -287,6 +287,8 @@ static inline void addr_limit_user_check(void) */ #ifndef CONFIG_ARCH_HAS_SYSCALL_WRAPPER asmlinkage long sys_io_setup(unsigned nr_reqs, aio_context_t __user *ctx); +asmlinkage long sys_io_setup2(unsigned, unsigned, struct iocb __user *, + aio_context_t __user *); asmlinkage long sys_io_destroy(aio_context_t ctx); asmlinkage long sys_io_submit(aio_context_t, long, struct iocb __user * __user *); diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 538546edbfbd..b4527ed373b0 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -738,9 +738,11 @@ __SYSCALL(__NR_statx, sys_statx) __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents) #define __NR_rseq 293 __SYSCALL(__NR_rseq, sys_rseq) +#define __NR_io_setup2 294 +__SYSCALL(__NR_io_setup2, sys_io_setup2) #undef __NR_syscalls -#define __NR_syscalls 294 +#define __NR_syscalls 295 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index df556175be50..17c8b4393669 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -37,6 +37,7 @@ asmlinkage long sys_ni_syscall(void) */ COND_SYSCALL(io_setup); +COND_SYSCALL(io_setup2); COND_SYSCALL_COMPAT(io_setup); COND_SYSCALL(io_destroy); COND_SYSCALL(io_submit); From patchwork Mon Nov 26 16:45:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698709 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89B8C17D5 for ; Mon, 26 Nov 2018 16:46:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B0DA2A083 for ; Mon, 26 Nov 2018 16:46:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F9312A088; Mon, 26 Nov 2018 16:46:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E1ADF2A087 for ; Mon, 26 Nov 2018 16:46:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727122AbeK0Dk7 (ORCPT ); Mon, 26 Nov 2018 22:40:59 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:56044 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727131AbeK0Dk7 (ORCPT ); Mon, 26 Nov 2018 22:40:59 -0500 Received: by mail-it1-f193.google.com with SMTP id o19so29265801itg.5 for ; Mon, 26 Nov 2018 08:46:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zWP6coMWt07nFsQ2bCKPJsd0fyiDRI0lMmSnLaRva2o=; b=usfYWlUi2uqiKReQCULh3/c58WLx/7awhBs0tRsajj8S4ioh/nqGHzbpUH7hRqDvqU 7b3/cZ09F2+uAFnK1Ywr2Ziv1oQcjlAnI0YbSxTZLjID82YwlpbhS/WTwIlnYWlyX51y 0XwZ2/YECZICJmd0cfNhtLzcbOPZL9fyKgoqcvCEV7dHYFxqXA81GvG4DCSBoECwfBUl ysi4me5m47AdLZA6TVTGLStIxhEX3mH2yzbMXYdwNPonuh8CX54jos6hA1IJoaXU9t3O lAF76i1ku0GZLbZRYFlKlYxf9s5dAPzfZ18lPRd2fElh03EHSdXpNPLiMlieH5PQR/FR S31g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zWP6coMWt07nFsQ2bCKPJsd0fyiDRI0lMmSnLaRva2o=; b=aD0AIHEhAX1Wwz/FOFfGnW/AcIH4z0HIbkznVfEJ7BJ5jNEEVhGXCWjC5lh5wcZElx nbfesDtrXUyQ8we2Wz2Nui7ifrQmahhGLvD9s9nRxBIdFUYl9zrwdLWPTZcD+os1bTkb MndWYy3pyDDOGeyj1x/E48vLzbXFS94pGg6AbdGR2RvEZKBavvjE3XxyZw2C8s5bN7gw uVgJuIoR9mkT0ea3LjqtoGdbK+yjvIWYCIilGLYZ2iT99V/CI3yDu2tGJCAW//G8vQ+l 4XzY/RvexQJN4XpNLyJYVtCztwVJeuywVumfWb1a288wMdEEi4j1QhNT8XQQxZ6eK6UP MXnQ== X-Gm-Message-State: AA+aEWbCLC3rZiaQEu7FMTGRUk+1ozH2vupCyVp0FMkwm5N6VKyspafe 22Hp5frLP72K0k8bQnMGZ01kjvAejok= X-Google-Smtp-Source: AFSGD/UIQ+qF9QnYQgfHW8Qu+jPteNhC11a5yA1vg3hjZUWhqYLVMmMH0Xfcv5jX9RFDhEGjT1GoEA== X-Received: by 2002:a24:c40b:: with SMTP id v11mr1251131itf.73.1543250779133; Mon, 26 Nov 2018 08:46:19 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:17 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 16/20] aio: add support for having user mapped iocbs Date: Mon, 26 Nov 2018 09:45:40 -0700 Message-Id: <20181126164544.5699-17-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For io_submit(), we have to first copy each pointer to an iocb, then copy the iocb. The latter is 64 bytes in size, and that's a lot of copying for a single IO. Add support for setting IOCTX_FLAG_USERIOCB through the new io_setup2() system call, which allows the iocbs to reside in userspace. If this flag is used, then io_submit() doesn't take pointers to iocbs anymore, it takes an index value into the array of iocbs instead. Similary, for io_getevents(), the iocb ->obj will be the index, not the pointer to the iocb. See the change made to fio to support this feature, it's pretty trivialy to adapt to. For applications, like fio, that previously embedded the iocb inside a application private structure, some sort of lookup table/structure is needed to find the private IO structure from the index at io_getevents() time. http://git.kernel.dk/cgit/fio/commit/?id=3c3168e91329c83880c91e5abc28b9d6b940fd95 Signed-off-by: Jens Axboe --- fs/aio.c | 99 +++++++++++++++++++++++++++++++----- include/uapi/linux/aio_abi.h | 2 + 2 files changed, 89 insertions(+), 12 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 74831ce2185e..e98121df92f6 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -121,6 +121,9 @@ struct kioctx { struct page **ring_pages; long nr_pages; + struct page **iocb_pages; + long iocb_nr_pages; + struct rcu_work free_rwork; /* see free_ioctx() */ /* @@ -216,6 +219,11 @@ static struct vfsmount *aio_mnt; static const struct file_operations aio_ring_fops; static const struct address_space_operations aio_ctx_aops; +static const unsigned int iocb_page_shift = + ilog2(PAGE_SIZE / sizeof(struct iocb)); + +static void aio_useriocb_free(struct kioctx *); + static struct file *aio_private_file(struct kioctx *ctx, loff_t nr_pages) { struct file *file; @@ -572,6 +580,7 @@ static void free_ioctx(struct work_struct *work) free_rwork); pr_debug("freeing %p\n", ctx); + aio_useriocb_free(ctx); aio_free_ring(ctx); free_percpu(ctx->cpu); percpu_ref_exit(&ctx->reqs); @@ -1281,6 +1290,45 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr, return ret; } +static void aio_useriocb_free(struct kioctx *ctx) +{ + int i; + + if (!ctx->iocb_nr_pages) + return; + + for (i = 0; i < ctx->iocb_nr_pages; i++) + put_page(ctx->iocb_pages[i]); + + kfree(ctx->iocb_pages); +} + +static int aio_useriocb_map(struct kioctx *ctx, struct iocb __user *iocbs, + unsigned int nr_events) +{ + int nr_pages, ret; + + if ((unsigned long) iocbs & ~PAGE_MASK) + return -EINVAL; + + nr_pages = sizeof(struct iocb) * nr_events; + nr_pages = (nr_pages + PAGE_SIZE - 1) >> PAGE_SHIFT; + + ctx->iocb_pages = kzalloc(nr_pages * sizeof(struct page *), GFP_KERNEL); + if (!ctx->iocb_pages) + return -ENOMEM; + + ret = get_user_pages_fast((unsigned long) iocbs, nr_pages, 0, + ctx->iocb_pages); + if (ret < nr_pages) { + kfree(ctx->iocb_pages); + return -ENOMEM; + } + + ctx->iocb_nr_pages = nr_pages; + return 0; +} + SYSCALL_DEFINE4(io_setup2, u32, nr_events, u32, flags, struct iocb * __user, iocbs, aio_context_t __user *, ctxp) { @@ -1288,7 +1336,7 @@ SYSCALL_DEFINE4(io_setup2, u32, nr_events, u32, flags, struct iocb * __user, unsigned long ctx; long ret; - if (flags) + if (flags & ~IOCTX_FLAG_USERIOCB) return -EINVAL; ret = get_user(ctx, ctxp); @@ -1300,9 +1348,17 @@ SYSCALL_DEFINE4(io_setup2, u32, nr_events, u32, flags, struct iocb * __user, if (IS_ERR(ioctx)) goto out; + if (flags & IOCTX_FLAG_USERIOCB) { + ret = aio_useriocb_map(ioctx, iocbs, nr_events); + if (ret) + goto err; + } + ret = put_user(ioctx->user_id, ctxp); - if (ret) + if (ret) { +err: kill_ioctx(current->mm, ioctx, NULL); + } percpu_ref_put(&ioctx->users); out: return ret; @@ -1851,10 +1907,13 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, } } - ret = put_user(KIOCB_KEY, &user_iocb->aio_key); - if (unlikely(ret)) { - pr_debug("EFAULT: aio_key\n"); - goto out_put_req; + /* Don't support cancel on user mapped iocbs */ + if (!(ctx->flags & IOCTX_FLAG_USERIOCB)) { + ret = put_user(KIOCB_KEY, &user_iocb->aio_key); + if (unlikely(ret)) { + pr_debug("EFAULT: aio_key\n"); + goto out_put_req; + } } req->ki_user_iocb = user_iocb; @@ -1908,12 +1967,26 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, bool compat) { - struct iocb iocb; + struct iocb iocb, *iocbp; - if (unlikely(copy_from_user(&iocb, user_iocb, sizeof(iocb)))) - return -EFAULT; + if (ctx->flags & IOCTX_FLAG_USERIOCB) { + unsigned long iocb_index = (unsigned long) user_iocb; + unsigned int page_index; - return __io_submit_one(ctx, &iocb, user_iocb, compat); + if (iocb_index >= ctx->nr_events) + return -EINVAL; + + page_index = iocb_index >> iocb_page_shift; + iocb_index &= ((1 << iocb_page_shift) - 1); + iocbp = page_address(ctx->iocb_pages[page_index]); + iocbp += iocb_index; + } else { + if (unlikely(copy_from_user(&iocb, user_iocb, sizeof(iocb)))) + return -EFAULT; + iocbp = &iocb; + } + + return __io_submit_one(ctx, iocbp, user_iocb, compat); } /* @@ -2063,6 +2136,9 @@ SYSCALL_DEFINE3(io_cancel, aio_context_t, ctx_id, struct iocb __user *, iocb, if (unlikely(!ctx)) return -EINVAL; + if (ctx->flags & IOCTX_FLAG_USERIOCB) + goto err; + spin_lock_irq(&ctx->ctx_lock); kiocb = lookup_kiocb(ctx, iocb); if (kiocb) { @@ -2079,9 +2155,8 @@ SYSCALL_DEFINE3(io_cancel, aio_context_t, ctx_id, struct iocb __user *, iocb, */ ret = -EINPROGRESS; } - +err: percpu_ref_put(&ctx->users); - return ret; } diff --git a/include/uapi/linux/aio_abi.h b/include/uapi/linux/aio_abi.h index 8387e0af0f76..814e6606c413 100644 --- a/include/uapi/linux/aio_abi.h +++ b/include/uapi/linux/aio_abi.h @@ -106,6 +106,8 @@ struct iocb { __u32 aio_resfd; }; /* 64 bytes */ +#define IOCTX_FLAG_USERIOCB (1 << 0) /* iocbs are user mapped */ + #undef IFBIG #undef IFLITTLE From patchwork Mon Nov 26 16:45:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698711 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EFF414E2 for ; Mon, 26 Nov 2018 16:46:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E3EE2A087 for ; Mon, 26 Nov 2018 16:46:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 51FCD2A091; Mon, 26 Nov 2018 16:46:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 141B42A087 for ; Mon, 26 Nov 2018 16:46:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727151AbeK0DlD (ORCPT ); Mon, 26 Nov 2018 22:41:03 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:39984 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727140AbeK0DlC (ORCPT ); Mon, 26 Nov 2018 22:41:02 -0500 Received: by mail-it1-f195.google.com with SMTP id h193so28733129ita.5 for ; Mon, 26 Nov 2018 08:46:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Oz3wGbZin5azN0OdUl5D8Ag+PdEen0dYBtBtNvuBJl4=; b=jInDgobr6x6UXBe2df3ieXAuutaRFlCJDv0uzq/SYBNq7LdIXO5NjBrclcJb0bbwEk E19l2SEP69e1qvgaVhsSFsHsPy23MI7XHiFON33x1ArvpR9OgFdvcZOwN5LizPo9twj3 NGnCB4P+wP8lmXkVLPW0XRG9luXA7WQF5Oas/Y3IG9Tjo565QnDN2yRArw8kwvYUcZPK PApNzxdBkqfRwWtw3ZTMUXFE8qcqepGW2aTiodzEeF/uX6OOy+ghRfkgEQPS+QXnNTqB Z1XNs81fE+aZh/fnxZV4ptnkykJSHr5pxld8744ppHpzh1QRg6mxcxdZVHfkjwv6J+qT Aktg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Oz3wGbZin5azN0OdUl5D8Ag+PdEen0dYBtBtNvuBJl4=; b=MzO3WtrGsT15KFt7oIAQN+kqe6vVZwyKIxql0DeRoqCPyUc2wTpRU09/HIXm/UgXd0 gRGv4zTH8kg2KPDfnSDwdz8cVUmU1LM+hME0Gb7it9t4kWyk7Rgth0pFLg/7+Oe9aXmn U8JWps8J2JgWxjJbAw/NyO6tib2fdBI6oDWo9qiUVZJ0N1QoNb1WG1plMyKzLnwxeMtK 1mM2lhDGGOq0KVJBBJJ2Pd2IWFfxqfUeY0W/KWF+am/X+wvURCBsxQE8xHt1x/ymyKjP 4AeN9jyBRUEYQonl0sQzeZXPeloLRo7wbYfZ8WVX0zGTesHgK8K84WFVBiA1yFuFyTds 3WIQ== X-Gm-Message-State: AGRZ1gLipTvNT/cCrmvme/+HUjVaXtCZCdTFfLwrT7W8WlIz0hlo6kGd 8zl5bdHKgvXRj4rmg3kcyawcy6ciZcE= X-Google-Smtp-Source: AFSGD/W4+V2Zdg+bMFMv0yfxgaeBHxLpiXDiEWyN6l2SN2IhWBedMYl3sC7NKmOEOM1l7fpJyuHAGw== X-Received: by 2002:a24:2943:: with SMTP id p64mr24361932itp.94.1543250781644; Mon, 26 Nov 2018 08:46:21 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:20 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 17/20] aio: support for IO polling Date: Mon, 26 Nov 2018 09:45:41 -0700 Message-Id: <20181126164544.5699-18-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add polled variants of PREAD/PREADV and PWRITE/PWRITEV. These act like their non-polled counterparts, except we expect to poll for completion of them. The polling happens at io_getevent() time, and works just like non-polled IO. To setup an io_context for polled IO, the application must call io_setup2() with IOCTX_FLAG_IOPOLL as one of the flags. It is illegal to mix and match polled and non-polled IO on an io_context. Polled IO doesn't support the user mapped completion ring. Events must be reaped through the io_getevents() system call. For non-irq driven poll devices, there's no way to support completion reaping from userspace by just looking at the ring. The application itself is the one that pulls completion entries. Signed-off-by: Jens Axboe --- fs/aio.c | 378 +++++++++++++++++++++++++++++++---- include/uapi/linux/aio_abi.h | 3 + 2 files changed, 345 insertions(+), 36 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index e98121df92f6..db73c8af1a0a 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -143,6 +143,18 @@ struct kioctx { atomic_t reqs_available; } ____cacheline_aligned_in_smp; + /* iopoll submission state */ + struct { + spinlock_t poll_lock; + struct list_head poll_submitted; + } ____cacheline_aligned_in_smp; + + /* iopoll completion state */ + struct { + struct list_head poll_completing; + struct mutex getevents_lock; + } ____cacheline_aligned_in_smp; + struct { spinlock_t ctx_lock; struct list_head active_reqs; /* used for cancellation */ @@ -195,14 +207,27 @@ struct aio_kiocb { __u64 ki_user_data; /* user's data for completion */ struct list_head ki_list; /* the aio core uses this - * for cancellation */ + * for cancellation, or for + * polled IO */ + + unsigned long ki_flags; +#define IOCB_POLL_COMPLETED 0 +#define IOCB_POLL_BUSY 1 + refcount_t ki_refcnt; - /* - * If the aio_resfd field of the userspace iocb is not zero, - * this is the underlying eventfd context to deliver events to. - */ - struct eventfd_ctx *ki_eventfd; + union { + /* + * If the aio_resfd field of the userspace iocb is not zero, + * this is the underlying eventfd context to deliver events to. + */ + struct eventfd_ctx *ki_eventfd; + + /* + * For polled IO, stash completion info here + */ + struct io_event ki_ev; + }; }; /*------ sysctl variables----*/ @@ -223,6 +248,7 @@ static const unsigned int iocb_page_shift = ilog2(PAGE_SIZE / sizeof(struct iocb)); static void aio_useriocb_free(struct kioctx *); +static void aio_iopoll_reap_events(struct kioctx *); static struct file *aio_private_file(struct kioctx *ctx, loff_t nr_pages) { @@ -461,11 +487,15 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) int i; struct file *file; - /* Compensate for the ring buffer's head/tail overlap entry */ - nr_events += 2; /* 1 is required, 2 for good luck */ - + /* + * Compensate for the ring buffer's head/tail overlap entry. + * IO polling doesn't require any io event entries + */ size = sizeof(struct aio_ring); - size += sizeof(struct io_event) * nr_events; + if (!(ctx->flags & IOCTX_FLAG_IOPOLL)) { + nr_events += 2; /* 1 is required, 2 for good luck */ + size += sizeof(struct io_event) * nr_events; + } nr_pages = PFN_UP(size); if (nr_pages < 0) @@ -747,6 +777,11 @@ static struct kioctx *io_setup_flags(unsigned long ctxid, INIT_LIST_HEAD(&ctx->active_reqs); + spin_lock_init(&ctx->poll_lock); + INIT_LIST_HEAD(&ctx->poll_submitted); + INIT_LIST_HEAD(&ctx->poll_completing); + mutex_init(&ctx->getevents_lock); + if (percpu_ref_init(&ctx->users, free_ioctx_users, 0, GFP_KERNEL)) goto err; @@ -818,11 +853,15 @@ static int kill_ioctx(struct mm_struct *mm, struct kioctx *ctx, { struct kioctx_table *table; + mutex_lock(&ctx->getevents_lock); spin_lock(&mm->ioctx_lock); if (atomic_xchg(&ctx->dead, 1)) { spin_unlock(&mm->ioctx_lock); + mutex_unlock(&ctx->getevents_lock); return -EINVAL; } + aio_iopoll_reap_events(ctx); + mutex_unlock(&ctx->getevents_lock); table = rcu_dereference_raw(mm->ioctx_table); WARN_ON(ctx != rcu_access_pointer(table->table[ctx->id])); @@ -1029,6 +1068,7 @@ static inline struct aio_kiocb *aio_get_req(struct kioctx *ctx) percpu_ref_get(&ctx->reqs); req->ki_ctx = ctx; INIT_LIST_HEAD(&req->ki_list); + req->ki_flags = 0; refcount_set(&req->ki_refcnt, 0); req->ki_eventfd = NULL; } @@ -1072,6 +1112,15 @@ static inline void iocb_put(struct aio_kiocb *iocb) } } +static void iocb_put_many(struct kioctx *ctx, void **iocbs, int *nr) +{ + if (nr) { + percpu_ref_put_many(&ctx->reqs, *nr); + kmem_cache_free_bulk(kiocb_cachep, *nr, iocbs); + *nr = 0; + } +} + static void aio_fill_event(struct io_event *ev, struct aio_kiocb *iocb, long res, long res2) { @@ -1261,6 +1310,166 @@ static bool aio_read_events(struct kioctx *ctx, long min_nr, long nr, return ret < 0 || *i >= min_nr; } +#define AIO_IOPOLL_BATCH 8 + +/* + * Process completed iocb iopoll entries, copying the result to userspace. + */ +static long aio_iopoll_reap(struct kioctx *ctx, struct io_event __user *evs, + unsigned int *nr_events, long max) +{ + void *iocbs[AIO_IOPOLL_BATCH]; + struct aio_kiocb *iocb, *n; + int to_free = 0, ret = 0; + + list_for_each_entry_safe(iocb, n, &ctx->poll_completing, ki_list) { + if (*nr_events == max) + break; + if (!test_bit(IOCB_POLL_COMPLETED, &iocb->ki_flags)) + continue; + if (to_free == AIO_IOPOLL_BATCH) + iocb_put_many(ctx, iocbs, &to_free); + + list_del(&iocb->ki_list); + iocbs[to_free++] = iocb; + + fput(iocb->rw.ki_filp); + + if (evs && copy_to_user(evs + *nr_events, &iocb->ki_ev, + sizeof(iocb->ki_ev))) { + ret = -EFAULT; + break; + } + (*nr_events)++; + } + + if (to_free) + iocb_put_many(ctx, iocbs, &to_free); + + return ret; +} + +static int __aio_iopoll_check(struct kioctx *ctx, struct io_event __user *event, + unsigned int *nr_events, long min, long max) +{ + struct aio_kiocb *iocb; + int to_poll, polled, ret; + + /* + * Check if we already have done events that satisfy what we need + */ + if (!list_empty(&ctx->poll_completing)) { + ret = aio_iopoll_reap(ctx, event, nr_events, max); + if (ret < 0) + return ret; + if (*nr_events >= min) + return 0; + } + + /* + * Take in a new working set from the submitted list, if possible. + */ + if (!list_empty_careful(&ctx->poll_submitted)) { + spin_lock(&ctx->poll_lock); + list_splice_init(&ctx->poll_submitted, &ctx->poll_completing); + spin_unlock(&ctx->poll_lock); + } + + if (list_empty(&ctx->poll_completing)) + return 0; + + /* + * Check again now that we have a new batch. + */ + ret = aio_iopoll_reap(ctx, event, nr_events, max); + if (ret < 0) + return ret; + if (*nr_events >= min) + return 0; + + /* + * Find up to 'max' worth of events to poll for, including the + * events we already successfully polled + */ + polled = to_poll = 0; + list_for_each_entry(iocb, &ctx->poll_completing, ki_list) { + /* + * Poll for needed events with spin == true, anything after + * that we just check if we have more, up to max. + */ + bool spin = polled + *nr_events >= min; + struct kiocb *kiocb = &iocb->rw; + + if (test_bit(IOCB_POLL_COMPLETED, &iocb->ki_flags)) + break; + if (++to_poll + *nr_events > max) + break; + + ret = kiocb->ki_filp->f_op->iopoll(kiocb, spin); + if (ret < 0) + return ret; + + polled += ret; + if (polled + *nr_events >= max) + break; + } + + ret = aio_iopoll_reap(ctx, event, nr_events, max); + if (ret < 0) + return ret; + if (*nr_events >= min) + return 0; + return to_poll; +} + +/* + * We can't just wait for polled events to come to us, we have to actively + * find and complete them. + */ +static void aio_iopoll_reap_events(struct kioctx *ctx) +{ + if (!(ctx->flags & IOCTX_FLAG_IOPOLL)) + return; + + while (!list_empty_careful(&ctx->poll_submitted) || + !list_empty(&ctx->poll_completing)) { + unsigned int nr_events = 0; + + __aio_iopoll_check(ctx, NULL, &nr_events, 1, UINT_MAX); + } +} + +static int aio_iopoll_check(struct kioctx *ctx, long min_nr, long nr, + struct io_event __user *event) +{ + unsigned int nr_events = 0; + int ret = 0; + + /* Only allow one thread polling at a time */ + if (!mutex_trylock(&ctx->getevents_lock)) + return -EBUSY; + if (unlikely(atomic_read(&ctx->dead))) { + ret = -EINVAL; + goto err; + } + + while (!nr_events || !need_resched()) { + int tmin = 0; + + if (nr_events < min_nr) + tmin = min_nr - nr_events; + + ret = __aio_iopoll_check(ctx, event, &nr_events, tmin, nr); + if (ret <= 0) + break; + ret = 0; + } + +err: + mutex_unlock(&ctx->getevents_lock); + return nr_events ? nr_events : ret; +} + static long read_events(struct kioctx *ctx, long min_nr, long nr, struct io_event __user *event, ktime_t until) @@ -1336,7 +1545,7 @@ SYSCALL_DEFINE4(io_setup2, u32, nr_events, u32, flags, struct iocb * __user, unsigned long ctx; long ret; - if (flags & ~IOCTX_FLAG_USERIOCB) + if (flags & ~(IOCTX_FLAG_USERIOCB | IOCTX_FLAG_IOPOLL)) return -EINVAL; ret = get_user(ctx, ctxp); @@ -1469,13 +1678,8 @@ static void aio_remove_iocb(struct aio_kiocb *iocb) spin_unlock_irqrestore(&ctx->ctx_lock, flags); } -static void aio_complete_rw(struct kiocb *kiocb, long res, long res2) +static void kiocb_end_write(struct kiocb *kiocb) { - struct aio_kiocb *iocb = container_of(kiocb, struct aio_kiocb, rw); - - if (!list_empty_careful(&iocb->ki_list)) - aio_remove_iocb(iocb); - if (kiocb->ki_flags & IOCB_WRITE) { struct inode *inode = file_inode(kiocb->ki_filp); @@ -1487,19 +1691,48 @@ static void aio_complete_rw(struct kiocb *kiocb, long res, long res2) __sb_writers_acquired(inode->i_sb, SB_FREEZE_WRITE); file_end_write(kiocb->ki_filp); } +} + +static void aio_complete_rw(struct kiocb *kiocb, long res, long res2) +{ + struct aio_kiocb *iocb = container_of(kiocb, struct aio_kiocb, rw); + + if (!list_empty_careful(&iocb->ki_list)) + aio_remove_iocb(iocb); + + kiocb_end_write(kiocb); fput(kiocb->ki_filp); aio_complete(iocb, res, res2); } -static int aio_prep_rw(struct kiocb *req, const struct iocb *iocb) +static void aio_complete_rw_poll(struct kiocb *kiocb, long res, long res2) { + struct aio_kiocb *iocb = container_of(kiocb, struct aio_kiocb, rw); + + kiocb_end_write(kiocb); + + /* + * Handle EAGAIN from resource limits with polled IO inline, don't + * pass the event back to userspace. + */ + if (unlikely(res == -EAGAIN)) + set_bit(IOCB_POLL_BUSY, &iocb->ki_flags); + else { + aio_fill_event(&iocb->ki_ev, iocb, res, res2); + set_bit(IOCB_POLL_COMPLETED, &iocb->ki_flags); + } +} + +static int aio_prep_rw(struct aio_kiocb *kiocb, const struct iocb *iocb) +{ + struct kioctx *ctx = kiocb->ki_ctx; + struct kiocb *req = &kiocb->rw; int ret; req->ki_filp = fget(iocb->aio_fildes); if (unlikely(!req->ki_filp)) return -EBADF; - req->ki_complete = aio_complete_rw; req->ki_pos = iocb->aio_offset; req->ki_flags = iocb_flags(req->ki_filp); if (iocb->aio_flags & IOCB_FLAG_RESFD) @@ -1525,9 +1758,35 @@ static int aio_prep_rw(struct kiocb *req, const struct iocb *iocb) if (unlikely(ret)) goto out_fput; - req->ki_flags &= ~IOCB_HIPRI; /* no one is going to poll for this I/O */ - return 0; + if (iocb->aio_flags & IOCB_FLAG_HIPRI) { + /* shares space in the union, and is rather pointless.. */ + ret = -EINVAL; + if (iocb->aio_flags & IOCB_FLAG_RESFD) + goto out_fput; + + /* can't submit polled IO to a non-polled ctx */ + if (!(ctx->flags & IOCTX_FLAG_IOPOLL)) + goto out_fput; + + ret = -EOPNOTSUPP; + if (!(req->ki_flags & IOCB_DIRECT) || + !req->ki_filp->f_op->iopoll) + goto out_fput; + + req->ki_flags |= IOCB_HIPRI; + req->ki_complete = aio_complete_rw_poll; + } else { + /* can't submit non-polled IO to a polled ctx */ + ret = -EINVAL; + if (ctx->flags & IOCTX_FLAG_IOPOLL) + goto out_fput; + + /* no one is going to poll for this I/O */ + req->ki_flags &= ~IOCB_HIPRI; + req->ki_complete = aio_complete_rw; + } + return 0; out_fput: fput(req->ki_filp); return ret; @@ -1570,17 +1829,43 @@ static inline void aio_rw_done(struct kiocb *req, ssize_t ret) default: req->ki_complete(req, ret, 0); } + } -static ssize_t aio_read(struct kiocb *req, const struct iocb *iocb, +/* + * After the iocb has been issued, it's safe to be found on the poll list. + * Adding the kiocb to the list AFTER submission ensures that we don't + * find it from a io_getevents() thread before the issuer is done accessing + * the kiocb cookie. + */ +static void aio_iopoll_iocb_issued(struct aio_kiocb *kiocb) +{ + /* + * For fast devices, IO may have already completed. If it has, add + * it to the front so we find it first. We can't add to the poll_done + * list as that's unlocked from the completion side. + */ + const int front_add = test_bit(IOCB_POLL_COMPLETED, &kiocb->ki_flags); + struct kioctx *ctx = kiocb->ki_ctx; + + spin_lock(&ctx->poll_lock); + if (front_add) + list_add(&kiocb->ki_list, &ctx->poll_submitted); + else + list_add_tail(&kiocb->ki_list, &ctx->poll_submitted); + spin_unlock(&ctx->poll_lock); +} + +static ssize_t aio_read(struct aio_kiocb *kiocb, const struct iocb *iocb, bool vectored, bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; + struct kiocb *req = &kiocb->rw; struct iov_iter iter; struct file *file; ssize_t ret; - ret = aio_prep_rw(req, iocb); + ret = aio_prep_rw(kiocb, iocb); if (ret) return ret; file = req->ki_filp; @@ -1605,15 +1890,16 @@ static ssize_t aio_read(struct kiocb *req, const struct iocb *iocb, return ret; } -static ssize_t aio_write(struct kiocb *req, const struct iocb *iocb, +static ssize_t aio_write(struct aio_kiocb *kiocb, const struct iocb *iocb, bool vectored, bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; + struct kiocb *req = &kiocb->rw; struct iov_iter iter; struct file *file; ssize_t ret; - ret = aio_prep_rw(req, iocb); + ret = aio_prep_rw(kiocb, iocb); if (ret) return ret; file = req->ki_filp; @@ -1884,7 +2170,8 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, return -EINVAL; } - if (!get_reqs_available(ctx)) + /* Poll IO doesn't need ring reservations */ + if (!(ctx->flags & IOCTX_FLAG_IOPOLL) && !get_reqs_available(ctx)) return -EAGAIN; ret = -EAGAIN; @@ -1907,8 +2194,8 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, } } - /* Don't support cancel on user mapped iocbs */ - if (!(ctx->flags & IOCTX_FLAG_USERIOCB)) { + /* Don't support cancel on user mapped iocbs or polled context */ + if (!(ctx->flags & (IOCTX_FLAG_USERIOCB | IOCTX_FLAG_IOPOLL))) { ret = put_user(KIOCB_KEY, &user_iocb->aio_key); if (unlikely(ret)) { pr_debug("EFAULT: aio_key\n"); @@ -1919,26 +2206,33 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, req->ki_user_iocb = user_iocb; req->ki_user_data = iocb->aio_data; + ret = -EINVAL; switch (iocb->aio_lio_opcode) { case IOCB_CMD_PREAD: - ret = aio_read(&req->rw, iocb, false, compat); + ret = aio_read(req, iocb, false, compat); break; case IOCB_CMD_PWRITE: - ret = aio_write(&req->rw, iocb, false, compat); + ret = aio_write(req, iocb, false, compat); break; case IOCB_CMD_PREADV: - ret = aio_read(&req->rw, iocb, true, compat); + ret = aio_read(req, iocb, true, compat); break; case IOCB_CMD_PWRITEV: - ret = aio_write(&req->rw, iocb, true, compat); + ret = aio_write(req, iocb, true, compat); break; case IOCB_CMD_FSYNC: + if (ctx->flags & IOCTX_FLAG_IOPOLL) + break; ret = aio_fsync(&req->fsync, iocb, false); break; case IOCB_CMD_FDSYNC: + if (ctx->flags & IOCTX_FLAG_IOPOLL) + break; ret = aio_fsync(&req->fsync, iocb, true); break; case IOCB_CMD_POLL: + if (ctx->flags & IOCTX_FLAG_IOPOLL) + break; ret = aio_poll(req, iocb); break; default: @@ -1954,13 +2248,21 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, */ if (ret) goto out_put_req; + if (ctx->flags & IOCTX_FLAG_IOPOLL) { + if (test_bit(IOCB_POLL_BUSY, &req->ki_flags)) { + ret = -EAGAIN; + goto out_put_req; + } + aio_iopoll_iocb_issued(req); + } return 0; out_put_req: if (req->ki_eventfd) eventfd_ctx_put(req->ki_eventfd); iocb_put(req); out_put_reqs_available: - put_reqs_available(ctx, 1); + if (!(ctx->flags & IOCTX_FLAG_IOPOLL)) + put_reqs_available(ctx, 1); return ret; } @@ -2136,7 +2438,7 @@ SYSCALL_DEFINE3(io_cancel, aio_context_t, ctx_id, struct iocb __user *, iocb, if (unlikely(!ctx)) return -EINVAL; - if (ctx->flags & IOCTX_FLAG_USERIOCB) + if (ctx->flags & (IOCTX_FLAG_USERIOCB | IOCTX_FLAG_IOPOLL)) goto err; spin_lock_irq(&ctx->ctx_lock); @@ -2171,8 +2473,12 @@ static long do_io_getevents(aio_context_t ctx_id, long ret = -EINVAL; if (likely(ioctx)) { - if (likely(min_nr <= nr && min_nr >= 0)) - ret = read_events(ioctx, min_nr, nr, events, until); + if (likely(min_nr <= nr && min_nr >= 0)) { + if (ioctx->flags & IOCTX_FLAG_IOPOLL) + ret = aio_iopoll_check(ioctx, min_nr, nr, events); + else + ret = read_events(ioctx, min_nr, nr, events, until); + } percpu_ref_put(&ioctx->users); } diff --git a/include/uapi/linux/aio_abi.h b/include/uapi/linux/aio_abi.h index 814e6606c413..ea0b9a19f4df 100644 --- a/include/uapi/linux/aio_abi.h +++ b/include/uapi/linux/aio_abi.h @@ -52,9 +52,11 @@ enum { * is valid. * IOCB_FLAG_IOPRIO - Set if the "aio_reqprio" member of the "struct iocb" * is valid. + * IOCB_FLAG_HIPRI - Use IO completion polling */ #define IOCB_FLAG_RESFD (1 << 0) #define IOCB_FLAG_IOPRIO (1 << 1) +#define IOCB_FLAG_HIPRI (1 << 2) /* read() from /dev/aio returns these structures. */ struct io_event { @@ -107,6 +109,7 @@ struct iocb { }; /* 64 bytes */ #define IOCTX_FLAG_USERIOCB (1 << 0) /* iocbs are user mapped */ +#define IOCTX_FLAG_IOPOLL (1 << 1) /* io_context is polled */ #undef IFBIG #undef IFLITTLE From patchwork Mon Nov 26 16:45:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698717 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E79414E2 for ; Mon, 26 Nov 2018 16:46:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2FA652A083 for ; Mon, 26 Nov 2018 16:46:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 241672A088; Mon, 26 Nov 2018 16:46:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78AD82A087 for ; Mon, 26 Nov 2018 16:46:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726735AbeK0DlE (ORCPT ); Mon, 26 Nov 2018 22:41:04 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:36153 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727140AbeK0DlD (ORCPT ); Mon, 26 Nov 2018 22:41:03 -0500 Received: by mail-it1-f196.google.com with SMTP id c9so28764088itj.1 for ; Mon, 26 Nov 2018 08:46:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OIl8Cn3W6krW4SCjJSLwhD4y1eoA0bru+Cwg1YFlJhk=; b=AIf98ERr1HTugFAQffOuzWX9GOmsqg6bIZwtsiJPOgGmOhk/XaPUDGvfoiz6bZ73Ug H+hi6bKuwiGY0iOWWaK1d3VQ8za1oSx8o6Dhuv4f0z4CxBnoH/h9Nm5NIPc0fsk3SlKZ IJZAVnHwG+GE/+PBwNItBP0rMUgbIn0+3VOLlUKpHiVmEEcado4QeFaJv0b/hc5pZwTF S57y11Y8QYB+wODOjO9yOaUpLmC8vBfo5czeCfjTKikrjXa5UAKXcU+v3dB+TGtu4Jza EuK6g8wwytUjUr4DNUj1KZXk1FukhdCbM2sahTu0ZDXOMmeJKWdvhDt2/4AvYTFQSFRJ pbQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OIl8Cn3W6krW4SCjJSLwhD4y1eoA0bru+Cwg1YFlJhk=; b=sVfmJnStbMFF/0wiV9+PuDTTvxi1UlqBKgSng7TWnJkjCKtYno9iG+pgeYjPNWOBbb T+86wbyoUM+C1AZJf7BnXOhQIi7JjApk8jGDAjHFVuZdZGciliD50TZg9zWP8zA/Ntbb sJ0n453EUDI6Zlkjcljxf0EhN0YnVA0dClKF1YsOVE9vyUpnMfo/zlUlL1zHpuHpvmTp KJwphv5EFfiBNFgxKOYDk551NQJdsAyLl3Qo+KtjCj24kQyQSPUf/UCb/XoeSl+OXn8F xmA+mpGRATo5i/j92ilfVLDj6yyh9dIizAO8kFzOsaStk0VB+CajpRaydak1MHUF9S3o IsEQ== X-Gm-Message-State: AGRZ1gIJuJJYd0gwzNRSM8ny1pUHflVjXHgYunAVJKTRokbFVgz78izs gedN3rVDkdaL/NOOLbbicpA0n/IKI0g= X-Google-Smtp-Source: AFSGD/V2NAJJ8+ykLJhzffrkpsxziOEl9uC+I73XOI19DuejhXfYm6U5UyhrCE1g5Co1cwVE5qOUQw== X-Received: by 2002:a02:2702:: with SMTP id g2mr23484940jaa.83.1543250783563; Mon, 26 Nov 2018 08:46:23 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:22 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 18/20] aio: add submission side request cache Date: Mon, 26 Nov 2018 09:45:42 -0700 Message-Id: <20181126164544.5699-19-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We have to add each submitted polled request to the io_context poll_submitted list, which means we have to grab the poll_lock. We already use the block plug to batch submissions if we're doing a batch of IO submissions, extend that to cover the poll requests internally as well. Signed-off-by: Jens Axboe --- fs/aio.c | 136 +++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 113 insertions(+), 23 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index db73c8af1a0a..74afd1ff7fc9 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -247,6 +247,15 @@ static const struct address_space_operations aio_ctx_aops; static const unsigned int iocb_page_shift = ilog2(PAGE_SIZE / sizeof(struct iocb)); +/* + * We rely on block level unplugs to flush pending requests, if we schedule + */ +#ifdef CONFIG_BLOCK +static const bool aio_use_state_req_list = true; +#else +static const bool aio_use_state_req_list = false; +#endif + static void aio_useriocb_free(struct kioctx *); static void aio_iopoll_reap_events(struct kioctx *); @@ -1724,6 +1733,21 @@ static void aio_complete_rw_poll(struct kiocb *kiocb, long res, long res2) } } +struct aio_submit_state { + struct kioctx *ctx; + + struct blk_plug plug; +#ifdef CONFIG_BLOCK + struct blk_plug_cb plug_cb; +#endif + + /* + * Polled iocbs that have been submitted, but not added to the ctx yet + */ + struct list_head req_list; + unsigned int req_count; +}; + static int aio_prep_rw(struct aio_kiocb *kiocb, const struct iocb *iocb) { struct kioctx *ctx = kiocb->ki_ctx; @@ -1832,13 +1856,28 @@ static inline void aio_rw_done(struct kiocb *req, ssize_t ret) } +/* + * Called either at the end of IO submission, or through a plug callback + * because we're going to schedule. Moves out local batch of requests to + * the ctx poll list, so they can be found for polling + reaping. + */ +static void aio_flush_state_reqs(struct kioctx *ctx, + struct aio_submit_state *state) +{ + spin_lock(&ctx->poll_lock); + list_splice_tail_init(&state->req_list, &ctx->poll_submitted); + spin_unlock(&ctx->poll_lock); + state->req_count = 0; +} + /* * After the iocb has been issued, it's safe to be found on the poll list. * Adding the kiocb to the list AFTER submission ensures that we don't * find it from a io_getevents() thread before the issuer is done accessing * the kiocb cookie. */ -static void aio_iopoll_iocb_issued(struct aio_kiocb *kiocb) +static void aio_iopoll_iocb_issued(struct aio_submit_state *state, + struct aio_kiocb *kiocb) { /* * For fast devices, IO may have already completed. If it has, add @@ -1848,12 +1887,21 @@ static void aio_iopoll_iocb_issued(struct aio_kiocb *kiocb) const int front_add = test_bit(IOCB_POLL_COMPLETED, &kiocb->ki_flags); struct kioctx *ctx = kiocb->ki_ctx; - spin_lock(&ctx->poll_lock); - if (front_add) - list_add(&kiocb->ki_list, &ctx->poll_submitted); - else - list_add_tail(&kiocb->ki_list, &ctx->poll_submitted); - spin_unlock(&ctx->poll_lock); + if (!state || !aio_use_state_req_list) { + spin_lock(&ctx->poll_lock); + if (front_add) + list_add(&kiocb->ki_list, &ctx->poll_submitted); + else + list_add_tail(&kiocb->ki_list, &ctx->poll_submitted); + spin_unlock(&ctx->poll_lock); + } else { + if (front_add) + list_add(&kiocb->ki_list, &state->req_list); + else + list_add_tail(&kiocb->ki_list, &state->req_list); + if (++state->req_count >= AIO_IOPOLL_BATCH) + aio_flush_state_reqs(ctx, state); + } } static ssize_t aio_read(struct aio_kiocb *kiocb, const struct iocb *iocb, @@ -2149,7 +2197,8 @@ static ssize_t aio_poll(struct aio_kiocb *aiocb, const struct iocb *iocb) } static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, - struct iocb __user *user_iocb, bool compat) + struct iocb __user *user_iocb, + struct aio_submit_state *state, bool compat) { struct aio_kiocb *req; ssize_t ret; @@ -2253,7 +2302,7 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, ret = -EAGAIN; goto out_put_req; } - aio_iopoll_iocb_issued(req); + aio_iopoll_iocb_issued(state, req); } return 0; out_put_req: @@ -2267,7 +2316,7 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, } static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, - bool compat) + struct aio_submit_state *state, bool compat) { struct iocb iocb, *iocbp; @@ -2288,7 +2337,44 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, iocbp = &iocb; } - return __io_submit_one(ctx, iocbp, user_iocb, compat); + return __io_submit_one(ctx, iocbp, user_iocb, state, compat); +} + +#ifdef CONFIG_BLOCK +static void aio_state_unplug(struct blk_plug_cb *cb, bool from_schedule) +{ + struct aio_submit_state *state; + + state = container_of(cb, struct aio_submit_state, plug_cb); + if (!list_empty(&state->req_list)) + aio_flush_state_reqs(state->ctx, state); +} +#endif + +/* + * Batched submission is done, ensure local IO is flushed out. + */ +static void aio_submit_state_end(struct aio_submit_state *state) +{ + blk_finish_plug(&state->plug); + if (!list_empty(&state->req_list)) + aio_flush_state_reqs(state->ctx, state); +} + +/* + * Start submission side cache. + */ +static void aio_submit_state_start(struct aio_submit_state *state, + struct kioctx *ctx) +{ + state->ctx = ctx; + INIT_LIST_HEAD(&state->req_list); + state->req_count = 0; +#ifdef CONFIG_BLOCK + state->plug_cb.callback = aio_state_unplug; + blk_start_plug(&state->plug); + list_add(&state->plug_cb.list, &state->plug.cb_list); +#endif } /* @@ -2312,10 +2398,10 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, struct iocb __user * __user *, iocbpp) { + struct aio_submit_state state, *statep = NULL; struct kioctx *ctx; long ret = 0; int i = 0; - struct blk_plug plug; if (unlikely(nr < 0)) return -EINVAL; @@ -2329,8 +2415,10 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, if (nr > ctx->nr_events) nr = ctx->nr_events; - if (nr > AIO_PLUG_THRESHOLD) - blk_start_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) { + aio_submit_state_start(&state, ctx); + statep = &state; + } for (i = 0; i < nr; i++) { struct iocb __user *user_iocb; @@ -2339,12 +2427,12 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, break; } - ret = io_submit_one(ctx, user_iocb, false); + ret = io_submit_one(ctx, user_iocb, statep, false); if (ret) break; } - if (nr > AIO_PLUG_THRESHOLD) - blk_finish_plug(&plug); + if (statep) + aio_submit_state_end(statep); percpu_ref_put(&ctx->users); return i ? i : ret; @@ -2354,10 +2442,10 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, int, nr, compat_uptr_t __user *, iocbpp) { + struct aio_submit_state state, *statep = NULL; struct kioctx *ctx; long ret = 0; int i = 0; - struct blk_plug plug; if (unlikely(nr < 0)) return -EINVAL; @@ -2371,8 +2459,10 @@ COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, if (nr > ctx->nr_events) nr = ctx->nr_events; - if (nr > AIO_PLUG_THRESHOLD) - blk_start_plug(&plug); + if (nr > AIO_PLUG_THRESHOLD) { + aio_submit_state_start(&state, ctx); + statep = &state; + } for (i = 0; i < nr; i++) { compat_uptr_t user_iocb; @@ -2381,12 +2471,12 @@ COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, break; } - ret = io_submit_one(ctx, compat_ptr(user_iocb), true); + ret = io_submit_one(ctx, compat_ptr(user_iocb), statep, true); if (ret) break; } - if (nr > AIO_PLUG_THRESHOLD) - blk_finish_plug(&plug); + if (statep) + aio_submit_state_end(statep); percpu_ref_put(&ctx->users); return i ? i : ret; From patchwork Mon Nov 26 16:45:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBFA517D5 for ; Mon, 26 Nov 2018 16:46:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BDB7C2A083 for ; Mon, 26 Nov 2018 16:46:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B21882A087; Mon, 26 Nov 2018 16:46:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4DA252A08C for ; Mon, 26 Nov 2018 16:46:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727140AbeK0DlG (ORCPT ); Mon, 26 Nov 2018 22:41:06 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:37833 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727131AbeK0DlG (ORCPT ); Mon, 26 Nov 2018 22:41:06 -0500 Received: by mail-it1-f193.google.com with SMTP id b5so28759200iti.2 for ; Mon, 26 Nov 2018 08:46:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=14kCfLiACV93AenWhL1xKvUDPabDTnEKBX7DXRr/kpI=; b=TsiE6o8Eb0baO9E/R/9Y+x1GNi8Ll19OJN+fBlr0JSLeQPh2XnDuYYnmupv0xtiIGy wsb0mw2JZkLxf8ZkQAzgQ7YLL12/4pRFYwm4Y4nPgo4jibzhvUeNfcvC0cKsdUYKhRsz CmALeNil06mvEh9rmSE1+xpBZUzhystU5aUncZdfB3tRDrvptSFOgUaEiiYOR/zlvy+y 2llqHNFHKTY4MSoVpgiFfOHvrx5AEfsQ6dsR7Vq9I6jn5y1udh9ECbsoXfUdwVFYOMwm xNx2eNfYJCmORjkGQPEriidigQKdwDNiz+r0dbZkLGhKcwv/AAFFSpV6W+DK0OeTV8SY 5onw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=14kCfLiACV93AenWhL1xKvUDPabDTnEKBX7DXRr/kpI=; b=o8CMrtxD+IzXj0xoU/U30dmkeDOZVsn7rWn6ZPkp/8/SsVjIgvzxtoLj4t3Sfx2G4B gdtbnKun/Op06y0bcjQaTxxUK1u7rmKKcj1tWM0YkqvuaaUuUbxfKzXb/H9eM8EOTlHt eK0FjE94dHMwiOhwSPHWRYTsg4+ZH8tSNwvsIuhnnRRgwPqSY++PAfd1I7/ziIdXPuUH 7e56NpWV9WZR+YUYsdfC92QpQ4fEkJuGhwx23RK0K2aW0rpUwSov1PJHhHMQ4bzAZF1O DiNxBUtHfUT4FyrO6XY7wd9NSSXj7m+bMOMdr1et3EqG9a5l14+Lty26hpbdBsn3rabd A4ag== X-Gm-Message-State: AGRZ1gI11JUGHN0NxG+78Z6ERAAsr2+XGBZvXrfR/XUAOePCf9SX6KzH 7+P81D/q7vW8TenVfzr3TQegAeXDTDg= X-Google-Smtp-Source: AFSGD/W0/Ihdfel1+3+1K66f2K+U9pBnmMyGYfrv5GLw2DRIQGsnMawLkeCIlfUSbBEvSDaV5SRrBA== X-Received: by 2002:a02:4352:: with SMTP id s79mr23135746jab.124.1543250785780; Mon, 26 Nov 2018 08:46:25 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:24 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 19/20] fs: add fget_many() and fput_many() Date: Mon, 26 Nov 2018 09:45:43 -0700 Message-Id: <20181126164544.5699-20-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some uses cases repeatedly get and put references to the same file, but the only exposed interface is doing these one at the time. As each of these entail an atomic inc or dec on a shared structure, that cost can add up. Add fget_many(), which works just like fget(), except it takes an argument for how many references to get on the file. Ditto fput_many(), which can drop an arbitrary number of references to a file. Signed-off-by: Jens Axboe --- fs/file.c | 15 ++++++++++----- fs/file_table.c | 10 ++++++++-- include/linux/file.h | 2 ++ include/linux/fs.h | 3 ++- 4 files changed, 22 insertions(+), 8 deletions(-) diff --git a/fs/file.c b/fs/file.c index 7ffd6e9d103d..ad9870edfd51 100644 --- a/fs/file.c +++ b/fs/file.c @@ -676,7 +676,7 @@ void do_close_on_exec(struct files_struct *files) spin_unlock(&files->file_lock); } -static struct file *__fget(unsigned int fd, fmode_t mask) +static struct file *__fget(unsigned int fd, fmode_t mask, unsigned int refs) { struct files_struct *files = current->files; struct file *file; @@ -691,7 +691,7 @@ static struct file *__fget(unsigned int fd, fmode_t mask) */ if (file->f_mode & mask) file = NULL; - else if (!get_file_rcu(file)) + else if (!get_file_rcu_many(file, refs)) goto loop; } rcu_read_unlock(); @@ -699,15 +699,20 @@ static struct file *__fget(unsigned int fd, fmode_t mask) return file; } +struct file *fget_many(unsigned int fd, unsigned int refs) +{ + return __fget(fd, FMODE_PATH, refs); +} + struct file *fget(unsigned int fd) { - return __fget(fd, FMODE_PATH); + return fget_many(fd, 1); } EXPORT_SYMBOL(fget); struct file *fget_raw(unsigned int fd) { - return __fget(fd, 0); + return __fget(fd, 0, 1); } EXPORT_SYMBOL(fget_raw); @@ -738,7 +743,7 @@ static unsigned long __fget_light(unsigned int fd, fmode_t mask) return 0; return (unsigned long)file; } else { - file = __fget(fd, mask); + file = __fget(fd, mask, 1); if (!file) return 0; return FDPUT_FPUT | (unsigned long)file; diff --git a/fs/file_table.c b/fs/file_table.c index e49af4caf15d..6a3964df33e4 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -326,9 +326,9 @@ void flush_delayed_fput(void) static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput); -void fput(struct file *file) +void fput_many(struct file *file, unsigned int refs) { - if (atomic_long_dec_and_test(&file->f_count)) { + if (atomic_long_sub_and_test(refs, &file->f_count)) { struct task_struct *task = current; if (likely(!in_interrupt() && !(task->flags & PF_KTHREAD))) { @@ -347,6 +347,12 @@ void fput(struct file *file) } } +void fput(struct file *file) +{ + fput_many(file, 1); +} + + /* * synchronous analog of fput(); for kernel threads that might be needed * in some umount() (and thus can't use flush_delayed_fput() without diff --git a/include/linux/file.h b/include/linux/file.h index 6b2fb032416c..3fcddff56bc4 100644 --- a/include/linux/file.h +++ b/include/linux/file.h @@ -13,6 +13,7 @@ struct file; extern void fput(struct file *); +extern void fput_many(struct file *, unsigned int); struct file_operations; struct vfsmount; @@ -44,6 +45,7 @@ static inline void fdput(struct fd fd) } extern struct file *fget(unsigned int fd); +extern struct file *fget_many(unsigned int fd, unsigned int refs); extern struct file *fget_raw(unsigned int fd); extern unsigned long __fdget(unsigned int fd); extern unsigned long __fdget_raw(unsigned int fd); diff --git a/include/linux/fs.h b/include/linux/fs.h index 6a5f71f8ae06..dc54a65c401a 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -952,7 +952,8 @@ static inline struct file *get_file(struct file *f) atomic_long_inc(&f->f_count); return f; } -#define get_file_rcu(x) atomic_long_inc_not_zero(&(x)->f_count) +#define get_file_rcu_many(x, cnt) atomic_long_add_unless(&(x)->f_count, (cnt), 0) +#define get_file_rcu(x) get_file_rcu_many((x), 1) #define fput_atomic(x) atomic_long_add_unless(&(x)->f_count, -1, 1) #define file_count(x) atomic_long_read(&(x)->f_count) From patchwork Mon Nov 26 16:45:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698725 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22ACD14E2 for ; Mon, 26 Nov 2018 16:46:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 13C972A083 for ; Mon, 26 Nov 2018 16:46:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 082392A087; Mon, 26 Nov 2018 16:46:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAD862A091 for ; Mon, 26 Nov 2018 16:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727131AbeK0DlI (ORCPT ); Mon, 26 Nov 2018 22:41:08 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:56070 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727162AbeK0DlI (ORCPT ); Mon, 26 Nov 2018 22:41:08 -0500 Received: by mail-it1-f196.google.com with SMTP id o19so29266588itg.5 for ; Mon, 26 Nov 2018 08:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RCnQX7quqlyYlttykNa4HgMwskCdrDExoGKPcgxpFHQ=; b=c3hb1E+d9QXzBRs/Vj20QkoV9a6R5TPpCc3LvYRgxqa/b1WL/DmlHSgSRuz+OgWOBI AxOBJhpvT04GLGzgveDIrI7STem15VqeTqEuXOC4HMe8lSim0HsRIIkmio/GExXzv+8R hfGXWJq/VzWI7TJVt/y4po3VQxFx4+4PuObceB2vgwR+nm43+dmE9DId0mTOsSJjTfe/ 2V5DDU7cOyD0KrUNMIUJTS3R6LG0MIrrwWVLKUlJELIFeNH62bK1fyV8U95CgIrwJkvE zMzg7jvcT1OsjEWH7F/7MutzBba8BnOfvhP+iKaJYLPcOhDhRX86YZea6jK84dCbba+j DOHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RCnQX7quqlyYlttykNa4HgMwskCdrDExoGKPcgxpFHQ=; b=D/Fdn9d3ehImN65nxkU6gpFgW81fMb0QxAhUhOf+KpGNFxyvieAKHvx6Z/DKxBlwx1 zNoQ7t43AcaRvJYGYZzV9l1YfyBCOVM3p7Gg81ItzeNDuSFE1GoHBCf62W1LtWhlREDq wwJ6bzU3nYlc1MkJBq81Xb1JuusHX/dZM9m5QAlTKVmV4EEL5BT6d+FxCFQkxQbTxODX QwFQZD8F4kxbe6ZxZ7O1hejoYySvtpXZ8GmldOjgCAE7bP5uTYUTbsnaSzuJxV9S7LAU ahpkwyjKOT4W5ePqWQKMVhEKdfBR4nLpeUQzg/bxy2rwExsG3wgfBipvutiPT4C786sA nZGg== X-Gm-Message-State: AGRZ1gK4oWtMppAuVxJHy1zGWCVX9EsxdptzqBbheDLVHsiDDiub2+UP 2fTq5N81oN/aP7XGCMf2mWpZHLlRX48= X-Google-Smtp-Source: AFSGD/US153htJbrsIIWy9GAZvqCAYZoSyBARs9fAkvhkkVg8o5YYyPdMiCtUW7VHdxoHTYPsec9PA== X-Received: by 2002:a05:660c:fd2:: with SMTP id m18mr24374071itn.1.1543250787889; Mon, 26 Nov 2018 08:46:27 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t65-v6sm486801ita.9.2018.11.26.08.46.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:46:26 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 20/20] aio: use fget/fput_many() for file references Date: Mon, 26 Nov 2018 09:45:44 -0700 Message-Id: <20181126164544.5699-21-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126164544.5699-1-axboe@kernel.dk> References: <20181126164544.5699-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On the submission side, add file reference batching to the aio_submit_state. We get as many references as the number of iocbs we are submitting, and drop unused ones if we end up switching files. The assumption here is that we're usually only dealing with one fd, and if there are multiple, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed. On the completion side we do the same thing, except this is trivially done just locally in aio_iopoll_reap(). Signed-off-by: Jens Axboe --- fs/aio.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 91 insertions(+), 15 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 74afd1ff7fc9..70d96c2e0579 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1329,7 +1329,8 @@ static long aio_iopoll_reap(struct kioctx *ctx, struct io_event __user *evs, { void *iocbs[AIO_IOPOLL_BATCH]; struct aio_kiocb *iocb, *n; - int to_free = 0, ret = 0; + int file_count, to_free = 0, ret = 0; + struct file *file = NULL; list_for_each_entry_safe(iocb, n, &ctx->poll_completing, ki_list) { if (*nr_events == max) @@ -1342,7 +1343,20 @@ static long aio_iopoll_reap(struct kioctx *ctx, struct io_event __user *evs, list_del(&iocb->ki_list); iocbs[to_free++] = iocb; - fput(iocb->rw.ki_filp); + /* + * Batched puts of the same file, to avoid dirtying the + * file usage count multiple times, if avoidable. + */ + if (!file) { + file = iocb->rw.ki_filp; + file_count = 1; + } else if (file == iocb->rw.ki_filp) { + file_count++; + } else { + fput_many(file, file_count); + file = iocb->rw.ki_filp; + file_count = 1; + } if (evs && copy_to_user(evs + *nr_events, &iocb->ki_ev, sizeof(iocb->ki_ev))) { @@ -1352,6 +1366,9 @@ static long aio_iopoll_reap(struct kioctx *ctx, struct io_event __user *evs, (*nr_events)++; } + if (file) + fput_many(file, file_count); + if (to_free) iocb_put_many(ctx, iocbs, &to_free); @@ -1746,15 +1763,69 @@ struct aio_submit_state { */ struct list_head req_list; unsigned int req_count; + + /* + * File reference cache + */ + struct file *file; + unsigned int fd; + unsigned int has_refs; + unsigned int used_refs; + unsigned int ios_left; }; -static int aio_prep_rw(struct aio_kiocb *kiocb, const struct iocb *iocb) +static void aio_file_put(struct aio_submit_state *state) +{ + if (state->file) { + int diff = state->has_refs - state->used_refs; + + if (diff) + fput_many(state->file, diff); + state->file = NULL; + } +} + +/* + * Get as many references to a file as we have IOs left in this submission, + * assuming most submissions are for one file, or at least that each file + * has more than one submission. + */ +static struct file *aio_file_get(struct aio_submit_state *state, int fd) +{ + if (!state) + return fget(fd); + + if (!state->file) { +get_file: + state->file = fget_many(fd, state->ios_left); + if (!state->file) + return NULL; + + state->fd = fd; + state->has_refs = state->ios_left; + state->used_refs = 1; + state->ios_left--; + return state->file; + } + + if (state->fd == fd) { + state->used_refs++; + state->ios_left--; + return state->file; + } + + aio_file_put(state); + goto get_file; +} + +static int aio_prep_rw(struct aio_kiocb *kiocb, const struct iocb *iocb, + struct aio_submit_state *state) { struct kioctx *ctx = kiocb->ki_ctx; struct kiocb *req = &kiocb->rw; int ret; - req->ki_filp = fget(iocb->aio_fildes); + req->ki_filp = aio_file_get(state, iocb->aio_fildes); if (unlikely(!req->ki_filp)) return -EBADF; req->ki_pos = iocb->aio_offset; @@ -1905,7 +1976,8 @@ static void aio_iopoll_iocb_issued(struct aio_submit_state *state, } static ssize_t aio_read(struct aio_kiocb *kiocb, const struct iocb *iocb, - bool vectored, bool compat) + struct aio_submit_state *state, bool vectored, + bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; struct kiocb *req = &kiocb->rw; @@ -1913,7 +1985,7 @@ static ssize_t aio_read(struct aio_kiocb *kiocb, const struct iocb *iocb, struct file *file; ssize_t ret; - ret = aio_prep_rw(kiocb, iocb); + ret = aio_prep_rw(kiocb, iocb, state); if (ret) return ret; file = req->ki_filp; @@ -1939,7 +2011,8 @@ static ssize_t aio_read(struct aio_kiocb *kiocb, const struct iocb *iocb, } static ssize_t aio_write(struct aio_kiocb *kiocb, const struct iocb *iocb, - bool vectored, bool compat) + struct aio_submit_state *state, bool vectored, + bool compat) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; struct kiocb *req = &kiocb->rw; @@ -1947,7 +2020,7 @@ static ssize_t aio_write(struct aio_kiocb *kiocb, const struct iocb *iocb, struct file *file; ssize_t ret; - ret = aio_prep_rw(kiocb, iocb); + ret = aio_prep_rw(kiocb, iocb, state); if (ret) return ret; file = req->ki_filp; @@ -2258,16 +2331,16 @@ static int __io_submit_one(struct kioctx *ctx, const struct iocb *iocb, ret = -EINVAL; switch (iocb->aio_lio_opcode) { case IOCB_CMD_PREAD: - ret = aio_read(req, iocb, false, compat); + ret = aio_read(req, iocb, state, false, compat); break; case IOCB_CMD_PWRITE: - ret = aio_write(req, iocb, false, compat); + ret = aio_write(req, iocb, state, false, compat); break; case IOCB_CMD_PREADV: - ret = aio_read(req, iocb, true, compat); + ret = aio_read(req, iocb, state, true, compat); break; case IOCB_CMD_PWRITEV: - ret = aio_write(req, iocb, true, compat); + ret = aio_write(req, iocb, state, true, compat); break; case IOCB_CMD_FSYNC: if (ctx->flags & IOCTX_FLAG_IOPOLL) @@ -2359,17 +2432,20 @@ static void aio_submit_state_end(struct aio_submit_state *state) blk_finish_plug(&state->plug); if (!list_empty(&state->req_list)) aio_flush_state_reqs(state->ctx, state); + aio_file_put(state); } /* * Start submission side cache. */ static void aio_submit_state_start(struct aio_submit_state *state, - struct kioctx *ctx) + struct kioctx *ctx, int max_ios) { state->ctx = ctx; INIT_LIST_HEAD(&state->req_list); state->req_count = 0; + state->file = NULL; + state->ios_left = max_ios; #ifdef CONFIG_BLOCK state->plug_cb.callback = aio_state_unplug; blk_start_plug(&state->plug); @@ -2416,7 +2492,7 @@ SYSCALL_DEFINE3(io_submit, aio_context_t, ctx_id, long, nr, nr = ctx->nr_events; if (nr > AIO_PLUG_THRESHOLD) { - aio_submit_state_start(&state, ctx); + aio_submit_state_start(&state, ctx, nr); statep = &state; } for (i = 0; i < nr; i++) { @@ -2460,7 +2536,7 @@ COMPAT_SYSCALL_DEFINE3(io_submit, compat_aio_context_t, ctx_id, nr = ctx->nr_events; if (nr > AIO_PLUG_THRESHOLD) { - aio_submit_state_start(&state, ctx); + aio_submit_state_start(&state, ctx, nr); statep = &state; } for (i = 0; i < nr; i++) {