From patchwork Mon Jul 30 07:15:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10548551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E55BA754 for ; Mon, 30 Jul 2018 07:16:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79DC929741 for ; Mon, 30 Jul 2018 07:16:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6E92629755; Mon, 30 Jul 2018 07:16:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22BC929741 for ; Mon, 30 Jul 2018 07:16:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726696AbeG3Itc (ORCPT ); Mon, 30 Jul 2018 04:49:32 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:33086 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726344AbeG3Itb (ORCPT ); Mon, 30 Jul 2018 04:49:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ALI25aMQxkllk3JebdVfnhKX+zJBphTEhy43gAvxda0=; b=jD3g3Rons9+GdqQWaktHyiYBJ 0g3z5M22HzdmNh/Ye7lhXL/gphIKnw68MgJJBHUILwSG/k+SBAoAPmkbIYuYOnY/fmB6A3sMn899v U3icotnbgrxq9khKSIHOZUhWSFSNwbh7PSpvCB6CNcti4w7iwSFraT6HUjidQMQA1GRUYb7LPTICB abcL/q7Rskimc5+QEtFqb6x34Xjmjrv2BqABXpS7Tjrs4BxvUkp4HaVpvBsz7a/LeysJdM+dXdAbN 8VbT9TIVHSOC27JfTVpJs/GNCtZ9LXJRXkeJu2LYJtcrrBgGVLJY6PPIT9IQBp1NsRnkB2VgtCqzW xwAuhDebA==; Received: from 089144213232.atnat0022.highway.a1.net ([89.144.213.232] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fk2PL-0008Ij-88; Mon, 30 Jul 2018 07:15:55 +0000 From: Christoph Hellwig To: viro@zeniv.linux.org.uk Cc: Avi Kivity , linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] timerfd: add support for keyed wakeups Date: Mon, 30 Jul 2018 09:15:41 +0200 Message-Id: <20180730071544.23998-2-hch@lst.de> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180730071544.23998-1-hch@lst.de> References: <20180730071544.23998-1-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This prepares timerfd for use with aio poll. Signed-off-by: Christoph Hellwig --- fs/timerfd.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/fs/timerfd.c b/fs/timerfd.c index cdad49da3ff7..f6c54fd56645 100644 --- a/fs/timerfd.c +++ b/fs/timerfd.c @@ -66,7 +66,7 @@ static void timerfd_triggered(struct timerfd_ctx *ctx) spin_lock_irqsave(&ctx->wqh.lock, flags); ctx->expired = 1; ctx->ticks++; - wake_up_locked(&ctx->wqh); + wake_up_locked_poll(&ctx->wqh, EPOLLIN); spin_unlock_irqrestore(&ctx->wqh.lock, flags); } @@ -107,7 +107,7 @@ void timerfd_clock_was_set(void) if (ctx->moffs != moffs) { ctx->moffs = KTIME_MAX; ctx->ticks++; - wake_up_locked(&ctx->wqh); + wake_up_locked_poll(&ctx->wqh, EPOLLIN); } spin_unlock_irqrestore(&ctx->wqh.lock, flags); } @@ -345,7 +345,7 @@ static long timerfd_ioctl(struct file *file, unsigned int cmd, unsigned long arg spin_lock_irq(&ctx->wqh.lock); if (!timerfd_canceled(ctx)) { ctx->ticks = ticks; - wake_up_locked(&ctx->wqh); + wake_up_locked_poll(&ctx->wqh, EPOLLIN); } else ret = -ECANCELED; spin_unlock_irq(&ctx->wqh.lock); From patchwork Mon Jul 30 07:15:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10548549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 709B4A754 for ; Mon, 30 Jul 2018 07:16:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D49529741 for ; Mon, 30 Jul 2018 07:16:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 51BB529755; Mon, 30 Jul 2018 07:16:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC8F629741 for ; Mon, 30 Jul 2018 07:16:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726767AbeG3Itf (ORCPT ); Mon, 30 Jul 2018 04:49:35 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:33100 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726366AbeG3Itf (ORCPT ); Mon, 30 Jul 2018 04:49:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BWtMZ2W9plgEcpyRGLCAFtPZ5rLA8SWc74nPm8axhjY=; b=aGpfAAmjxP0hHWIscuewssazE LfVx4vMt9vWqYkraB0Ls5OkC7ArV4DE5dpy+0dHUMtIQ4mqJ8/Gr5obj8JO79DTVDACvzQ7Z1Gwlx +SmstgRijILTFKQAdTvSEJeVeiCwdBkPgGdjbnuggyQcuP+jL3wTEHTmMt19BHLdzx+DbOQ3+lNhi bPOYY1nCjwpaB9u99HSZxkxMgNHny95+JNNjQvuWrMFe0Qby6gtg9UjWd1lzAW3tFM4btZ9HujpgW ODfpnxHAxXafoE4aZ55l1Vxs1SyIlqO8EQ5q4g4P4tjNuhRIqgfiT8flE9EJIziGzzvKXpfjGmk34 V/rzmPmsQ==; Received: from 089144213232.atnat0022.highway.a1.net ([89.144.213.232] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fk2PP-0000Dn-57; Mon, 30 Jul 2018 07:15:59 +0000 From: Christoph Hellwig To: viro@zeniv.linux.org.uk Cc: Avi Kivity , linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] aio: add a iocb refcount Date: Mon, 30 Jul 2018 09:15:42 +0200 Message-Id: <20180730071544.23998-3-hch@lst.de> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180730071544.23998-1-hch@lst.de> References: <20180730071544.23998-1-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is needed to prevent races caused by the way the ->poll API works. To avoid introducing overhead for other users of the iocbs we initialize it to zero and only do refcount operations if it is non-zero in the completion path. Signed-off-by: Christoph Hellwig --- fs/aio.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 27454594e37a..fe2018ada32c 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -178,6 +179,7 @@ struct aio_kiocb { struct list_head ki_list; /* the aio core uses this * for cancellation */ + refcount_t ki_refcnt; /* * If the aio_resfd field of the userspace iocb is not zero, @@ -1015,6 +1017,7 @@ static inline struct aio_kiocb *aio_get_req(struct kioctx *ctx) percpu_ref_get(&ctx->reqs); INIT_LIST_HEAD(&req->ki_list); + refcount_set(&req->ki_refcnt, 0); req->ki_ctx = ctx; return req; out_put: @@ -1049,6 +1052,15 @@ static struct kioctx *lookup_ioctx(unsigned long ctx_id) return ret; } +static inline void iocb_put(struct aio_kiocb *iocb) +{ + if (refcount_read(&iocb->ki_refcnt) == 0 || + refcount_dec_and_test(&iocb->ki_refcnt)) { + percpu_ref_put(&iocb->ki_ctx->reqs); + kmem_cache_free(kiocb_cachep, iocb); + } +} + /* aio_complete * Called when the io request on the given iocb is complete. */ @@ -1118,8 +1130,6 @@ static void aio_complete(struct aio_kiocb *iocb, long res, long res2) eventfd_ctx_put(iocb->ki_eventfd); } - kmem_cache_free(kiocb_cachep, iocb); - /* * We have to order our ring_info tail store above and test * of the wait list below outside the wait lock. This is @@ -1130,8 +1140,7 @@ static void aio_complete(struct aio_kiocb *iocb, long res, long res2) if (waitqueue_active(&ctx->wait)) wake_up(&ctx->wait); - - percpu_ref_put(&ctx->reqs); + iocb_put(iocb); } /* aio_read_events_ring From patchwork Mon Jul 30 07:15:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10548545 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 082D7139A for ; Mon, 30 Jul 2018 07:16:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E710329741 for ; Mon, 30 Jul 2018 07:16:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DB4CB29755; Mon, 30 Jul 2018 07:16:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED7EE29741 for ; Mon, 30 Jul 2018 07:16:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726801AbeG3Itk (ORCPT ); Mon, 30 Jul 2018 04:49:40 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:33130 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726797AbeG3Itk (ORCPT ); Mon, 30 Jul 2018 04:49:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=TfP5yvZ5jC9Yb2ceEXEId7IzaRB7qOmxYe/TYXpTfe8=; b=WDTUUQp0cwpJdsR8tUj6S12/b QGCD3yGKJSnIyYL6F6u9CW6emGRvHAtV5MWLP/dZodkQY6lhJAF6YBB5yXMWwoSfYV3r4tVbVKeYn xeIvdLRnrh+rM9jDOEjnvD2731c1FcJlIYT1C5jZhr4fQJU9I8zOEn0DhGk0rwJP4Ycmof4m8SVBy OaMIY4qHHfUl5DSV9tESJx5STcBf9eOF6QtAkeRCC7pPL2PxhEuAFo9c+3pVW3qRr67lH8vXkgRw+ DOz/zPorFvxx9ZXCQvAT4fo7ZY3ibwgdP204T0GZGbYmxpZmizu8yEr+96K+upicrkIz8tqKD+xkb nuky17M5Q==; Received: from 089144213232.atnat0022.highway.a1.net ([89.144.213.232] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fk2PT-0000gt-Of; Mon, 30 Jul 2018 07:16:04 +0000 From: Christoph Hellwig To: viro@zeniv.linux.org.uk Cc: Avi Kivity , linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] aio: implement IOCB_CMD_POLL Date: Mon, 30 Jul 2018 09:15:43 +0200 Message-Id: <20180730071544.23998-4-hch@lst.de> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180730071544.23998-1-hch@lst.de> References: <20180730071544.23998-1-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Simple one-shot poll through the io_submit() interface. To poll for a file descriptor the application should submit an iocb of type IOCB_CMD_POLL. It will poll the fd for the events specified in the the first 32 bits of the aio_buf field of the iocb. Unlike poll or epoll without EPOLLONESHOT this interface always works in one shot mode, that is once the iocb is completed, it will have to be resubmitted. Signed-off-by: Christoph Hellwig --- fs/aio.c | 178 +++++++++++++++++++++++++++++++++++ include/uapi/linux/aio_abi.h | 6 +- 2 files changed, 180 insertions(+), 4 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index fe2018ada32c..6993684d0665 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -5,6 +5,7 @@ * Implements an efficient asynchronous io interface. * * Copyright 2000, 2001, 2002 Red Hat, Inc. All Rights Reserved. + * Copyright 2018 Christoph Hellwig. * * See ../COPYING for licensing terms. */ @@ -165,10 +166,21 @@ struct fsync_iocb { bool datasync; }; +struct poll_iocb { + struct file *file; + struct wait_queue_head *head; + __poll_t events; + bool cancelled; + bool done; + struct wait_queue_entry wait; + struct work_struct work; +}; + struct aio_kiocb { union { struct kiocb rw; struct fsync_iocb fsync; + struct poll_iocb poll; }; struct kioctx *ki_ctx; @@ -1601,6 +1613,169 @@ static int aio_fsync(struct fsync_iocb *req, struct iocb *iocb, bool datasync) return 0; } +static inline void aio_poll_complete(struct aio_kiocb *iocb, __poll_t mask) +{ + struct file *file = iocb->poll.file; + + aio_complete(iocb, mangle_poll(mask), 0); + fput(file); +} + +static void aio_poll_complete_work(struct work_struct *work) +{ + struct poll_iocb *req = container_of(work, struct poll_iocb, work); + struct aio_kiocb *iocb = container_of(req, struct aio_kiocb, poll); + struct poll_table_struct pt = { ._key = req->events }; + struct kioctx *ctx = iocb->ki_ctx; + __poll_t mask; + + if (READ_ONCE(req->cancelled)) { + /* synchronize with ki_list removal in the callers: */ + spin_lock_irq(&ctx->ctx_lock); + WARN_ON_ONCE(!list_empty(&iocb->ki_list)); + spin_unlock_irq(&ctx->ctx_lock); + + aio_poll_complete(iocb, 0); + return; + } + + mask = vfs_poll(req->file, &pt) & req->events; + if (!mask) { + add_wait_queue(req->head, &req->wait); + return; + } + + spin_lock_irq(&ctx->ctx_lock); + req->done = true; + list_del(&iocb->ki_list); + spin_unlock_irq(&ctx->ctx_lock); + + aio_poll_complete(iocb, mask); +} + +/* assumes we are called with irqs disabled */ +static int aio_poll_cancel(struct kiocb *iocb) +{ + struct aio_kiocb *aiocb = container_of(iocb, struct aio_kiocb, rw); + struct poll_iocb *req = &aiocb->poll; + + spin_lock(&req->head->lock); + if (!list_empty(&req->wait.entry)) { + WRITE_ONCE(req->cancelled, true); + list_del_init(&req->wait.entry); + schedule_work(&aiocb->poll.work); + } + spin_unlock(&req->head->lock); + + return 0; +} + +static int aio_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, + void *key) +{ + struct poll_iocb *req = container_of(wait, struct poll_iocb, wait); + __poll_t mask = key_to_poll(key); + + /* for instances that support it check for an event match first: */ + if (mask && !(mask & req->events)) + return 0; + + list_del_init(&req->wait.entry); + schedule_work(&req->work); + return 1; +} + +struct aio_poll_table { + struct poll_table_struct pt; + struct aio_kiocb *iocb; + int error; +}; + +static void +aio_poll_queue_proc(struct file *file, struct wait_queue_head *head, + struct poll_table_struct *p) +{ + struct aio_poll_table *pt = container_of(p, struct aio_poll_table, pt); + + /* multiple wait queues per file are not supported */ + if (unlikely(pt->iocb->poll.head)) { + pt->error = -EINVAL; + return; + } + + pt->error = 0; + pt->iocb->poll.head = head; + add_wait_queue(head, &pt->iocb->poll.wait); +} + +static ssize_t aio_poll(struct aio_kiocb *aiocb, struct iocb *iocb) +{ + struct kioctx *ctx = aiocb->ki_ctx; + struct poll_iocb *req = &aiocb->poll; + struct aio_poll_table apt; + __poll_t mask; + + /* reject any unknown events outside the normal event mask. */ + if ((u16)iocb->aio_buf != iocb->aio_buf) + return -EINVAL; + /* reject fields that are not defined for poll */ + if (iocb->aio_offset || iocb->aio_nbytes || iocb->aio_rw_flags) + return -EINVAL; + + INIT_WORK(&req->work, aio_poll_complete_work); + req->events = demangle_poll(iocb->aio_buf) | EPOLLERR | EPOLLHUP; + req->file = fget(iocb->aio_fildes); + if (unlikely(!req->file)) + return -EBADF; + + apt.pt._qproc = aio_poll_queue_proc; + apt.pt._key = req->events; + apt.iocb = aiocb; + apt.error = -EINVAL; /* same as no support for IOCB_CMD_POLL */ + + /* initialized the list so that we can do list_empty checks */ + INIT_LIST_HEAD(&req->wait.entry); + init_waitqueue_func_entry(&req->wait, aio_poll_wake); + + /* one for removal from waitqueue, one for this function */ + refcount_set(&aiocb->ki_refcnt, 2); + + mask = vfs_poll(req->file, &apt.pt) & req->events; + if (mask || apt.error) { + bool removed = false; + + /* we did not manage to set up a waitqueue, done */ + if (unlikely(!req->head)) + goto out_fput; + + spin_lock_irq(&req->head->lock); + if (!list_empty(&req->wait.entry)) { + list_del_init(&req->wait.entry); + removed = true; + } + spin_unlock_irq(&req->head->lock); + + if (removed) { + if (apt.error) + goto out_fput; + aio_poll_complete(aiocb, mask); + } + } else { + spin_lock_irq(&ctx->ctx_lock); + if (!req->done) { + list_add_tail(&aiocb->ki_list, &ctx->active_reqs); + aiocb->ki_cancel = aio_poll_cancel; + } + spin_unlock_irq(&ctx->ctx_lock); + } + + iocb_put(aiocb); + return 0; +out_fput: + fput(req->file); + return apt.error; +} + static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, bool compat) { @@ -1674,6 +1849,9 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb, case IOCB_CMD_FDSYNC: ret = aio_fsync(&req->fsync, &iocb, true); break; + case IOCB_CMD_POLL: + ret = aio_poll(req, &iocb); + break; default: pr_debug("invalid aio operation %d\n", iocb.aio_lio_opcode); ret = -EINVAL; diff --git a/include/uapi/linux/aio_abi.h b/include/uapi/linux/aio_abi.h index d4593a6062ef..ce43d340f010 100644 --- a/include/uapi/linux/aio_abi.h +++ b/include/uapi/linux/aio_abi.h @@ -38,10 +38,8 @@ enum { IOCB_CMD_PWRITE = 1, IOCB_CMD_FSYNC = 2, IOCB_CMD_FDSYNC = 3, - /* These two are experimental. - * IOCB_CMD_PREADX = 4, - * IOCB_CMD_POLL = 5, - */ + /* 4 was the experimental IOCB_CMD_PREADX */ + IOCB_CMD_POLL = 5, IOCB_CMD_NOOP = 6, IOCB_CMD_PREADV = 7, IOCB_CMD_PWRITEV = 8, From patchwork Mon Jul 30 07:15:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10548547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95F8DA754 for ; Mon, 30 Jul 2018 07:16:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 81F6629741 for ; Mon, 30 Jul 2018 07:16:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 766AD29755; Mon, 30 Jul 2018 07:16:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04E5D29741 for ; Mon, 30 Jul 2018 07:16:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726852AbeG3Itr (ORCPT ); Mon, 30 Jul 2018 04:49:47 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:33468 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726797AbeG3Itq (ORCPT ); Mon, 30 Jul 2018 04:49:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=zVifwcnZBWlJrKiu7x5RvybRHcPmPcZJmynAIWjhYB4=; b=eS9l7v0Ar9E9nrOeDZNDOqzKo zNEePtCgRnJahL3QpJVtpWFNEwJfSmhVEApwb8On8Kqt4YYV3wapApaCZn3p1vbbhfLFlUoDb0EP8 z7rh4unEPnf0gIQk0UotihjROX5DrKrJHcKvkHc0STWKK75XOrErBHvmAEtIgQZ1zNDPjqnD/YIDD D9swTSYRCzD6bEwjcEi0RKCQOVb0uCk/DiUMhiiOIKHjCA6FR6CG0HNHQoy532KQJSNqff26toIQS At76g50I+H4whepee2clqL2loU4JMMVSkOKbFpoNrpfecYMT+ZE+a6YKji222h9ecc0qOCJOlcx+b fImkMR0Cw==; Received: from 089144213232.atnat0022.highway.a1.net ([89.144.213.232] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fk2PZ-0001KO-5k; Mon, 30 Jul 2018 07:16:09 +0000 From: Christoph Hellwig To: viro@zeniv.linux.org.uk Cc: Avi Kivity , linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] aio: allow direct aio poll comletions for keyed wakeups Date: Mon, 30 Jul 2018 09:15:44 +0200 Message-Id: <20180730071544.23998-5-hch@lst.de> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180730071544.23998-1-hch@lst.de> References: <20180730071544.23998-1-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we get a keyed wakeup for a aio poll waitqueue and wake can acquire the ctx_lock without spinning we can just complete the iocb straight from the wakeup callback to avoid a context switch. Signed-off-by: Christoph Hellwig --- fs/aio.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 6993684d0665..158c5e41b17c 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1674,11 +1674,25 @@ static int aio_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, void *key) { struct poll_iocb *req = container_of(wait, struct poll_iocb, wait); + struct aio_kiocb *iocb = container_of(req, struct aio_kiocb, poll); __poll_t mask = key_to_poll(key); /* for instances that support it check for an event match first: */ - if (mask && !(mask & req->events)) - return 0; + if (mask) { + if (!(mask & req->events)) + return 0; + + /* try to complete the iocb inline if we can: */ + if (spin_trylock(&iocb->ki_ctx->ctx_lock)) { + req->done = true; + list_del(&iocb->ki_list); + spin_unlock(&iocb->ki_ctx->ctx_lock); + + list_del_init(&req->wait.entry); + aio_poll_complete(iocb, mask); + return 1; + } + } list_del_init(&req->wait.entry); schedule_work(&req->work);