From patchwork Fri Apr 19 09:57:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Stefan_B=C3=BChler?= X-Patchwork-Id: 10909517 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB9551515 for ; Fri, 19 Apr 2019 18:49:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1DDB28DCF for ; Fri, 19 Apr 2019 18:49:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9617D28DD4; Fri, 19 Apr 2019 18:49:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_INVALID,DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 40C1C28DCF for ; Fri, 19 Apr 2019 18:49:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728454AbfDSStU (ORCPT ); Fri, 19 Apr 2019 14:49:20 -0400 Received: from mail.stbuehler.de ([5.9.32.208]:51614 "EHLO mail.stbuehler.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728448AbfDSStT (ORCPT ); Fri, 19 Apr 2019 14:49:19 -0400 Received: from chromobil.fritz.box (unknown [IPv6:2a02:8070:a29c:5000:823f:5dff:fe0f:b5b6]) by mail.stbuehler.de (Postfix) with ESMTPSA id 475A8C02FFB; Fri, 19 Apr 2019 09:57:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=stbuehler.de; s=stbuehler1; t=1555667867; bh=E+pBak04zce3NEXwB6UZ70CJv2w7yxoGBE8YB3hIvbc=; h=From:To:Subject:Date:In-Reply-To:References:From; b=UkxfDmmaBlMsD+SvMH6htxk7T+F5toeN5ZN0zoSXu/FBpS60woRYbiFmw6eWlJ/Ar U/bZhzmg/6B3mPkEfERFnar9ySsFeLHT0L6fpHFtajil5Vzg49H98Qa3lSGvmQUM+l tEybbw5yzCjxMpjeQvFwmcU/Sf2TCY4IFDJ4tkQQ= From: =?utf-8?q?Stefan_B=C3=BChler?= To: Jens Axboe , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v1 1/3] io_uring: fix race condition reading SQ entries Date: Fri, 19 Apr 2019 11:57:44 +0200 Message-Id: <20190419095746.22894-1-source@stbuehler.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <54496e17-97de-9f9a-9972-c448226bb768@stbuehler.de> References: <54496e17-97de-9f9a-9972-c448226bb768@stbuehler.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A read memory barrier is required between reading SQ tail and reading the actual data belonging to the SQ entry. Userspace needs a matching write barrier between writing SQ entries and updating SQ tail (using smp_store_release to update tail will do). Signed-off-by: Stefan Bühler --- fs/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index f65f85d89217..96863e4780b7 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1739,7 +1739,8 @@ static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s) head = ctx->cached_sq_head; /* See comment at the top of this file */ smp_rmb(); - if (head == READ_ONCE(ring->r.tail)) + /* make sure SQ entry isn't read before tail */ + if (head == smp_load_acquire(&ring->r.tail)) return false; head = READ_ONCE(ring->array[head & ctx->sq_mask]); From patchwork Fri Apr 19 09:57:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Stefan_B=C3=BChler?= X-Patchwork-Id: 10909509 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2779417E0 for ; Fri, 19 Apr 2019 18:49:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C14028DCD for ; Fri, 19 Apr 2019 18:49:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1071128DCF; Fri, 19 Apr 2019 18:49:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_INVALID,DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A851B28DCD for ; Fri, 19 Apr 2019 18:49:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728023AbfDSStU (ORCPT ); Fri, 19 Apr 2019 14:49:20 -0400 Received: from mail.stbuehler.de ([5.9.32.208]:51616 "EHLO mail.stbuehler.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728454AbfDSStU (ORCPT ); Fri, 19 Apr 2019 14:49:20 -0400 Received: from chromobil.fritz.box (unknown [IPv6:2a02:8070:a29c:5000:823f:5dff:fe0f:b5b6]) by mail.stbuehler.de (Postfix) with ESMTPSA id 90FC1C03099; Fri, 19 Apr 2019 09:57:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=stbuehler.de; s=stbuehler1; t=1555667867; bh=Ny3nPJ9Jd45yAaHUdAjuhmCfTy/7e8qTCdVb/ocHtok=; h=From:To:Subject:Date:In-Reply-To:References:From; b=V1dBgANQtdwhME2WNGM6MvB0u/KB1TcqIRkduhtHFpSzS/HMAdnHGkLooWMICtmi8 l7Bk6uzzfykWOMuVW2WDZ6nXloQYhfcpBStnA4I8o5M843Cxxjdw0NSF0zGk44Qqsy XQ6cymlnR0f6YtJDtgvMVekGmHGAEPU1mTTCntaE= From: =?utf-8?q?Stefan_B=C3=BChler?= To: Jens Axboe , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v1 2/3] io_uring: fix race condition when sq threads goes sleeping Date: Fri, 19 Apr 2019 11:57:45 +0200 Message-Id: <20190419095746.22894-2-source@stbuehler.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190419095746.22894-1-source@stbuehler.de> References: <54496e17-97de-9f9a-9972-c448226bb768@stbuehler.de> <20190419095746.22894-1-source@stbuehler.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Reading the SQ tail needs to come after setting IORING_SQ_NEED_WAKEUP in flags; there is no cheap barrier for ordering a store before a load, a full memory barrier is required. Userspace needs a full memory barrier between updating SQ tail and checking for the IORING_SQ_NEED_WAKEUP too. Signed-off-by: Stefan Bühler --- fs/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 96863e4780b7..912b0b304d90 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1865,7 +1865,8 @@ static int io_sq_thread(void *data) /* Tell userspace we may need a wakeup call */ ctx->sq_ring->flags |= IORING_SQ_NEED_WAKEUP; - smp_wmb(); + /* make sure to read SQ tail after writing flags */ + smp_mb(); if (!io_get_sqring(ctx, &sqes[0])) { if (kthread_should_stop()) { From patchwork Fri Apr 19 09:57:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Stefan_B=C3=BChler?= X-Patchwork-Id: 10909513 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FB7317E0 for ; Fri, 19 Apr 2019 18:49:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4553A28DCE for ; Fri, 19 Apr 2019 18:49:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39A3128DCF; Fri, 19 Apr 2019 18:49:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_INVALID,DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5C6628DCE for ; Fri, 19 Apr 2019 18:49:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728448AbfDSStU (ORCPT ); Fri, 19 Apr 2019 14:49:20 -0400 Received: from mail.stbuehler.de ([5.9.32.208]:51612 "EHLO mail.stbuehler.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727812AbfDSStT (ORCPT ); Fri, 19 Apr 2019 14:49:19 -0400 Received: from chromobil.fritz.box (unknown [IPv6:2a02:8070:a29c:5000:823f:5dff:fe0f:b5b6]) by mail.stbuehler.de (Postfix) with ESMTPSA id D12E5C030C5; Fri, 19 Apr 2019 09:57:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=stbuehler.de; s=stbuehler1; t=1555667868; bh=wb/i8BcN5fqgSFtX3SfggrE1MGsKCMGq/XvRrOHXams=; h=From:To:Subject:Date:In-Reply-To:References:From; b=O1cIkxV4R1bYjDmqx2QOq2FUcethemPxBsHsTN4nG323xtTUytLoGFkpFkWg+MJC3 hQoNW/dTHuNtvbG2VPUagGtqVo6FEq9DSDUIDoBxlNDg+rr2uK6wT10KgRBiu15MYy MD1LChw5x1m7uaVj4c3RvnCaHSi++IfAfYITiaJo= From: =?utf-8?q?Stefan_B=C3=BChler?= To: Jens Axboe , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v1 3/3] io_uring: fix poll full SQ detection Date: Fri, 19 Apr 2019 11:57:46 +0200 Message-Id: <20190419095746.22894-3-source@stbuehler.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190419095746.22894-1-source@stbuehler.de> References: <54496e17-97de-9f9a-9972-c448226bb768@stbuehler.de> <20190419095746.22894-1-source@stbuehler.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP io_uring_poll shouldn't signal EPOLLOUT | EPOLLWRNORM if the queue is full; the old check would always signal EPOLLOUT | EPOLLWRNORM (unless there were U32_MAX - 1 entries in the SQ queue). Signed-off-by: Stefan Bühler --- fs/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 912b0b304d90..047a0aa7a58c 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2576,7 +2576,8 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait) poll_wait(file, &ctx->cq_wait, wait); /* See comment at the top of this file */ smp_rmb(); - if (READ_ONCE(ctx->sq_ring->r.tail) + 1 != ctx->cached_sq_head) + if (READ_ONCE(ctx->sq_ring->r.tail) - ctx->cached_sq_head != + ctx->sq_ring->ring_entries) mask |= EPOLLOUT | EPOLLWRNORM; if (READ_ONCE(ctx->cq_ring->r.head) != ctx->cached_cq_tail) mask |= EPOLLIN | EPOLLRDNORM;