From patchwork Wed Jul 10 19:52:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11038877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EF12138D for ; Wed, 10 Jul 2019 19:52:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6273E28765 for ; Wed, 10 Jul 2019 19:52:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 567CC289DA; Wed, 10 Jul 2019 19:52:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E00CD28765 for ; Wed, 10 Jul 2019 19:52:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728210AbfGJTwd (ORCPT ); Wed, 10 Jul 2019 15:52:33 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:42839 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725911AbfGJTwb (ORCPT ); Wed, 10 Jul 2019 15:52:31 -0400 Received: by mail-qk1-f195.google.com with SMTP id 201so2894727qkm.9 for ; Wed, 10 Jul 2019 12:52:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id; bh=7LrlK+yNNAjChYy/KKDWPKU1AjfJ6+kVoZtH7HVXgDk=; b=mbcEIb07lqT+Z3l0i7Q6mn7bsmxOkVz5EIr4O7is29m/Z9VmB8Lwt2qzOiG59dHNx5 CAqIC/UQjbm1xjptJif+P4Gul1WL9gab/L+V9jpBrOvNY8Gct3CEhCS6kSYMvq2oYWp+ yjwL1jRAlMvuUOnD1Z2jMpWoobCQwipkoDBrFptMDxdj27XFsG9UBPv4sOGwhryGArV+ /10YrTZqTni+LzVKfKw9cgWK96dUpRQYlWDAs1+pJgvYghcY8ed060idj1Nfuz5KNPox ZVcoPTpEYlvrK0bqJHjp6qVW+2YGQrvJwV7BQdg8nhPJxwL8LXwGWOIWZqdGc0MeccJ5 NETw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=7LrlK+yNNAjChYy/KKDWPKU1AjfJ6+kVoZtH7HVXgDk=; b=Gx4G/1OENlSjBF7AFLuuIoWGZVfgQA1sfRG+ute3XnBDETHK4qa2n6xktSJ44kK6/B fBHyZfCr+oBLpMl9v5jXDuKKjOGkwiO59FVSn4gdERkhI6wdTWthE4mOBtMAdPJc8JOQ VsO9C3OSUsj3hB4yO2mxTsiOQH3oJaazrL8RxTGosKeOl6EGMrqu/MJtN1QU45A5dFMI wSED3yM2a7OsAeJtQWsmA7OXyVoBSVn551SQEGrcc54jkNKp9G7FcqY+SvTAqnSJAZ95 +2NOtqeGtGOAQp1IGAD1XqWZyhTTf5w2ZRdyRCj+GAmddNjbvEKozjzpjekWnU2EKG8E iN9g== X-Gm-Message-State: APjAAAU+8wTeUmIp0nbyBzv+Jlx59BgZ867phmsNT+5XSYqk1DccsD+X Q+MJvjaI+W4b6uwthYncgWaDEhR0s7DIJg== X-Google-Smtp-Source: APXvYqzrX5Mu1F+rJEn/mze6e18ppOiP76Z8WHG3lMMHqzs63hTFSa9n5x3JOorwaVTHB5e/gE1Qww== X-Received: by 2002:a37:9d13:: with SMTP id g19mr19799619qke.124.1562788349890; Wed, 10 Jul 2019 12:52:29 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5ce]) by smtp.gmail.com with ESMTPSA id f133sm1554254qke.62.2019.07.10.12.52.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Jul 2019 12:52:29 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, axboe@kernel.dk, linux-block@vger.kernel.org Subject: [PATCH 1/2] wait: add wq_has_multiple_sleepers helper Date: Wed, 10 Jul 2019 15:52:26 -0400 Message-Id: <20190710195227.92322-1-josef@toxicpanda.com> X-Mailer: git-send-email 2.13.5 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP rq-qos sits in the io path so we want to take locks as sparingly as possible. To accomplish this we try not to take the waitqueue head lock unless we are sure we need to go to sleep, and we have an optimization to make sure that we don't starve out existing waiters. Since we check if there are existing waiters locklessly we need to be able to update our view of the waitqueue list after we've added ourselves to the waitqueue. Accomplish this by adding this helper to see if there are more than two waiters on the waitqueue. Suggested-by: Jens Axboe Signed-off-by: Josef Bacik --- include/linux/wait.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/wait.h b/include/linux/wait.h index b6f77cf60dd7..89c41a7b3046 100644 --- a/include/linux/wait.h +++ b/include/linux/wait.h @@ -126,6 +126,27 @@ static inline int waitqueue_active(struct wait_queue_head *wq_head) return !list_empty(&wq_head->head); } +/** + * wq_has_multiple_sleepers - check if there are multiple waiting prcesses + * @wq_head: wait queue head + * + * Returns true of wq_head has multiple waiting processes. + * + * Please refer to the comment for waitqueue_active. + */ +static inline bool wq_has_multiple_sleepers(struct wait_queue_head *wq_head) +{ + /* + * We need to be sure we are in sync with the + * add_wait_queue modifications to the wait queue. + * + * This memory barrier should be paired with one on the + * waiting side. + */ + smp_mb(); + return !list_is_singular(&wq_head->head); +} + /** * wq_has_sleeper - check if there are any waiting processes * @wq_head: wait queue head From patchwork Wed Jul 10 19:52:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11038875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E10DE6C5 for ; Wed, 10 Jul 2019 19:52:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D33E428765 for ; Wed, 10 Jul 2019 19:52:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C74AB289DA; Wed, 10 Jul 2019 19:52:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DF2428765 for ; Wed, 10 Jul 2019 19:52:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728243AbfGJTwf (ORCPT ); Wed, 10 Jul 2019 15:52:35 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:46986 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728211AbfGJTwd (ORCPT ); Wed, 10 Jul 2019 15:52:33 -0400 Received: by mail-qt1-f195.google.com with SMTP id h21so3771293qtn.13 for ; Wed, 10 Jul 2019 12:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=h2XA2X1ZY1EHxtyN0Imads8NvEkfjbdW711+ZDB5c94=; b=1p+GRmWLsFIydILSpH4SUU4qr33pL1CfrGVK7adXHtu2RSh053cgJVt/KL03koY5Gn +IE3FsqsOtjrvoaqjzN+STTVBDYcATwAhiiZ72usKmSe+2GmdGTw1VqUJ81hmVRPqt9H WVj4/Hj7zFsQ8S6Nbzk0xZZZsKW6+YytJ25j+yLDZfZhiDbpUnAxCfHiWBhQTNDZ1k6X cogq57vJw2oU8h0NdUvaJkKFy1dL3TbV6KTF9ZGhf1fy1PVEBOBt9AxARFnuvgi3ohVY QKgO1aJtdScOA9O3P99DUQsZDSLrrRDeG2axQh+RCu2+Oqna0uk/n/Ihmh9h7Bc0bSLL l8ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=h2XA2X1ZY1EHxtyN0Imads8NvEkfjbdW711+ZDB5c94=; b=gJoYNr+MpaQRkBXCKjRScAvPRIyUX2goyuwJn6cNa3V6h9nF6yYAG3u55Iq21vvM2h sJAXQNwT3lr79mMeYgk4PgELRiHwu2TwJQ7j1onbuYmg75DBFGk295a4Onuwz755X4aS SUJL8IHxJYXgnpga8LjNUkB3qVBRSZFX3pd54OoplbBmTM59riHY47bkXZLvIMQDvBkm uCh/4OJwfQ6axcNRSolnGHEBEpySc94NgJz/Fp9P1nBx7S81fv0TbdaxJA7kibTdqwvO 1xAzuLAiaEBhgmplvtQCJLRUFdeGG9zWiIYcxpNXEQ3FXmY3JqGvFlMCC0WESLmXzQOX 4toQ== X-Gm-Message-State: APjAAAXuA4PERaU9f671CQXI57NUfzlulI8rQLn2Dt7YTF7cfRvR3E/3 DGkX3mk1b61hHEcXUGYYaRTDlQ== X-Google-Smtp-Source: APXvYqyBfvrfYhxjdfcV4ySFNAvtIm75K1LNcIgBlruRP7C/8rwENvmllFbXCx4jBoSGkRQebtMdHQ== X-Received: by 2002:ac8:2fa8:: with SMTP id l37mr25255325qta.358.1562788352412; Wed, 10 Jul 2019 12:52:32 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5ce]) by smtp.gmail.com with ESMTPSA id i1sm1386769qtb.7.2019.07.10.12.52.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Jul 2019 12:52:31 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, axboe@kernel.dk, linux-block@vger.kernel.org Subject: [PATCH 2/2] rq-qos: fix missed wake-ups in rq_qos_throttle Date: Wed, 10 Jul 2019 15:52:27 -0400 Message-Id: <20190710195227.92322-2-josef@toxicpanda.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20190710195227.92322-1-josef@toxicpanda.com> References: <20190710195227.92322-1-josef@toxicpanda.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We saw a hang in production with WBT where there was only one waiter in the throttle path and no outstanding IO. This is because of the has_sleepers optimization that is used to make sure we don't steal an inflight counter for new submitters when there are people already on the list. We can race with our check to see if the waitqueue has any waiters (this is done locklessly) and the time we actually add ourselves to the waitqueue. If this happens we'll go to sleep and never be woken up because nobody is doing IO to wake us up. Fix this by checking if the waitqueue has multiple sleepers after we add ourselves to the list, that way we have an uptodate view of the list. Signed-off-by: Josef Bacik --- block/blk-rq-qos.c | 1 + 1 file changed, 1 insertion(+) diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index 659ccb8b693f..b39b5f3fb01b 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -244,6 +244,7 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_data, return; prepare_to_wait_exclusive(&rqw->wait, &data.wq, TASK_UNINTERRUPTIBLE); + has_sleeper = wq_has_multiple_sleepers(&rqw->wait); do { if (data.got_token) break;