From patchwork Wed Oct 28 02:45:50 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arnd Bergmann X-Patchwork-Id: 7506251 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 984D9BEEA4 for ; Wed, 28 Oct 2015 02:46:40 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 9721D2080B for ; Wed, 28 Oct 2015 02:46:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6F26120802 for ; Wed, 28 Oct 2015 02:46:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755270AbbJ1Cqb (ORCPT ); Tue, 27 Oct 2015 22:46:31 -0400 Received: from mout.kundenserver.de ([212.227.126.131]:63667 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755235AbbJ1Cq3 (ORCPT ); Tue, 27 Oct 2015 22:46:29 -0400 Received: from wuerfel.lan. ([134.3.118.24]) by mrelayeu.kundenserver.de (mreue003) with ESMTPSA (Nemesis) id 0M9tPw-1ZkdyF41U0-00B7IR; Wed, 28 Oct 2015 03:46:20 +0100 From: Arnd Bergmann To: tglx@linutronix.de Cc: Arnd Bergmann , Roland Dreier , Eli Cohen , Yevgeny Petrilin , netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 02/25] IB/mthca, net/mlx4: remove counting semaphores Date: Wed, 28 Oct 2015 03:45:50 +0100 Message-Id: <1446000373-1823620-3-git-send-email-arnd@arndb.de> X-Mailer: git-send-email 2.1.0.rc2 In-Reply-To: <1446000373-1823620-1-git-send-email-arnd@arndb.de> References: <1446000373-1823620-1-git-send-email-arnd@arndb.de> X-Provags-ID: V03:K0:O6foWwGAbuebqFCBeegic44eS10F/bXQWG1VJJA/nrVZ5Age6YI ROt06jr2W67IkQ6aD7vmERpiWTk9UX8QAi9iBuso9Sx6ajAvGS2epbSDPnDAbVU8/w1tbTN Y5M6NVhx0jkykuIwrP0j6/1eylUU0KHYiOMO6VfnJIA3AsoKtpzYsNroillHbXMxfquVFiG 0we9lh+rsGUZF0oothiBA== X-UI-Out-Filterresults: notjunk:1; V01:K0:as1KjeSQEh0=:12pcPoRk10DJCE5kKclfNb pvTeW3E1nu7gEwgHjjKpuNliBF2zy3v1vyNbxJ1hYcpOVve50a6qOQ2o25W2gDeydaFPLgLtx 7z6GU6jYevYbGe0DW33lBeX/girzcvQUw8OcFsQKXJOKZQTiQt/80GPk710ZXpm3W3eqtwfNU xa/zwcyOLoB298cBoZffjtY2Y6btu2rIT2vRnAlDLZQl+wJOFsWBiOnsnLU6YXJ7iAxTqwaAu J0+pBEif7BElh+p1AoN7pK9KeLtr+QSOidjPmW/MFsqsU5y6gnRnz0ovIZfu16FQioOjPTMSa EzWoYXfpngkcBbQGRV463iJRTuRLpN6WdzQJwzCrxGKDi3WzCk0oLHMEDbDeaaUV8tV+1cDhu NGgt7eodr69U6hL1FqldQ5eZ7kNojq17ug6JwEzi/BWzC2/Xb/KcOpJLVtSJGd0gnJiPD41xr leHtbKG+V8tc3oOrm7hZMSpEb+cM/bHxDN8sUFAepEekGGLdVf6C4VKwJpJhkpkkSZhQrhmbp Crx966QLSxDxGDuU2rBMSMqJfNLtuwdswFs+wwxZHd0lEiiQeILDW97oTh9F/USQWKSCaQFCT UkbIOwHmSCq9C1kcfK5d/BWPQwr8B4b2IBCI4NuNhgs8nwnkoxZJ2sk/XDWGcsyrwkTBwa9au Fk+N1mEZly1GfjOq6FucelF6yJYiTTPOf/RaU9vlWQdrk4kN3pJEzqViKqCku8utwE6Q= Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mthca and mlx4 device drivers use the same method to switch between polling and event-driven command mode, abusing two semaphores to create a mutual exclusion between one polled command or multiple concurrent event driven commands. Since we want to make counting semaphores go away, this patch replaces the semaphore counting the event-driven commands with an open-coded wait-queue, which should be an equivalent transformation of the code, although it does not make it any nicer. As far as I can tell, there is a preexisting race condition regarding the cmd->use_events flag, which is not protected by any lock. When this flag is toggled while another command is being started, that command gets stuck until the mode is toggled back. A better solution that would solve the race condition and at the same time improve the code readability would create a new locking primitive that replaces both semaphores, like static int mlx4_use_events(struct mlx4_cmd *cmd) { int ret = -EAGAIN; spin_lock(&cmd->lock); if (cmd->use_events && cmd->commands < cmd->max_commands) { cmd->commands++; ret = 1; } else if (!cmd->use_events && cmd->commands == 0) { cmd->commands = 1; ret = 0; } spin_unlock(&cmd->lock); return ret; } static bool mlx4_use_events(struct mlx4_cmd *cmd) { int ret; wait_event(cmd->events_wq, ret = __mlx4_use_events(cmd) >= 0); return ret; } Cc: Roland Dreier Cc: Eli Cohen Cc: Yevgeny Petrilin Cc: netdev@vger.kernel.org Cc: linux-rdma@vger.kernel.org Signed-off-by: Arnd Bergmann Conflicts: drivers/net/mlx4/cmd.c drivers/net/mlx4/mlx4.h --- drivers/infiniband/hw/mthca/mthca_cmd.c | 12 ++++++++---- drivers/infiniband/hw/mthca/mthca_dev.h | 3 ++- drivers/net/ethernet/mellanox/mlx4/cmd.c | 12 ++++++++---- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 3 ++- 4 files changed, 20 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/hw/mthca/mthca_cmd.c b/drivers/infiniband/hw/mthca/mthca_cmd.c index 9d3e5c1ac60e..aad1852e8e10 100644 --- a/drivers/infiniband/hw/mthca/mthca_cmd.c +++ b/drivers/infiniband/hw/mthca/mthca_cmd.c @@ -417,7 +417,8 @@ static int mthca_cmd_wait(struct mthca_dev *dev, int err = 0; struct mthca_cmd_context *context; - down(&dev->cmd.event_sem); + wait_event(dev->cmd.event_wait, + atomic_add_unless(&dev->cmd.commands, -1, 0)); spin_lock(&dev->cmd.context_lock); BUG_ON(dev->cmd.free_head < 0); @@ -459,7 +460,8 @@ out: dev->cmd.free_head = context - dev->cmd.context; spin_unlock(&dev->cmd.context_lock); - up(&dev->cmd.event_sem); + atomic_inc(&dev->cmd.commands); + wake_up(&dev->cmd.event_wait); return err; } @@ -571,7 +573,8 @@ int mthca_cmd_use_events(struct mthca_dev *dev) dev->cmd.context[dev->cmd.max_cmds - 1].next = -1; dev->cmd.free_head = 0; - sema_init(&dev->cmd.event_sem, dev->cmd.max_cmds); + init_waitqueue_head(&dev->cmd.event_wait); + atomic_set(&dev->cmd.commands, dev->cmd.max_cmds); spin_lock_init(&dev->cmd.context_lock); for (dev->cmd.token_mask = 1; @@ -597,7 +600,8 @@ void mthca_cmd_use_polling(struct mthca_dev *dev) dev->cmd.flags &= ~MTHCA_CMD_USE_EVENTS; for (i = 0; i < dev->cmd.max_cmds; ++i) - down(&dev->cmd.event_sem); + wait_event(dev->cmd.event_wait, + atomic_add_unless(&dev->cmd.commands, -1, 0)); kfree(dev->cmd.context); diff --git a/drivers/infiniband/hw/mthca/mthca_dev.h b/drivers/infiniband/hw/mthca/mthca_dev.h index 7e6a6d64ad4e..3055f5c12ac8 100644 --- a/drivers/infiniband/hw/mthca/mthca_dev.h +++ b/drivers/infiniband/hw/mthca/mthca_dev.h @@ -121,7 +121,8 @@ struct mthca_cmd { struct pci_pool *pool; struct mutex hcr_mutex; struct semaphore poll_sem; - struct semaphore event_sem; + wait_queue_head_t event_wait; + atomic_t commands; int max_cmds; spinlock_t context_lock; int free_head; diff --git a/drivers/net/ethernet/mellanox/mlx4/cmd.c b/drivers/net/ethernet/mellanox/mlx4/cmd.c index 78f5a1a0b8c8..60134a4245ef 100644 --- a/drivers/net/ethernet/mellanox/mlx4/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx4/cmd.c @@ -273,7 +273,8 @@ static int mlx4_cmd_wait(struct mlx4_dev *dev, u64 in_param, u64 *out_param, struct mlx4_cmd_context *context; int err = 0; - down(&cmd->event_sem); + wait_event(cmd->event_wait, + atomic_add_unless(&cmd->commands, -1, 0)); spin_lock(&cmd->context_lock); BUG_ON(cmd->free_head < 0); @@ -305,7 +306,8 @@ out: cmd->free_head = context - cmd->context; spin_unlock(&cmd->context_lock); - up(&cmd->event_sem); + atomic_inc(&cmd->commands); + wake_up(&cmd->event_wait); return err; } @@ -380,7 +382,8 @@ int mlx4_cmd_use_events(struct mlx4_dev *dev) priv->cmd.context[priv->cmd.max_cmds - 1].next = -1; priv->cmd.free_head = 0; - sema_init(&priv->cmd.event_sem, priv->cmd.max_cmds); + init_waitqueue_head(&priv->cmd.event_wait); + atomic_set(&priv->cmd.commands, priv->cmd.max_cmds); spin_lock_init(&priv->cmd.context_lock); for (priv->cmd.token_mask = 1; @@ -407,7 +410,8 @@ void mlx4_cmd_use_polling(struct mlx4_dev *dev) priv->cmd.use_events = 0; for (i = 0; i < priv->cmd.max_cmds; ++i) - down(&priv->cmd.event_sem); + wait_event(priv->cmd.event_wait, + atomic_add_unless(&priv->cmd.commands, -1, 0)); kfree(priv->cmd.context); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 5dfa68ffc11c..5a2c55d8ccfd 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -189,7 +189,8 @@ struct mlx4_cmd { void __iomem *hcr; struct mutex hcr_mutex; struct semaphore poll_sem; - struct semaphore event_sem; + wait_queue_head_t event_wait; + atomic_t commands; int max_cmds; spinlock_t context_lock; int free_head;