From patchwork Wed Mar 8 20:03:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9611951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8C47B60414 for ; Wed, 8 Mar 2017 21:01:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8046A28512 for ; Wed, 8 Mar 2017 21:01:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74F5628635; Wed, 8 Mar 2017 21:01:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D258128512 for ; Wed, 8 Mar 2017 21:01:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754230AbdCHVBH (ORCPT ); Wed, 8 Mar 2017 16:01:07 -0500 Received: from bombadil.infradead.org ([65.50.211.133]:44986 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754465AbdCHVBE (ORCPT ); Wed, 8 Mar 2017 16:01:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=1gHh9KHc5qfsqEUIjdQm6lbakqI9hgKImqbQcSjGftg=; b=MWzXWifFNjJPpvUW+azGBHKuv XldzdUkvXYYcpLpQEBSWYzZAZuF1TmAV+8fW/q6ea6SqMGHaN77kKp+qaEoshO2/UV5IneNHBOAss 5cG8tn47faXoftL5OFNO/DwAzZEM97UyYLtnnOhGH0pob2uVLLu2Txm5f7drjjGnnY1CbaH2hQ/CK Mg/K5cq2e6fFq7nmZcKSsaLIIMidykG2OKlkf4CN6swacVY/VbzNKV05th/TzqnffWGkZKzS9XIXf VkiYBm3IfNyA0kRag/e76SEXijJoTPWpFnBIoIdAZZ5/O1MWBm3elOyvFXgpZHA9+bnwP8s9dOjGJ 28Je939WA==; Received: from bzq-82-81-101-184.red.bezeqint.net ([82.81.101.184] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.87 #1 (Red Hat Linux)) id 1clhns-0007NO-R4; Wed, 08 Mar 2017 20:03:21 +0000 From: Sagi Grimberg To: linux-rdma@vger.kernel.org Cc: Christoph Hellwig , Bart Van Assche Subject: [PATCH RESEND] IB/device: Convert ib-comp-wq to be CPU-bound Date: Wed, 8 Mar 2017 22:03:17 +0200 Message-Id: <1489003397-2321-1-git-send-email-sagi@grimberg.me> X-Mailer: git-send-email 2.7.4 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This workqueue is used by our storage target mode ULPs via the new CQ API. Recent observations when working with very high-end flash storage devices reveal that UNBOUND workqueue threads can migrate between cpu cores and even numa nodes (although some numa locality is accounted for). While this attribute can be useful in some workloads, it does not fit in very nicely with the normal run-to-completion model we usually use in our target-mode ULPs and the block-mq irq<->cpu affinity facilities. The whole block-mq concept is that the completion will land on the same cpu where the submission was performed. The fact that our submitter thread is migrating cpus can break this locality. We assume that as a target mode ULP, we will serve multiple initiators/clients and we can spread the load enough without having to use unbound kworkers. Also, while we're at it, expose this workqueue via sysfs which is harmless and can be useful for debug. Signed-off-by: Sagi Grimberg Reviewed-by: Bart Van Assche -- --- drivers/infiniband/core/device.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 593d2ce6ec7c..fb5ae9564e80 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -1015,8 +1015,7 @@ static int __init ib_core_init(void) return -ENOMEM; ib_comp_wq = alloc_workqueue("ib-comp-wq", - WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, - WQ_UNBOUND_MAX_ACTIVE); + WQ_HIGHPRI | WQ_MEM_RECLAIM | WQ_SYSFS, 0); if (!ib_comp_wq) { ret = -ENOMEM; goto err;