From patchwork Wed Mar 1 16:46:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9598821 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E0DBF600CB for ; Wed, 1 Mar 2017 16:46:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D54CB283E8 for ; Wed, 1 Mar 2017 16:46:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C9FC428490; Wed, 1 Mar 2017 16:46:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64B91283E8 for ; Wed, 1 Mar 2017 16:46:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751742AbdCAQqS (ORCPT ); Wed, 1 Mar 2017 11:46:18 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:36180 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752330AbdCAQqS (ORCPT ); Wed, 1 Mar 2017 11:46:18 -0500 Received: by mail-wr0-f196.google.com with SMTP id l37so6254158wrc.3 for ; Wed, 01 Mar 2017 08:46:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=RQ5eHqSyi6LlSNR9g2UH4m5eghZt10mnk36coKE2E+k=; b=OqpLvf8iK6hbhh4yyltF8wutjFD95l/MqWekYBPZY0kyQLM5nhOSTVKFyXfikBIZ1E RK86ENOKt8Hd/O42c9FyzSbQOng87uQsplVDmNMHuE60g7NQTtIS5Y0VfYzChaP3lVwJ 5j1PWcageykHrTAIr1IfTl0WrEcA3IV1RTHLCpat9Pmya9s8Fon75R+pjDQxJKaN/CxD CVvsFoxZ6BdX4GwpppxqDDOe+sQXsrFYFPkuDBxOHPfPphXRg2i2qxAWYU8aH+O3CZaX Ca4zVe7k4UO+09/Wyl7sxFZTzpQOtGYUCW1rzwikyauwyTMCQH9aKcwgcGvYkF1q1EB8 W2ww== X-Gm-Message-State: AMke39mMoE00a0+vHDO9JvDH6sQ7AwlPt7H5xE+pn4gpSkomGp/pVF4tGbTpeaDkuSYscw== X-Received: by 10.223.152.239 with SMTP id w102mr8219405wrb.72.1488386776329; Wed, 01 Mar 2017 08:46:16 -0800 (PST) Received: from [192.168.64.116] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id i82sm97818wmf.1.2017.03.01.08.46.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Mar 2017 08:46:15 -0800 (PST) Subject: Re: Poll CQ syncing problem To: Noa Osherovich , hch@lst.de References: <3ba1baab-e2ac-358d-3b3b-ff4a27405c93@mellanox.com> <0786659a-da12-e8f7-329e-3caa8cc8791f@grimberg.me> Cc: linux-rdma@vger.kernel.org, Majd Dibbiny From: Sagi Grimberg Message-ID: <5f8f11de-e659-dbd8-a105-a32d7c239e37@grimberg.me> Date: Wed, 1 Mar 2017 18:46:14 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <0786659a-da12-e8f7-329e-3caa8cc8791f@grimberg.me> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP > Hmm. I'm wandering if this is really the root-cause... Can it be the > fact that ib_comp_wq is unbound causing the worker to migrate cpu cores > in its lifetime? > > I wanted to change that a while ago and sent a patch for it [1]. [1]: --- IB/device: Convert ib-comp-wq to be CPU-bound This workqueue is used by our storage target mode ULPs via the new CQ API. Recent observations when working with very high-end flash storage devices reveal that UNBOUND workqueue threads can migrate between cpu cores and even numa nodes (although some numa locality is accounted for). While this attribute can be useful in some workloads, it does not fit in very nicely with the normal run-to-completion model we usually use in our target-mode ULPs and the block-mq irq<->cpu affinity facilities. The whole block-mq concept is that the completion will land on the same cpu where the submission was performed. The fact that our submitter thread is migrating cpus can break this locality. We assume that as a target mode ULP, we will serve multiple initiators/clients and we can spread the load enough without having to use unbound kworkers. Also, while we're at it, expose this workqueue via sysfs which is harmless and can be useful for debug. Signed-off-by: Sagi Grimberg --- drivers/infiniband/core/device.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 760ef603a468..15f4bdf89fe1 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -999,8 +999,7 @@ static int __init ib_core_init(void) return -ENOMEM; ib_comp_wq = alloc_workqueue("ib-comp-wq", - WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, - WQ_UNBOUND_MAX_ACTIVE); + WQ_HIGHPRI | WQ_MEM_RECLAIM | WQ_SYSFS, 0); if (!ib_comp_wq) { ret = -ENOMEM; goto err;