From patchwork Fri May 12 18:24:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9724761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 950A460382 for ; Fri, 12 May 2017 18:24:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 925C826E47 for ; Fri, 12 May 2017 18:24:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 86B0F2846C; Fri, 12 May 2017 18:24:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A98232863F for ; Fri, 12 May 2017 18:24:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758161AbdELSYj (ORCPT ); Fri, 12 May 2017 14:24:39 -0400 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:15310 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758149AbdELSYi (ORCPT ); Fri, 12 May 2017 14:24:38 -0400 X-IronPort-AV: E=Sophos;i="5.38,330,1491235200"; d="scan'208,223";a="119792089" Received: from mail-co1nam03lp0018.outbound.protection.outlook.com (HELO NAM03-CO1-obe.outbound.protection.outlook.com) ([216.32.181.18]) by ob1.hgst.iphmx.com with ESMTP; 13 May 2017 02:24:36 +0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector1-sharedspace-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Dp/Qv0vuvocWVd6Fy/7lB6oEISlOHFlPE0cvoGdoOdY=; b=QMBGFwFTbBVL3ZEuDn3yUGyZ9J7q0rwQGa4YL4FRHr39Tn/65qSPfF7nZPCNqpntaIrDUq+5aCZledrUgcPHAynksDQC2Hj5gezMiE1D16/qH/+UDdowBxsY/Scj02bG5QXF/LTTu7bYRLolDY//CC0tuj6c+HIbFTXVSrl0wKc= Received: from CY1PR0401MB1536.namprd04.prod.outlook.com (10.163.19.154) by CY1PR0401MB1534.namprd04.prod.outlook.com (10.163.19.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1084.16; Fri, 12 May 2017 18:24:35 +0000 Received: from CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) by CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) with mapi id 15.01.1084.022; Fri, 12 May 2017 18:24:35 +0000 From: Bart Van Assche To: "jgunthorpe@obsidianresearch.com" CC: "linux-rdma@vger.kernel.org" , "noas@mellanox.com" , "benjamin.drung@profitbricks.com" , "leon@kernel.org" , "talatb@mellanox.com" Subject: Re: rdma-core in Dabian Thread-Topic: rdma-core in Dabian Thread-Index: AQHSxv1KDashkZ2vCkmI2b5hjfvrZqHsSXaAgAADLwCAAAaWgIAEuMuA Date: Fri, 12 May 2017 18:24:35 +0000 Message-ID: <1494613473.14477.12.camel@sandisk.com> References: <20170507064349.GM22833@mtr-leonro.local> <1494351789.3752.5.camel@profitbricks.com> <1494352472.2518.10.camel@sandisk.com> <20170509181807.GB9715@obsidianresearch.com> In-Reply-To: <20170509181807.GB9715@obsidianresearch.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: obsidianresearch.com; dkim=none (message not signed) header.d=none; obsidianresearch.com; dmarc=none action=none header.from=sandisk.com; x-originating-ip: [63.163.107.100] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; CY1PR0401MB1534; 7:ptxVDnGGCa+Veywxm13ovlSqRvIk4kZMu5WEkvrhHdemMl1XW32zd82NwTThLyUh8Fxvzlp5BAybm47KHB+Z1isaIzIOxLtIQoysJMvRPemRrSy97WiOfW4pCY1qds+MrCQPBSIR6C4n9qKxcZnaXsW58Oy5dvRHf55GS4qlrrJhItj1lzM5wu3N9t6rumRjQgRjzWDdPRcYcGpR9NFKy33/wp7HTn8FidL80nqVt0Ip7oOGyIDkFb6otAnwHQAoD20m1bXwlnz8ElaMpp3hgp0tyGS6sdQHqzk9JkT/VMXUNnR5YoxsIX8WOHhILu83LWWeK5PRSNBdMz85d7/j0Q==; 20:xO0mVnwQEHU0DatRuXzMa2fXfIzpa25I9SRs5vyeNjVAcceDIXQW9ScBD9BbEgZJx1ki++VFT7PLBvKkTrxNdPHk1OeY/eq0uXAwswC8hzG+8W3yXm/Yp+djdVFKqYtL9iAYIabZ0SYaZCGw6KcS/m61p1zqOHX4LrLhYfFRLZc= x-ms-office365-filtering-correlation-id: 6e72b9d7-3531-4a6d-9786-08d499642474 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(48565401081)(201703131423075)(201703031133081)(201702281549075); SRVR:CY1PR0401MB1534; wdcipoutbound: EOP-TRUE x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(6055026)(6041248)(20161123560025)(20161123558100)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148); SRVR:CY1PR0401MB1534; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0401MB1534; x-forefront-prvs: 0305463112 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(39860400002)(39400400002)(39410400002)(39840400002)(39850400002)(39450400003)(24454002)(377424004)(478600001)(8936002)(229853002)(3480700004)(103116003)(3660700001)(5660300001)(122556002)(305945005)(7736002)(102836003)(6116002)(81166006)(1730700003)(8676002)(93886004)(38730400002)(110136004)(4326008)(86362001)(66066001)(6486002)(25786009)(77096006)(36756003)(53936002)(50986999)(76176999)(6506006)(54356999)(2950100002)(6512007)(2351001)(3846002)(3280700002)(6916009)(2900100001)(189998001)(2501003)(33646002)(5890100001)(54906002)(6436002)(2906002)(72206003)(99936001)(99286003)(5640700003); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0401MB1534; H:CY1PR0401MB1536.namprd04.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-originalarrivaltime: 12 May 2017 18:24:35.1295 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1534 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Tue, 2017-05-09 at 12:18 -0600, Jason Gunthorpe wrote: > On Tue, May 09, 2017 at 05:54:33PM +0000, Bart Van Assche wrote: > > > * Bonus points: consolidate the srp daemon. Debian ships a different > > > service file than upstream, but I am against an additional layer > > > introduced by srp_daemon.sh. It would also be nice to have a systemd > > > service shipped by upstream (and not just in the redhat directory) > > > > I will have a look at this too. > > My thoughts.. > > It looked to me like srp_daemon needed one process per port. > > The best path looked to me like using systemd templates (eg > srp_daemon@mlx4_0/0) to allow systemd to directly manage the per port > srp_daemon. > > Then the question is how to request the right templates are > created.. Perhaps udev rules can do this directly, but I'm not sure > about how to get the port #, perhaps an udev triggered script or > something can do it. > > Perhaps there could be an inbetween progarm that took the udev events > and asked systemd to make the right units. > > Another option is to have a srp_daemon 'runner' that monitors udev and > actively manages a set of fork'd childern so that the children cover > all required ports. But this is actually somewhat hard to do well.. > > The big issue with the redhat script is that it is not hotplug safe, > and needs special boot ordering, which we really need to get away from > in general for robustness. Hello Jason, Thanks for having shared your thoughts. When using systemd templates instead of the current approach it would become cumbersome for users to stop and start the srp_daemon on all ports simultaneously. Additionally, if an RDMA adapter is hot-added, should the srp_daemon be started or should it not be started for the newly added ports? An in-between program could indeed do the job of listening for udev events and actively managing srp_daemon children. Since we already have an in-between program, namely srp_daemon.sh, the simplest approach is to modify that shell script. Can you have a look at the attached patch? Thanks, Bart. From bec3884e36298ab2b6fb09c533b575fa95bd3378 Mon Sep 17 00:00:00 2001 From: Bart Van Assche Date: Fri, 12 May 2017 10:24:44 -0700 Subject: [PATCH] srp_daemon.service: Add support for hotplugging Instead of only starting srp_daemon.sh after the RDMA subsystem has been initialized, make srp_daemon.sh scan periodically for RDMA ports. An advantage of the new approach is that it supports hot-add of RDMA adapters. Signed-off-by: Bart Van Assche --- redhat/srp_daemon.service | 4 ---- srp_daemon/srp_daemon.sh.in | 41 ++++++++++++++++++++++++++--------------- 2 files changed, 26 insertions(+), 19 deletions(-) diff --git a/redhat/srp_daemon.service b/redhat/srp_daemon.service index 9510f5fb..cf98aa74 100644 --- a/redhat/srp_daemon.service +++ b/redhat/srp_daemon.service @@ -3,10 +3,6 @@ Description=Start or stop the daemon that attaches to SRP devices Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf DefaultDependencies=false Conflicts=emergency.target emergency.service -Requires=rdma.service -Wants=opensm.service -After=rdma.service opensm.service -After=network.target Before=remote-fs-pre.target [Service] diff --git a/srp_daemon/srp_daemon.sh.in b/srp_daemon/srp_daemon.sh.in index 75e8a31b..74c08d7a 100755 --- a/srp_daemon/srp_daemon.sh.in +++ b/srp_daemon/srp_daemon.sh.in @@ -37,6 +37,7 @@ rescan_interval=60 pids=() pidfile="@CMAKE_INSTALL_FULL_RUNDIR@/srp_daemon.sh.pid" mypid=$$ +umad_devs=() trap_handler() { @@ -49,6 +50,17 @@ trap_handler() exit 0 } +# Check whether $1 is equal to one of $2..${$#} +contains() +{ + local v + + for v in "${@:2}"; do + [ "$v" = "$1" ] && return 0 + done + return 1 +} + # Check if there is another copy running of srp_daemon.sh if [ -f "$pidfile" ]; then if [ -e "/proc/$(cat "$pidfile" 2>/dev/null)/status" ]; then @@ -66,19 +78,18 @@ fi trap 'trap_handler' 2 15 -while [ ! -d ${ibdir} ] -do - sleep 30 +while :; do + for d in ${ibdir}_mad/umad*; do + [ -e "$d" ] || continue + contains "$d" "${umad_devs[@]}" && continue + hca_id="$(<"$d/ibdev")" + port="$(<"$d/port")" + add_target="${ibdir}_srp/srp-${hca_id}-${port}/add_target" + if [ -e "${add_target}" ]; then + ${prog} -e -c -n -i "${hca_id}" -p "${port}" -R "${rescan_interval}" "${params[@]}" >/dev/null 2>&1 & + pids+=($!) + umad_dev+=($d) + fi + done + sleep $rescan_interval done - -for d in ${ibdir}_mad/umad*; do - hca_id="$(<"$d/ibdev")" - port="$(<"$d/port")" - add_target="${ibdir}_srp/srp-${hca_id}-${port}/add_target" - if [ -e "${add_target}" ]; then - ${prog} -e -c -n -i "${hca_id}" -p "${port}" -R "${rescan_interval}" "${params[@]}" >/dev/null 2>&1 & - pids+=($!) - fi -done - -wait -- 2.12.2