From patchwork Thu Jun 29 15:16:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13297083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27475EB64D9 for ; Thu, 29 Jun 2023 15:16:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232475AbjF2PQg (ORCPT ); Thu, 29 Jun 2023 11:16:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232248AbjF2PQf (ORCPT ); Thu, 29 Jun 2023 11:16:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C5681BD1 for ; Thu, 29 Jun 2023 08:16:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 958C56156C for ; Thu, 29 Jun 2023 15:16:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81B1DC433C0; Thu, 29 Jun 2023 15:16:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1688051793; bh=4avJEdfWOu7+j0BNKREDlNdzSA16EPIWvquz8T3FyGU=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=sQU3VAqVVRHqAZkGzBtYp5pn4R2x/sn5PxlIigxaThl0Pvj6n0s6topPZJGoMboh7 HnDJmll92P6vwGWdeIMhYz0gMDUOyt+Y6mI2RUrqlCrE6sFeRDQEuB/kZR/WmtKf5r SJFF14CADo1L/kBGGhbS46Haq9mA013oIuLtPEAcwQ1fAQFSQLi2S0Zaq+X+RvuQqz Th4jtor3756UMfxKgxNDm5lvt4wurzIYpROq16A6srRwdzSZKK6na8QCJ5C2Nq6NAG z1LBou6jG5vHSKRdtTTh5ugsLFIB/d//JcBn9vJgW2sdhKExAz7wX2btG8bDXomXHQ /pbgkhPbwLD2Q== Subject: [PATCH v5 1/4] RDMA/siw: Fabricate a GID on tun and loopback devices From: Chuck Lever To: jgg@nvidia.com Cc: Tom Talpey , Bernard Metzler , Tom Talpey , Chuck Lever , tom@talpey.com, BMT@zurich.ibm.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev Date: Thu, 29 Jun 2023 11:16:31 -0400 Message-ID: <168805179153.164730.12186761638624428354.stgit@manet.1015granger.net> In-Reply-To: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> References: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Chuck Lever LOOPBACK and NONE (tunnel) devices have all-zero MAC addresses. Currently, siw_device_create() falls back to copying the IB device's name in those cases, because an all-zero MAC address breaks the RDMA core address resolution mechanism. However, at the point when siw_device_create() constructs a GID, the ib_device::name field is uninitialized, leaving the MAC address to remain in an all-zero state. Fabricate a random artificial GID for such devices, and ensure this artificial GID is returned for all device query operations. Reported-by: Tom Talpey Link: https://lore.kernel.org/linux-rdma/SA0PR15MB391986C07C4D41E107E79659994FA@SA0PR15MB3919.namprd15.prod.outlook.com/T/#t Fixes: a2d36b02c15d ("RDMA/siw: Enable siw on tunnel devices") Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey Signed-off-by: Chuck Lever --- drivers/infiniband/sw/siw/siw.h | 1 + drivers/infiniband/sw/siw/siw_main.c | 22 ++++++++-------------- drivers/infiniband/sw/siw/siw_verbs.c | 4 ++-- 3 files changed, 11 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 2f3a9cda3850..8b4a710b82bc 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -74,6 +74,7 @@ struct siw_device { u32 vendor_part_id; int numa_node; + char raw_gid[ETH_ALEN]; /* physical port state (only one port per device) */ enum ib_port_state state; diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c index 65b5cda5457b..f45600d169ae 100644 --- a/drivers/infiniband/sw/siw/siw_main.c +++ b/drivers/infiniband/sw/siw/siw_main.c @@ -75,8 +75,7 @@ static int siw_device_register(struct siw_device *sdev, const char *name) return rv; } - siw_dbg(base_dev, "HWaddr=%pM\n", sdev->netdev->dev_addr); - + siw_dbg(base_dev, "HWaddr=%pM\n", sdev->raw_gid); return 0; } @@ -313,24 +312,19 @@ static struct siw_device *siw_device_create(struct net_device *netdev) return NULL; base_dev = &sdev->base_dev; - sdev->netdev = netdev; - if (netdev->type != ARPHRD_LOOPBACK && netdev->type != ARPHRD_NONE) { - addrconf_addr_eui48((unsigned char *)&base_dev->node_guid, - netdev->dev_addr); + if (netdev->addr_len) { + memcpy(sdev->raw_gid, netdev->dev_addr, + min_t(unsigned int, netdev->addr_len, ETH_ALEN)); } else { /* - * This device does not have a HW address, - * but connection mangagement lib expects gid != 0 + * This device does not have a HW address, but + * connection mangagement requires a unique gid. */ - size_t len = min_t(size_t, strlen(base_dev->name), 6); - char addr[6] = { }; - - memcpy(addr, base_dev->name, len); - addrconf_addr_eui48((unsigned char *)&base_dev->node_guid, - addr); + eth_random_addr(sdev->raw_gid); } + addrconf_addr_eui48((u8 *)&base_dev->node_guid, sdev->raw_gid); base_dev->uverbs_cmd_mask |= BIT_ULL(IB_USER_VERBS_CMD_POST_SEND); diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index 398ec13db624..32b0befd25e2 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -157,7 +157,7 @@ int siw_query_device(struct ib_device *base_dev, struct ib_device_attr *attr, attr->vendor_part_id = sdev->vendor_part_id; addrconf_addr_eui48((u8 *)&attr->sys_image_guid, - sdev->netdev->dev_addr); + sdev->raw_gid); return 0; } @@ -218,7 +218,7 @@ int siw_query_gid(struct ib_device *base_dev, u32 port, int idx, /* subnet_prefix == interface_id == 0; */ memset(gid, 0, sizeof(*gid)); - memcpy(&gid->raw[0], sdev->netdev->dev_addr, 6); + memcpy(gid->raw, sdev->raw_gid, ETH_ALEN); return 0; } From patchwork Thu Jun 29 15:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13297084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0FD0EB64DD for ; Thu, 29 Jun 2023 15:16:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232511AbjF2PQm (ORCPT ); Thu, 29 Jun 2023 11:16:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232504AbjF2PQl (ORCPT ); Thu, 29 Jun 2023 11:16:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 976401BD3 for ; Thu, 29 Jun 2023 08:16:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2D4126156E for ; Thu, 29 Jun 2023 15:16:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28F66C433C9; Thu, 29 Jun 2023 15:16:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1688051799; bh=tDBsqL5NlQ8X0YV7ODIeH9tuaa9S3XC9sn5ZSeG6NTQ=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=gqCf/3v4y62eI0ui0xUYSSqPN+oEy7h3R527bJ1vJ3norp5NcsAckvn8RnN29nFf2 jHjNV6JeUROmjP7MAjVz4hE3hqk/0MvO1WwP84D+TQg4PTJaLMpoXRcu7UPvVr48+D jJWjXTmGS1INwFImBrMZcR9kr3vKkJKmKErVk5iseaqS3Kp+bL9RUNkIilp1G6jke5 +N1ZczwNUfJC/3hGvFlIpFQqwmd+Vz/vrs8oeQOL0AQtzLEQEoEygQeUBB1gAYYr+Y PRP+MGUz5dIN6HC4/SbtDovL4Gvps3FZIIBxTxwK4iiaaI4iH2h3RjnhuM+i7k5O4t BCUTJW6ksZ+Hw== Subject: [PATCH v5 2/4] RDMA/core: Set gid_attr.ndev for iWARP devices From: Chuck Lever To: jgg@nvidia.com Cc: Tom Talpey , Chuck Lever , tom@talpey.com, BMT@zurich.ibm.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev Date: Thu, 29 Jun 2023 11:16:38 -0400 Message-ID: <168805179821.164730.12205925143704039946.stgit@manet.1015granger.net> In-Reply-To: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> References: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Chuck Lever Have the iwarp side properly set the ndev in the device's sgid_attrs so that address resolution can treat it more like a RoCE device. Suggested-by: Jason Gunthorpe Reviewed-by: Tom Talpey Signed-off-by: Chuck Lever --- drivers/infiniband/core/cache.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c index 2e91d8879326..33f9d02f9b60 100644 --- a/drivers/infiniband/core/cache.c +++ b/drivers/infiniband/core/cache.c @@ -1457,6 +1457,17 @@ static int config_non_roce_gid_cache(struct ib_device *device, i); goto err; } + + if (rdma_protocol_iwarp(device, port)) { + struct net_device *ndev; + + ndev = ib_device_get_netdev(device, port); + if (!ndev) + continue; + RCU_INIT_POINTER(gid_attr.ndev, ndev); + dev_put(ndev); + } + gid_attr.index = i; tprops->subnet_prefix = be64_to_cpu(gid_attr.gid.global.subnet_prefix); From patchwork Thu Jun 29 15:16:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13297085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60A8DEB64DC for ; Thu, 29 Jun 2023 15:16:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232488AbjF2PQv (ORCPT ); Thu, 29 Jun 2023 11:16:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232523AbjF2PQs (ORCPT ); Thu, 29 Jun 2023 11:16:48 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A801E58 for ; Thu, 29 Jun 2023 08:16:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A45A36156C for ; Thu, 29 Jun 2023 15:16:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A60F6C433C0; Thu, 29 Jun 2023 15:16:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1688051806; bh=9fMviFvWZceSsBt94QW3FS2ZCAc4TJ8cIBwarSUP6rE=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=sNoKZgRa7TWDFBXDwrrz/D1JAFcuKq+cRdvWb52+QQW6nqzJ/cI8Hq7T0q+8MyZ39 2QBOR0DZw0ia0HtoOLXKAReS2Ob1Y9RM5pLqnNeGpZ+zZSOObwEgDmswJ/WsO2LsdO Q9HLfVbVxPiAE7cm262lPf/M8AqZL/5Rv73225UIhTruxh7TtvSpKWhSk60mVA8GbV cb5xF4ft2aRf4yqkWSiHGOhd+GhHhXHjFlowsvaFcwAmTYTqTlExLH42mBeCwqBzV6 I4H3xQmfQybM/hcOpvTNcfBt7YQ560gi9pns7m/WeuAZzqa1OihPystdBbAQPWZQAj waVLPwXbKAY4w== Subject: [PATCH v5 3/4] RDMA/cma: Deduplicate error flow in cma_validate_port() From: Chuck Lever To: jgg@nvidia.com Cc: Chuck Lever , tom@talpey.com, BMT@zurich.ibm.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev Date: Thu, 29 Jun 2023 11:16:44 -0400 Message-ID: <168805180479.164730.3680697581731936765.stgit@manet.1015granger.net> In-Reply-To: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> References: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Chuck Lever Clean up to prepare for the addition of new logic. Signed-off-by: Chuck Lever --- drivers/infiniband/core/cma.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 6b3f4384e46a..889b3e4ea980 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -686,30 +686,31 @@ cma_validate_port(struct ib_device *device, u32 port, struct rdma_id_private *id_priv) { struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr; + const struct ib_gid_attr *sgid_attr = ERR_PTR(-ENODEV); int bound_if_index = dev_addr->bound_dev_if; - const struct ib_gid_attr *sgid_attr; int dev_type = dev_addr->dev_type; struct net_device *ndev = NULL; if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net)) - return ERR_PTR(-ENODEV); + goto out; if ((dev_type == ARPHRD_INFINIBAND) && !rdma_protocol_ib(device, port)) - return ERR_PTR(-ENODEV); + goto out; if ((dev_type != ARPHRD_INFINIBAND) && rdma_protocol_ib(device, port)) - return ERR_PTR(-ENODEV); + goto out; if (dev_type == ARPHRD_ETHER && rdma_protocol_roce(device, port)) { ndev = dev_get_by_index(dev_addr->net, bound_if_index); if (!ndev) - return ERR_PTR(-ENODEV); + goto out; } else { gid_type = IB_GID_TYPE_IB; } sgid_attr = rdma_find_gid_by_port(device, gid, gid_type, port, ndev); dev_put(ndev); +out: return sgid_attr; } From patchwork Thu Jun 29 15:16:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13297086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D978EB64D9 for ; Thu, 29 Jun 2023 15:17:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232571AbjF2PRR (ORCPT ); Thu, 29 Jun 2023 11:17:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232584AbjF2PRK (ORCPT ); Thu, 29 Jun 2023 11:17:10 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5A7830F0 for ; Thu, 29 Jun 2023 08:16:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4C0AC61574 for ; Thu, 29 Jun 2023 15:16:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48E2CC433C0; Thu, 29 Jun 2023 15:16:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1688051812; bh=06N97gGQc8csLjHHjlwWYQmJaMokqjKbBQ7y/4wSYQ0=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=T9UFVGLv9lmt+6n+ZtdN+0M+bLkDd+r/iaSlX7X9faTbNwM9hIPrz8Jprvq34R3NK jzTJmonrKdINgn3szA+UwIMcbAIrEPydBBcKhZB84heYd1M4I2x7/RUcVDHy1OdfCs xeEO+eW9PR8kiJn8/HEu29gFeES4JCRAHUkyDkmMLPEIq7Pfar6M60ZiQPf622MCgf U0XQpjojvaD8lsj1zCL/BFFNdnRDuXDgWtsleF+9759PZWBQZOKy5N5NYj6lfgZ8XZ zsiqx7CkaD9ozC8CZQBYHmcFrDvlNYy0YjBW6zhHZzbHr3/u4e06Ob3yOlH09Bvsdg QQQWeMdVjR1ug== Subject: [PATCH v5 4/4] RDMA/cma: Avoid GID lookups on iWARP devices From: Chuck Lever To: jgg@nvidia.com Cc: Chuck Lever , tom@talpey.com, BMT@zurich.ibm.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev Date: Thu, 29 Jun 2023 11:16:51 -0400 Message-ID: <168805181129.164730.67097436102991889.stgit@manet.1015granger.net> In-Reply-To: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> References: <168805171754.164730.1318944278265675377.stgit@manet.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Chuck Lever We would like to enable the use of siw on top of a VPN that is constructed and managed via a tun device. That hasn't worked up until now because ARPHRD_NONE devices (such as tun devices) have no GID for the RDMA/core to look up. But it turns out that the egress device has already been picked for us -- no GID is necessary. addr_handler() just has to do the right thing with it. Suggested-by: Jason Gunthorpe Signed-off-by: Chuck Lever --- drivers/infiniband/core/cma.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 889b3e4ea980..07bb5ac4019d 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -700,6 +700,21 @@ cma_validate_port(struct ib_device *device, u32 port, if ((dev_type != ARPHRD_INFINIBAND) && rdma_protocol_ib(device, port)) goto out; + /* Linux iWARP devices have but one port */ + if (rdma_protocol_iwarp(device, port)) { + sgid_attr = rdma_get_gid_attr(device, port, 0); + if (IS_ERR(sgid_attr)) + goto out; + + rcu_read_lock(); + ndev = rcu_dereference(sgid_attr->ndev); + if (!net_eq(dev_net(ndev), dev_addr->net) || + ndev->ifindex != bound_if_index) + sgid_attr = ERR_PTR(-ENODEV); + rcu_read_unlock(); + goto out; + } + if (dev_type == ARPHRD_ETHER && rdma_protocol_roce(device, port)) { ndev = dev_get_by_index(dev_addr->net, bound_if_index); if (!ndev)