From patchwork Fri Jun 21 16:22:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13707815 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEEA1757F8 for ; Fri, 21 Jun 2024 16:22:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986964; cv=none; b=CqEii4NMoMBHcH79zQ/Z4/bbRegJmL3GtrAzI65AER8GZxJc+WvSDVU0iPQ46zzGXZEh6k0DkaPnS3gLrLcfOQAot8s+4eop3NuiecV3iqCzKe1OZjaaG5qa/GCmOIYTYykZt7aHhhpCtF+Bt7BhMY584x/ugz69XuDEnzMH9mE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986964; c=relaxed/simple; bh=VGWkXsOIrcL0K44gGoN4XC20rsgkpm6CuZolyejrPDw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WxTymJ9GqJL7H/Zvlw+9NOAdoVSfYrHPLlNQNb0ul1whazkWPrmTm/rFGng/UnrdPy/slJWXJsHqjPSZx2fN7u9nzMNGxvRNZhoZcJU8HqATe3rped6TG9hCSezX36ISkPj1sB/m5Y3l/gMjoTX1l97nYfasbkNWLtg8RnyZT7s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DMns3dnL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DMns3dnL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39C86C3277B; Fri, 21 Jun 2024 16:22:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718986964; bh=VGWkXsOIrcL0K44gGoN4XC20rsgkpm6CuZolyejrPDw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DMns3dnLFWXIWySVVkyUJW8vu1vHO0NRKBfFg3zVuzJCjECoIYwzG10Et9eMtgUeE TDxLSLgZwE5BOuh2G+tPrOCzyth6Fd/vtJbhcvI4VR2joyrec+bAQZ7eR4TYDF4tNL myBm9H9UuB/ZO2DZYoYmkIprkwoWVHALuwtINNYdVZyvUewMQ2QkXlV3gKluLpug4x 4z/q4K++sF5KqwOonYlIYQFjrC6GVfU9PyjQXNGY+nQWru5bFcgSG4PoKK0x1jl/Sg Vy7wtEjd2mLfko6dxkLoCD81GgWvWexlT4twn+muzOjwdhCZkPuG4knZp9GXpitWa/ iV4/Ln1litA9Q== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [PATCH v2 1/4] nfs/blocklayout: Fix premature PR key unregistration Date: Fri, 21 Jun 2024 12:22:29 -0400 Message-ID: <20240621162227.215412-7-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240621162227.215412-6-cel@kernel.org> References: <20240621162227.215412-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7169; i=chuck.lever@oracle.com; h=from:subject; bh=seYo1ktlLJ7RNV/jCJ4kC1U9ecb3ceeLiqQyhLZI/98=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmdajI9fqOmRvkvZYiNFqa5pVRM1OhAWySMA5UN 8VPnPgVNBmJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnWoyAAKCRAzarMzb2Z/ lzPgD/9p2I5ki4WhEqUEP6SPQqg3NixwWSHXxQ9R8KhvEOfnW5GWDTpTO1KjEDn8ju9wuX061ip sB73vp68R63ls5w8qVLsB2j9Nn7ZeWCifsZYy2ye55wL2leS+8HOoDQ4lCq4+GQkQayuzr1srnr E+lIv738q9f1MKUHnQGUqPnHf4PVuwgHe8obddRExjxMY2bx6JHpCAzmrMBaWRXPXpQLG0omCdO 3aTbRw7yXdv59UrcThy4W9z/ynvv5MOZQzTAYH52cWJZpGjzsYtduJChqZzlumzYC2tpZbyZQAH J1V8aalsepedXZSAjuiW1BrDDj/RjA7ZnMeouNL8sdhTJyXzKh9UJ5ZM0rjYy/pqWXGm8tvv7bv duAroyV1my41aYWs/kkCWjcUKtOcl5JUR3gAJddkp0qR+Lg8ojQQnJhWnXTJshXQNLWNI+dbhLK Kknw8u5YEZpCLgRSa0Na8AgqxN2Wbr6YO589QhFrm/9X6eFVuAMZg6dbkAkrulYi/8HBpVq9Ox4 9iXoD9j/5U9z6Cop91qVu3pFLkywcyL0LNI8FxWO37c9oBfy8HbxPQrXxFT5q2lZhMaYePdmdTs CrRBsC3K6SSi1Q3Fo8HliMypg3G4BujQ6Jga3d6B8B9p1w1tQOJfX/76rquxUyFQFP27RPyPySs kBvxNNzcHWEhe5A== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever During generic/069 runs with pNFS SCSI layouts, the NFS client emits the following in the system journal: kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#16 CDB: Write(10) 2a 00 00 00 00 50 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 80 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#18 CDB: Write(10) 2a 00 00 00 00 60 00 00 08 00 kernel: sd 6:0:0:1: [sdb] tag#17 CDB: Write(10) 2a 00 00 00 00 58 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 96 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 kernel: reservation conflict error, dev sdb, sector 88 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 systemd[1]: fstests-generic-069.scope: Deactivated successfully. systemd[1]: fstests-generic-069.scope: Consumed 5.092s CPU time. systemd[1]: media-test.mount: Deactivated successfully. systemd[1]: media-scratch.mount: Deactivated successfully. kernel: sd 6:0:0:1: reservation conflict kernel: failed to unregister PR key. This appears to be due to a race. bl_alloc_lseg() calls this: 561 static struct nfs4_deviceid_node * 562 bl_find_get_deviceid(struct nfs_server *server, 563 const struct nfs4_deviceid *id, const struct cred *cred, 564 gfp_t gfp_mask) 565 { 566 struct nfs4_deviceid_node *node; 567 unsigned long start, end; 568 569 retry: 570 node = nfs4_find_get_deviceid(server, id, cred, gfp_mask); 571 if (!node) 572 return ERR_PTR(-ENODEV); nfs4_find_get_deviceid() does a lookup without the spin lock first. If it can't find a matching deviceid, it creates a new device_info (which calls bl_alloc_deviceid_node, and that registers the device's PR key). Then it takes the nfs4_deviceid_lock and looks up the deviceid again. If it finds it this time, bl_find_get_deviceid() frees the spare (new) device_info, which unregisters the PR key for the same device. Any subsequent I/O from this client on that device gets EBADE. The umount later unregisters the device's PR key again. To prevent this problem, register the PR key after the deviceid_node lookup. Signed-off-by: Chuck Lever --- fs/nfs/blocklayout/blocklayout.c | 13 +++++++++-- fs/nfs/blocklayout/blocklayout.h | 8 ++++++- fs/nfs/blocklayout/dev.c | 39 +++++++++++++++++++++++--------- 3 files changed, 46 insertions(+), 14 deletions(-) diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index 6be13e0ec170..947b2c523440 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -564,6 +564,7 @@ bl_find_get_deviceid(struct nfs_server *server, gfp_t gfp_mask) { struct nfs4_deviceid_node *node; + struct pnfs_block_dev *d; unsigned long start, end; retry: @@ -571,9 +572,16 @@ bl_find_get_deviceid(struct nfs_server *server, if (!node) return ERR_PTR(-ENODEV); - if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags) == 0) - return node; + if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags)) + goto transient; + d = container_of(node, struct pnfs_block_dev, node); + if (d->pr_register) + if (!d->pr_register(d)) + goto out_put; + return node; + +transient: end = jiffies; start = end - PNFS_DEVICE_RETRY_TIMEOUT; if (!time_in_range(node->timestamp_unavailable, start, end)) { @@ -581,6 +589,7 @@ bl_find_get_deviceid(struct nfs_server *server, goto retry; } +out_put: nfs4_put_deviceid_node(node); return ERR_PTR(-ENODEV); } diff --git a/fs/nfs/blocklayout/blocklayout.h b/fs/nfs/blocklayout/blocklayout.h index f1eeb4914199..cc788e8ce909 100644 --- a/fs/nfs/blocklayout/blocklayout.h +++ b/fs/nfs/blocklayout/blocklayout.h @@ -110,12 +110,18 @@ struct pnfs_block_dev { struct file *bdev_file; u64 disk_offset; + unsigned long flags; u64 pr_key; - bool pr_registered; bool (*map)(struct pnfs_block_dev *dev, u64 offset, struct pnfs_block_dev_map *map); + bool (*pr_register)(struct pnfs_block_dev *dev); +}; + +/* pnfs_block_dev flag bits */ +enum { + PNFS_BDEV_REGISTERED = 0, }; /* sector_t fields are all in 512-byte sectors */ diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 519c310c745d..83753a08a19d 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -23,9 +23,9 @@ bl_free_device(struct pnfs_block_dev *dev) bl_free_device(&dev->children[i]); kfree(dev->children); } else { - if (dev->pr_registered) { - const struct pr_ops *ops = - file_bdev(dev->bdev_file)->bd_disk->fops->pr_ops; + if (test_and_clear_bit(PNFS_BDEV_REGISTERED, &dev->flags)) { + struct block_device *bdev = file_bdev(dev->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; int error; error = ops->pr_register(file_bdev(dev->bdev_file), @@ -226,6 +226,30 @@ static bool bl_map_stripe(struct pnfs_block_dev *dev, u64 offset, return true; } +/** + * bl_pr_register_scsi - Register a SCSI PR key for @d + * @dev: pNFS block device, key to register is already in @d->pr_key + * + * Returns true if the device's PR key is registered, otherwise false. + */ +static bool bl_pr_register_scsi(struct pnfs_block_dev *dev) +{ + struct block_device *bdev = file_bdev(dev->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; + int status; + + if (test_and_set_bit(PNFS_BDEV_REGISTERED, &dev->flags)) + return true; + + status = ops->pr_register(bdev, 0, dev->pr_key, true); + if (status) { + pr_err("pNFS: failed to register key for block device %s.", + bdev->bd_disk->disk_name); + return false; + } + return true; +} + static int bl_parse_deviceid(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask); @@ -367,14 +391,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - error = ops->pr_register(file_bdev(d->bdev_file), 0, d->pr_key, true); - if (error) { - pr_err("pNFS: failed to register key for block device %s.", - file_bdev(d->bdev_file)->bd_disk->disk_name); - goto out_blkdev_put; - } - - d->pr_registered = true; + d->pr_register = bl_pr_register_scsi; return 0; out_blkdev_put: From patchwork Fri Jun 21 16:22:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13707816 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5DE7757F8 for ; Fri, 21 Jun 2024 16:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986966; cv=none; b=dtNBD639NBb94lXr87hfYsggDEcyXtSxJhxS/wP/H+KrQcMjEGWc1FYZ2daAjcpT2tnd2snV6G9DtMJjASvRlNU7jbhowzdWdFWcgrH9Zln5g1lAaR6vuORnPpuxdQ9/8rZcaOwc6wlaLwr1j8iSZWMOHT4mhrZPMKohcTzKnxs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986966; c=relaxed/simple; bh=zJ2XQCK0MP3WAtBTnxZaYNqZ+n/5Y3zGhBDNbYIwEn4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Yu2I07nKU4UqhUSe17Z2wWtlMCnWR6es6mJ4p7EUh+xT6qhdjYn6xJlLow08u0P7jzK9PrTUcQRuExtrJRwcEm3jcCKpGelwUdppJYpamihb8+Jl0x/Ag5/FjaOZcrx9FU8Ev5L3EP/iByn+5FdtxiVKgxwA0yhNYUCYtZxoZAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HZBz+q/7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HZBz+q/7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E4A5C2BBFC; Fri, 21 Jun 2024 16:22:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718986966; bh=zJ2XQCK0MP3WAtBTnxZaYNqZ+n/5Y3zGhBDNbYIwEn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HZBz+q/7eG0PzDOmU3+x/Qjb/NR7JXi6nL3GYaX+RJoziDxcW6baAm3U91gIaCSy1 eDZklLdrZh5zA0eR3RuPrikojmsxJXNjm/53X7f35oKUFsUlAqrGxCFVkrfCE6Fe8u QasyOEERrc9XnPcYs3lzViiIDviB62gYGOXMIGfOzPa/b0PaiyPLT58lJOhGbmzS8i BH07yhqSneZEaUymM516cHp5/rQC4Img1vXFx+1a+akjaVnJvXch8tRXzKY6pZ8Cwe JUkXXqPhPpETu/EupDjwwILez3yfAzOiF9IGujxcpvK7NRDzgpLO2H1X8AcrAcb8cf 6v/nXmCX43qDg== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [PATCH v2 2/4] nfs/blocklayout: Use bulk page allocation APIs Date: Fri, 21 Jun 2024 12:22:30 -0400 Message-ID: <20240621162227.215412-8-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240621162227.215412-6-cel@kernel.org> References: <20240621162227.215412-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1730; i=chuck.lever@oracle.com; h=from:subject; bh=64CPDNNydmmWUps9bRG2EqBzhLBVti4LBhnx7kukaOQ=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmdajIifVBaxsBLvS44MUM7iy6e1jrCUVyxCgw8 ZVuIdrS49aJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnWoyAAKCRAzarMzb2Z/ lzI0D/4gn9+PFchmuYIEcXhpZF5hTjcLsUnp8DMFZlquas3S19p/C9i+rMnPhWc01Pu3NCtueSH tflAI77za3hlZyBC6M/HRD/LutKDZ2UP7e7PhDbtqT7kQPmuSEaTZdzI1OhhCyaKgN3TlH9dQ3X 2ZcdZDBHUHe8XOyh0LnmYgncSbBprnkyB/1Yn/CpW2Zg8LfxRKqWRxL+pgH7ZLxdMWs/VTxx5gk tjnrz3ZTbkE3mYt7/efq9IPQB5Zc9J7BpVoNPnYPOvxM2gsRUXQgfY5RSYnPHa0aBqk5oaNL0ZT ZuAHPE6aHCL9WsTFGzC8/LPAwNLrAfaSzaHpTooshErax5H5aECnn4FObCuVH9CsVgloyV0Fgjq 8nBCI0J/vMDJrCW+zvPHhpvTRFCw6CtYiXlRjgJPkYjI7bx23dDNFG+nPF2aErawHKLaKvCvVfz 6FKKR9XrYRHRc5GTNaBRa4r1fPDzV5kIDgOfk1pmEOlhbT1G95SVf2LCHIa/GgxH8gNSzMOoVno sHYrlxRWqg1BllZYDxH/4exBSbvfK91xGIRzVezBQ5NBPgpuG+ZvRLTSSrmn+fDEcn8TkogMRfC qaFgfXl//IppzVOcnzREs/ko9tAf/3y5sbP+ftt+7DEnMgEBK1aiHy9gbCj9VIsWYNnWWzgYrmx U0c0vRKXuiJq2KQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever nfs4_get_device_info() frequently requests more than a few pages when provisioning a nfs4_deviceid_node object. Make this more efficient by using alloc_pages_bulk_array(). This API is known to be several times faster than an open-coded loop around alloc_page(). release_pages() is folio-enabled so it is also more efficient than repeatedly invoking __free_pages(). Signed-off-by: Chuck Lever Reviewed-by: Christoph Hellwig --- fs/nfs/pnfs_dev.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/fs/nfs/pnfs_dev.c b/fs/nfs/pnfs_dev.c index 178001c90156..26a78d69acab 100644 --- a/fs/nfs/pnfs_dev.c +++ b/fs/nfs/pnfs_dev.c @@ -101,9 +101,8 @@ nfs4_get_device_info(struct nfs_server *server, struct nfs4_deviceid_node *d = NULL; struct pnfs_device *pdev = NULL; struct page **pages = NULL; + int rc, i, max_pages; u32 max_resp_sz; - int max_pages; - int rc, i; /* * Use the session max response size as the basis for setting @@ -125,11 +124,9 @@ nfs4_get_device_info(struct nfs_server *server, if (!pages) goto out_free_pdev; - for (i = 0; i < max_pages; i++) { - pages[i] = alloc_page(gfp_flags); - if (!pages[i]) - goto out_free_pages; - } + i = alloc_pages_bulk_array(GFP_KERNEL, max_pages, pages); + if (i != max_pages) + goto out_free_pages; memcpy(&pdev->dev_id, dev_id, sizeof(*dev_id)); pdev->layout_type = server->pnfs_curr_ld->id; @@ -154,8 +151,8 @@ nfs4_get_device_info(struct nfs_server *server, set_bit(NFS_DEVICEID_NOCACHE, &d->flags); out_free_pages: - while (--i >= 0) - __free_page(pages[i]); + if (i) + release_pages(pages, i); kfree(pages); out_free_pdev: kfree(pdev); From patchwork Fri Jun 21 16:22:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13707817 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0AF5615FA8E for ; Fri, 21 Jun 2024 16:22:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986968; cv=none; b=I3mEIbA7BMlq76bXXyP0QICsIVaTXiyvsdMeSOS8YFoCrxUKiBOVOgckKhE9fzxmTPklueXJEBW4RiOfIm0xGpgGxFYqeIyPvU8jotLg5s4dXOX3mxYfNdPXxbr2cJYqy6ug3Ba1PPYgOPVrM9LNkG6BlmAbVAEhOKW9U4uSA+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986968; c=relaxed/simple; bh=XxHBKFUgCKg2xyacrqzKz+eonu2yKQTOyCo1Zf4Bm6Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PkkypS+de2QLTNwePaWAeT12JcK1mOuy12TaqB4SONV+VUERfeJIEGMVQlOmkQgxObBI8lqAXwZhr16x3dyxs058obYaJUUUTAkCBGF3bdKATO7m5djVBMZ8ZstXn5AgsaokV3zpwFxxUYd6/s+DFRvKkEHoB117SAt1eXy/hdM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IibhThhU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IibhThhU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8BBB7C3277B; Fri, 21 Jun 2024 16:22:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718986967; bh=XxHBKFUgCKg2xyacrqzKz+eonu2yKQTOyCo1Zf4Bm6Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IibhThhUrRBC7fY4DW3kXilVAJG7pA9ALdlcUZUA3M0BbEs0maXqvTrrTNH7IbFyz vo/AiyfEYp3ekj0roH0mny7WBRQzWCGXA3Vo96lRd2dZdTEyu0in9+7MCe4L1PNfb3 /eZ6TIumnWst1wA6t09ea2jcl/SK1K82l070hDIA73zTDU18Ab0FyaQb3iEjFrHNKk k/UjmWR7gI9Vk6su90WgvcEpy78fOLanH9yGoqkUivJp/7uv2EdEu9mkBfC1t+jWHP VzqSg11Dw/YJ2Q1X7bfogD0WzxkVy7UmZg3r2sTjHU2dzK1D2YWD6uey28mqQZn2oc LC/GvIkOYGekw== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [PATCH v2 3/4] nfs/blocklayout: Report only when /no/ device is found Date: Fri, 21 Jun 2024 12:22:31 -0400 Message-ID: <20240621162227.215412-9-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240621162227.215412-6-cel@kernel.org> References: <20240621162227.215412-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1766; i=chuck.lever@oracle.com; h=from:subject; bh=JXCWdR0tBvdkJh40ZArpq43pt9oBHejJ+7HeGkgKvPw=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmdajIp4dVUnDsv7nPQuod8ZNFjVCA1BfiJOQUm HAghjG59lyJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnWoyAAKCRAzarMzb2Z/ l951EACAwg8sJ2da9gEYbvlXaHFqekHiB/P/4Ewfkvj85d7eRvMjLQPF8KmQ6/gOoN/Xbz1/fSi 4aWMa+koU9n7lhy6MoLxdRAlmYJpBrCEJ83uPGzvxft2/nTGDX5QX2IcOWZeedonmfg8AiwUbC/ EQmlw/xHbU5ExMXnwUcciw+GJp167a2Qtl3Z3OfVFK2HqtxWvY9N4jNwIS6LU9i0b2mMnhFrvRM ra6OVe8on9qRbASqA69GXFdQqa0XUwO/AoulZuh61f8Se8ce6297BpQWMh9msNwXlDnA047xiAV UsSCJBYyMd3BY2GmT7QEccn7rPP3a+0enQeW3ntpkwM+WgeFcYqzHroa+YtDfi7e1Jsa7vWYKuQ +CGkmL6f2EGalVsLr5Gf+deBicsf9SpEM91p0BUUg9cZUoFHW/05KO6NHUvFuNoemRXzJWaLeKI cqwlVn8dgsOmdmUgLPA+j6eoQidHJSuY3vDWyQgh0snWamk0VZ0Q0FAsDveCC2+fCBlIHA8D87J yPoG2KjItUlhC0gitJKVE/ji+6eBkJCVbV7H2SEKqRMWoGTRiglqFgA93RVnN5i12xbJ8e/e2xl mtrWmVPuYYv9rVbLRYUJQxUkXY84wADcqGylnRPgWxfuzyaKyOXwLQBESXZ7EOYmiPww+3VXq1M k/r1Gmly++74ecA== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever Since commit f931d8374cad ("nfs/blocklayout: refactor block device opening"), an error is reported when no multi-path device is found. But this isn't a fatal error if the subsequent device open is successful. On systems without multi-path devices, this message always appears whether there is a problem or not. Instead, generate less system journal noise by reporting an error only when both open attempts fail. The new error message is more actionable since it indicates that there is a real configuration issue to be addressed. Reviewed-by: Christoph Hellwig Signed-off-by: Chuck Lever --- fs/nfs/blocklayout/dev.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 83753a08a19d..568f685dee4b 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -338,7 +338,7 @@ bl_open_path(struct pnfs_block_volume *v, const char *prefix) bdev_file = bdev_file_open_by_path(devname, BLK_OPEN_READ | BLK_OPEN_WRITE, NULL, NULL); if (IS_ERR(bdev_file)) { - pr_warn("pNFS: failed to open device %s (%ld)\n", + dprintk("failed to open device %s (%ld)\n", devname, PTR_ERR(bdev_file)); } @@ -367,8 +367,11 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, bdev_file = bl_open_path(v, "dm-uuid-mpath-0x"); if (IS_ERR(bdev_file)) bdev_file = bl_open_path(v, "wwn-0x"); - if (IS_ERR(bdev_file)) + if (IS_ERR(bdev_file)) { + pr_warn("pNFS: no device found for volume %*phN\n", + v->scsi.designator_len, v->scsi.designator); return PTR_ERR(bdev_file); + } d->bdev_file = bdev_file; d->len = bdev_nr_bytes(file_bdev(d->bdev_file)); From patchwork Fri Jun 21 16:22:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13707818 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65FD0757F8 for ; Fri, 21 Jun 2024 16:22:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986969; cv=none; b=WIqxX+5xYdfLB/MA+0tNaPrxmV8kDwLFVCl81HoElSfQ0YiJuTwWEHC1AlLya9OLIj6YSM6uON/yEvED018j+lhwiG5kwatG0uWMQTmGKNVcFYjMemVEAmA8SEap8Kr2JC/q2LjqKnbdaapdiobnxhleN8lHjf98cRVLbSQuOuA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718986969; c=relaxed/simple; bh=3AyN7eBGp/n6ypHxMBe6DgNA5mGNClFo9oRXpD8FlGI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NBdSvoT9vrz1X19PCE3retSOz7P13lhPHzdfoHJzPTFsaXHKmcbwVrSHK4y/65vPrhOTRyrgab1QK6n7ePrVqjUCL1IbkcKAtvTubPP2QXQh0Fqx588+fFbzhaM0NNajmBw6j9XiOnRt/a2n9ROEUITLXB5KIz0xbIPWFNJfb4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iJ3ZaC/Z; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iJ3ZaC/Z" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7C77C2BBFC; Fri, 21 Jun 2024 16:22:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718986969; bh=3AyN7eBGp/n6ypHxMBe6DgNA5mGNClFo9oRXpD8FlGI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iJ3ZaC/ZXuiYKcwZszKAwQ8mFeNj5BIxMVf0S1KQGzzyFZPMvJaPjoyqYlkNYfej4 1QqsUPcX/1sqfXKBkAoIzDfWlWM3XH+O7QfmOdlkR6UanSzPbtvydvy4Inp86h65u3 MdNIYkTf8AMLbairpM0jkIRg4/fjRTz9YVhvIq+xGzszt44L2MWlPkuz3ObEqysxtL Br8uHttdLfCShE5IzGbwnlBMHl9BQ50e6us3Xg45fk7A7gKIfGqmTYFVkpMVpjzSXH jv1988cnBrR2HGsgs1ny/zXQvMyXZc8Gsn64a6BMVXzipOIfnaCtG8mfxagsX/aZzR f60goZjOKx8yA== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [PATCH v2 4/4] nfs/blocklayout: SCSI layout trace points for reservation key reg/unreg Date: Fri, 21 Jun 2024 12:22:32 -0400 Message-ID: <20240621162227.215412-10-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240621162227.215412-6-cel@kernel.org> References: <20240621162227.215412-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7677; i=chuck.lever@oracle.com; h=from:subject; bh=NlKYBrxzhnWHDbaYiUYQu2l0Pz3FH1hQafDBlBBUiGY=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmdajJW5zBQ1H0pr6CmxN0ujkGVzAdx5BkbUkZs DrPvMeWwpaJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnWoyQAKCRAzarMzb2Z/ l2RDD/4vQHoyNxuncMPRbY+uuvavSeOs4d2liNP3VkmOKOB8yuP40cKve0ohGRyXOoQgeR6n5WV VeIVx7Ml3UVKe/dN2htf9etvqR/KzPFyL57Q/UBBbqG9bAVRl22jH5++78/sQxlnF6+GF4GVOtN XCDMZxlQL0AaSEdWwMYVPzyPBA345CS2hqHBkMoK52ZwvDrjUc4VMkcNJQHzQ/j+lx86+PpLZ10 zjCnr23MXvte9vbV9/Z9QK1N3BE+P25YIT0NeGgBDL8fSeZhVlZVUhVW3eyqEMM34hOXKfIN/iB ZHEZ0L5mOVXnot1UH/e058S+OcKDD+TGp1cyppOG6PdPKYiNQnuNOsX13UyrrMQ0vjTc43CNUlQ fiOhg8a2yDJSoepHBRfPgNjk5rrB7wsb1/kzXe4+ytH6eZXYoter8qyRXYneVnPiX0BydcyOxb1 WE81kdFCvTnR3i5Z0dkRhkWipNBNQR5QhGZ9yAKOWxE2SM82DRYJc/JWwvgWNkz6kvWRatlhccN WgEPe4GH6YcN+zkLncKqj+uv0338D2A7TP+OkyBDj7bZEooN6Uf80WjNEFjuvwzc1nlyFO/oWGS 1L9q+DtdjjsDaL8tdKysLhL6kwYeY4BuOJtik2AuATUw476+3ZKqgwNVfxI9TsmFAg1+NiiDSfd KWtWJVy82lhe9VQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever An administrator cannot take action on these messages, but the reported errors might be helpful for troubleshooting. Transition them to trace points so these events appear in the trace log and can be easily lined up with other traced NFS client operations. Examples: append_writer-6147 [000] 80.247393: bl_pr_key_reg: device=sdb key=0x666dcdabf29514fe append_writer-6147 [000] 80.247842: bl_pr_key_unreg: device=sdb key=0x666dcdabf29514fe umount.nfs4-6172 [002] 84.950409: bl_pr_key_unreg_err: device=sdb key=0x666dcdabf29514fe error=24 Christoph points out that: > ... Note that the disk_name isn't really what > we'd want to trace anyway, as it misses the partition information. > The normal way to print the device name is the %pg printk specifier, > but I'm not sure how to correctly use that for tracing which wants > a string in the entry for binary tracing. The trace points copy the pr_info() that they are replacing, and show only the parent device name and not the partition. I'm still looking into how to record both parts of the device name. Signed-off-by: Chuck Lever Reviewed-by: Christoph Hellwig --- fs/nfs/blocklayout/dev.c | 30 +++++++------- fs/nfs/nfs4trace.c | 7 ++++ fs/nfs/nfs4trace.h | 88 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 111 insertions(+), 14 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 568f685dee4b..6c5d290ca81d 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -10,6 +10,7 @@ #include #include "blocklayout.h" +#include "../nfs4trace.h" #define NFSDBG_FACILITY NFSDBG_PNFS_LD @@ -26,12 +27,14 @@ bl_free_device(struct pnfs_block_dev *dev) if (test_and_clear_bit(PNFS_BDEV_REGISTERED, &dev->flags)) { struct block_device *bdev = file_bdev(dev->bdev_file); const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; - int error; + int status; - error = ops->pr_register(file_bdev(dev->bdev_file), - dev->pr_key, 0, false); - if (error) - pr_err("failed to unregister PR key.\n"); + status = ops->pr_register(bdev, dev->pr_key, 0, false); + if (status) + trace_bl_pr_key_unreg_err(bdev, dev->pr_key, + status); + else + trace_bl_pr_key_unreg(bdev, dev->pr_key); } if (dev->bdev_file) @@ -243,10 +246,10 @@ static bool bl_pr_register_scsi(struct pnfs_block_dev *dev) status = ops->pr_register(bdev, 0, dev->pr_key, true); if (status) { - pr_err("pNFS: failed to register key for block device %s.", - bdev->bd_disk->disk_name); + trace_bl_pr_key_reg_err(bdev, dev->pr_key, status); return false; } + trace_bl_pr_key_reg(bdev, dev->pr_key); return true; } @@ -351,8 +354,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask) { struct pnfs_block_volume *v = &volumes[idx]; - struct file *bdev_file; + struct block_device *bdev; const struct pr_ops *ops; + struct file *bdev_file; int error; if (!bl_validate_designator(v)) @@ -373,8 +377,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, return PTR_ERR(bdev_file); } d->bdev_file = bdev_file; + bdev = file_bdev(bdev_file); - d->len = bdev_nr_bytes(file_bdev(d->bdev_file)); + d->len = bdev_nr_bytes(bdev); d->map = bl_map_simple; d->pr_key = v->scsi.pr_key; @@ -383,13 +388,10 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - pr_info("pNFS: using block device %s (reservation key 0x%llx)\n", - file_bdev(d->bdev_file)->bd_disk->disk_name, d->pr_key); - - ops = file_bdev(d->bdev_file)->bd_disk->fops->pr_ops; + ops = bdev->bd_disk->fops->pr_ops; if (!ops) { pr_err("pNFS: block device %s does not support reservations.", - file_bdev(d->bdev_file)->bd_disk->disk_name); + bdev->bd_disk->disk_name); error = -EINVAL; goto out_blkdev_put; } diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c index d22c6670f770..389941ccc9c9 100644 --- a/fs/nfs/nfs4trace.c +++ b/fs/nfs/nfs4trace.c @@ -2,6 +2,8 @@ /* * Copyright (c) 2013 Trond Myklebust */ +#include +#include #include #include "nfs4_fs.h" #include "internal.h" @@ -29,5 +31,10 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_write_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_commit_error); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg_err); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg_err); + EXPORT_TRACEPOINT_SYMBOL_GPL(fl_getdevinfo); #endif diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index 4de8780a7c48..f2090a491fcb 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -2153,6 +2153,94 @@ TRACE_EVENT(ff_layout_commit_error, ) ); +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_class, + TP_PROTO( + const struct block_device *bdev, + u64 key + ), + TP_ARGS(bdev, key), + TP_STRUCT__entry( + __field(u64, key) + __field(dev_t, dev) + __string(device, bdev->bd_disk->disk_name) + ), + TP_fast_assign( + __entry->key = key; + __entry->dev = bdev->bd_dev; + __assign_str(device); + ), + TP_printk("dev=%d,%d (%s) key=0x%016llx", + MAJOR(__entry->dev), MINOR(__entry->dev), + __get_str(device), __entry->key + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_class, name, \ + TP_PROTO( \ + const struct block_device *bdev, \ + u64 key \ + ), \ + TP_ARGS(bdev, key)) +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_reg); +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_unreg); + +/* + * From uapi/linux/pr.h + */ +TRACE_DEFINE_ENUM(PR_STS_SUCCESS); +TRACE_DEFINE_ENUM(PR_STS_IOERR); +TRACE_DEFINE_ENUM(PR_STS_RESERVATION_CONFLICT); +TRACE_DEFINE_ENUM(PR_STS_RETRY_PATH_FAILURE); +TRACE_DEFINE_ENUM(PR_STS_PATH_FAST_FAILED); +TRACE_DEFINE_ENUM(PR_STS_PATH_FAILED); + +#define show_pr_status(x) \ + __print_symbolic(x, \ + { PR_STS_SUCCESS, "SUCCESS" }, \ + { PR_STS_IOERR, "IOERR" }, \ + { PR_STS_RESERVATION_CONFLICT, "RESERVATION_CONFLICT" }, \ + { PR_STS_RETRY_PATH_FAILURE, "RETRY_PATH_FAILURE" }, \ + { PR_STS_PATH_FAST_FAILED, "PATH_FAST_FAILED" }, \ + { PR_STS_PATH_FAILED, "PATH_FAILED" }) + +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_err_class, + TP_PROTO( + const struct block_device *bdev, + u64 key, + int status + ), + TP_ARGS(bdev, key, status), + TP_STRUCT__entry( + __field(u64, key) + __field(dev_t, dev) + __field(unsigned long, status) + __string(device, bdev->bd_disk->disk_name) + ), + TP_fast_assign( + __entry->key = key; + __entry->dev = bdev->bd_dev; + __entry->status = status; + __assign_str(device); + ), + TP_printk("dev=%d,%d (%s) key=0x%016llx status=%s", + MAJOR(__entry->dev), MINOR(__entry->dev), + __get_str(device), __entry->key, + show_pr_status(__entry->status) + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_err_class, name, \ + TP_PROTO( \ + const struct block_device *bdev, \ + u64 key, \ + int status \ + ), \ + TP_ARGS(bdev, key, status)) +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_reg_err); +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_unreg_err); + #ifdef CONFIG_NFS_V4_2 TRACE_DEFINE_ENUM(NFS4_CONTENT_DATA); TRACE_DEFINE_ENUM(NFS4_CONTENT_HOLE);