| Message ID | 169513768769.145733.5037542987990908432.stgit@manet.1015granger.net (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v1] SUNRPC: Remove BUG_ON call sites | expand |
On Wed, 20 Sep 2023, Chuck Lever wrote: > From: Chuck Lever <chuck.lever@oracle.com> > > There is no need to take down the whole system for these assertions. > > I'd rather not attempt a heroic save here, as some bug has occurred > that has left the transport data structures in an unknown state. > Just warn and then leak the left-over resources. > > Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > Acked-by: Christian Brauner <brauner@kernel.org> > --- > net/sunrpc/svc.c | 9 +++++---- > 1 file changed, 5 insertions(+), 4 deletions(-) > > Changes since v1: > - Use WARN_ONCE() instead of pr_warn() > > diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c > index 587811a002c9..3237f7dfde1e 100644 > --- a/net/sunrpc/svc.c > +++ b/net/sunrpc/svc.c > @@ -575,11 +575,12 @@ svc_destroy(struct kref *ref) > timer_shutdown_sync(&serv->sv_temptimer); > > /* > - * The last user is gone and thus all sockets have to be destroyed to > - * the point. Check this. > + * Remaining transports at this point are not expected. > */ > - BUG_ON(!list_empty(&serv->sv_permsocks)); > - BUG_ON(!list_empty(&serv->sv_tempsocks)); > + WARN_ONCE(!list_empty(&serv->sv_permsocks), > + "SVC: permsocks remain for %s\n", serv->sv_program->pg_name); > + WARN_ONCE(!list_empty(&serv->sv_tempsocks), > + "SVC: tempsocks remain for %s\n", serv->sv_program->pg_name); > > cache_clean_deferred(serv); > > Reviewed-by: NeilBrown <neilb@suse.de> The stack trace might not be helpful, but this circumstance really really shouldn't happen so if it ever does, I think we really want as much context as practicable. Thanks, NeilBrown
On 9/20/23 00:11, NeilBrown wrote: > On Wed, 20 Sep 2023, Chuck Lever wrote: >> From: Chuck Lever <chuck.lever@oracle.com> >> >> There is no need to take down the whole system for these assertions. >> >> I'd rather not attempt a heroic save here, as some bug has occurred >> that has left the transport data structures in an unknown state. >> Just warn and then leak the left-over resources. >> >> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> >> Acked-by: Christian Brauner <brauner@kernel.org> >> --- >> net/sunrpc/svc.c | 9 +++++---- >> 1 file changed, 5 insertions(+), 4 deletions(-) >> >> Changes since v1: >> - Use WARN_ONCE() instead of pr_warn() >> >> diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c >> index 587811a002c9..3237f7dfde1e 100644 >> --- a/net/sunrpc/svc.c >> +++ b/net/sunrpc/svc.c >> @@ -575,11 +575,12 @@ svc_destroy(struct kref *ref) >> timer_shutdown_sync(&serv->sv_temptimer); >> >> /* >> - * The last user is gone and thus all sockets have to be destroyed to >> - * the point. Check this. >> + * Remaining transports at this point are not expected. >> */ >> - BUG_ON(!list_empty(&serv->sv_permsocks)); >> - BUG_ON(!list_empty(&serv->sv_tempsocks)); >> + WARN_ONCE(!list_empty(&serv->sv_permsocks), >> + "SVC: permsocks remain for %s\n", serv->sv_program->pg_name); >> + WARN_ONCE(!list_empty(&serv->sv_tempsocks), >> + "SVC: tempsocks remain for %s\n", serv->sv_program->pg_name); >> >> cache_clean_deferred(serv); >> >> > > Reviewed-by: NeilBrown <neilb@suse.de> > > The stack trace might not be helpful, but this circumstance really > really shouldn't happen so if it ever does, I think we really want as > much context as practicable. I've just wanted to leave a small note, that we've hit this recently (with 6.6.12, so BUG_ON). Before we've hit the BUG_ON, we've run some performance tests, which involved changing the number of nfsd threads multiple times. After a period without problems, these changes started to fail: 2024-06-19T15:52:22+02:00 wayofthedodo rpc.nfsd[4145]: error starting threads: errno 12 (Cannot allocate memory) 2024-06-19T15:54:05+02:00 wayofthedodo rpc.nfsd[4393]: error starting threads: errno 12 (Cannot allocate memory) 2024-06-19T15:54:19+02:00 wayofthedodo rpc.nfsd[4424]: error starting threads: errno 12 (Cannot allocate memory) 2024-06-19T15:54:40+02:00 wayofthedodo rpc.nfsd[4476]: error starting threads: errno 12 (Cannot allocate memory) 2024-06-19T15:55:18+02:00 wayofthedodo rpc.nfsd[4558]: error starting threads: errno 12 (Cannot allocate memory) Which is - looking at nfs-utils source - ENOMEM from the open or write of /proc/fs/nfsd/threads. ( Or a truncated write to that file, which nfs-utils would report with a random (unset) errno value, but I don't think this can happen with the nfsd filesystem ) Anyway. then 'systemctl restart nfsd' was attempted, which in our setting stops and starts mountd as the service process with Pre and Post commands to stop and start the nfsd: ExecStartPre=/usr/sbin/exportfs -ra ExecStart=/usr/sbin/rpc.mountd --foreground --manage-gids ExecStartPost=/usr/sbin/rpc.nfsd --lease-time 90 --grace-time 90 --no-nfs-version 3 8 ExecStartPost=bash -c "(sleep 10;exportfs -r;sleep 20;exportfs -r;sleep 30;exportfs -r)&" ExecReload=/usr/sbin/exportfs -ra ExecStopPost=/usr/sbin/rpc.nfsd 0 ; /usr/sbin/exportfs -ua Restart=always This resulted in: 2024-06-19T15:55:34+02:00 wayofthedodo rpc.mountd[807]: Caught signal 15, un-registering and exiting. 2024-06-19T15:55:34.561970+02:00 wayofthedodo kernel: [2502367.958818] nfsd: last server has exited, flushing export cache 2024-06-19T15:55:34+02:00 wayofthedodo rpc.mountd[4578]: Version 2.5.4 starting 2024-06-19T15:55:35.870933+02:00 wayofthedodo kernel: [2502369.261987] NFSD: Using UMH upcall client tracking operations. 2024-06-19T15:55:35.870949+02:00 wayofthedodo kernel: [2502369.268678] NFSD: starting 90-second grace period (net f0000000) 2024-06-19T15:55:35.887646+02:00 wayofthedodo kernel: [2502369.285013] ------------[ cut here ]------------ 2024-06-19T15:55:35.887662+02:00 wayofthedodo kernel: [2502369.291230] kernel BUG at net/sunrpc/svc.c:581! 2024-06-19T15:55:36.268015+02:00 wayofthedodo kernel: [2502369.297008] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI 2024-06-19T15:55:36.268028+02:00 wayofthedodo kernel: [2502369.303548] CPU: 9 PID: 4579 Comm: rpc.nfsd Not tainted 6.6.12.mx64.461 #1 2024-06-19T15:55:36.268031+02:00 wayofthedodo kernel: [2502369.311741] Hardware name: Dell Inc. PowerEdge T440/021KCD, BIOS 2.12.2 07/09/2021 2024-06-19T15:55:36.268034+02:00 wayofthedodo kernel: [2502369.320696] RIP: 0010:svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.268037+02:00 wayofthedodo kernel: [2502369.327474] Code: 00 00 00 be 01 00 00 00 e8 d4 f2 54 e1 41 3b 6d 74 72 bc 49 8b 7d 7c e8 95 40 1c e1 4c 89 e7 5b 5d 41 5c 41 5d e9 87 40 1c e1 <0f> 0b 48 8b 47 ec 48 c7 c7 f9 5a 15 a0 48 8b 70 20 e8 c1 87 01 e1 2024-06-19T15:55:36.268040+02:00 wayofthedodo kernel: [2502369.349863] RSP: 0018:ffffc9000e26bd60 EFLAGS: 00010206 2024-06-19T15:55:36.268043+02:00 wayofthedodo kernel: [2502369.356573] RAX: ffff88886064e130 RBX: ffff88886064e114 RCX: 0000000000000010 2024-06-19T15:55:36.268048+02:00 wayofthedodo kernel: [2502369.365173] RDX: ffff889092d73018 RSI: 0000000000000246 RDI: ffff88a03fc1cfc0 2024-06-19T15:55:36.268050+02:00 wayofthedodo kernel: [2502369.373879] RBP: 0000000000000040 R08: 000000000000000f R09: 0000000000000001 2024-06-19T15:55:36.268053+02:00 wayofthedodo kernel: [2502369.382474] R10: ffff889092d71000 R11: 0000000000000000 R12: ffff88886064e100 2024-06-19T15:55:36.268055+02:00 wayofthedodo kernel: [2502369.391115] R13: ffff88886064e114 R14: ffff88886064e100 R15: ffff8881061d6000 2024-06-19T15:55:36.268058+02:00 wayofthedodo kernel: [2502369.399730] FS: 00007f610ac30740(0000) GS:ffff88a03fd00000(0000) knlGS:0000000000000000 2024-06-19T15:55:36.268059+02:00 wayofthedodo kernel: [2502369.409410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-06-19T15:55:36.268062+02:00 wayofthedodo kernel: [2502369.416667] CR2: 000000000069adf8 CR3: 00000004ba14a002 CR4: 00000000007706e0 2024-06-19T15:55:36.268064+02:00 wayofthedodo kernel: [2502369.425524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2024-06-19T15:55:36.268066+02:00 wayofthedodo kernel: [2502369.434240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 2024-06-19T15:55:36.268068+02:00 wayofthedodo kernel: [2502369.442880] PKRU: 55555554 2024-06-19T15:55:36.268070+02:00 wayofthedodo kernel: [2502369.447193] Call Trace: 2024-06-19T15:55:36.268071+02:00 wayofthedodo kernel: [2502369.451211] <TASK> 2024-06-19T15:55:36.268073+02:00 wayofthedodo kernel: [2502369.454982] ? die+0x36/0x90 2024-06-19T15:55:36.268075+02:00 wayofthedodo kernel: [2502369.459421] ? do_trap+0xda/0x100 2024-06-19T15:55:36.268077+02:00 wayofthedodo kernel: [2502369.464337] ? svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.268079+02:00 wayofthedodo kernel: [2502369.470479] ? do_error_trap+0x65/0x80 2024-06-19T15:55:36.268080+02:00 wayofthedodo kernel: [2502369.475857] ? svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.268082+02:00 wayofthedodo kernel: [2502369.481924] ? exc_invalid_op+0x50/0x70 2024-06-19T15:55:36.268084+02:00 wayofthedodo kernel: [2502369.487390] ? svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.268099+02:00 wayofthedodo kernel: [2502369.493402] ? asm_exc_invalid_op+0x1a/0x20 2024-06-19T15:55:36.268131+02:00 wayofthedodo kernel: [2502369.498494] ? svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.268134+02:00 wayofthedodo kernel: [2502369.504826] nfsd_svc+0x28c/0x3d0 [nfsd] 2024-06-19T15:55:36.268136+02:00 wayofthedodo kernel: [2502369.510836] write_threads+0xe4/0x190 [nfsd] 2024-06-19T15:55:36.268138+02:00 wayofthedodo kernel: [2502369.517184] ? __pfx_write_threads+0x10/0x10 [nfsd] 2024-06-19T15:55:36.268140+02:00 wayofthedodo kernel: [2502369.524580] nfsctl_transaction_write+0x4a/0x80 [nfsd] 2024-06-19T15:55:36.268141+02:00 wayofthedodo kernel: [2502369.531495] vfs_write+0xcf/0x450 2024-06-19T15:55:36.268143+02:00 wayofthedodo kernel: [2502369.535578] ksys_write+0x6f/0xf0 2024-06-19T15:55:36.268145+02:00 wayofthedodo kernel: [2502369.540415] do_syscall_64+0x43/0x90 2024-06-19T15:55:36.268146+02:00 wayofthedodo kernel: [2502369.545455] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 2024-06-19T15:55:36.268148+02:00 wayofthedodo kernel: [2502369.551988] RIP: 0033:0x7f610ad3aa20 2024-06-19T15:55:36.268151+02:00 wayofthedodo kernel: [2502369.557030] Code: 40 00 48 8b 15 e1 b3 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d c1 3b 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89 2024-06-19T15:55:36.268153+02:00 wayofthedodo kernel: [2502369.578504] RSP: 002b:00007fff4d8deaf8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 2024-06-19T15:55:36.268155+02:00 wayofthedodo kernel: [2502369.587720] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f610ad3aa20 2024-06-19T15:55:36.268157+02:00 wayofthedodo kernel: [2502369.596419] RDX: 0000000000000003 RSI: 000000000040d540 RDI: 0000000000000003 2024-06-19T15:55:36.268160+02:00 wayofthedodo kernel: [2502369.604613] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fff4d8de990 2024-06-19T15:55:36.268163+02:00 wayofthedodo kernel: [2502369.613258] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000040 2024-06-19T15:55:36.268165+02:00 wayofthedodo kernel: [2502369.621276] R13: 0000000000000001 R14: 000000000040e2a0 R15: 000000000040910e 2024-06-19T15:55:36.268166+02:00 wayofthedodo kernel: [2502369.629927] </TASK> 2024-06-19T15:55:36.268171+02:00 wayofthedodo kernel: [2502369.632849] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs i915 iosf_mbi drm_buddy drm_display_helper ttm intel_gtt video 8021q garp stp mrp llc x86_pkg_temp_thermal coretemp kvm_intel tg3 kvm irqbypass crc32c_intel wmi_bmof mgag200 i2c_algo_bit libphy iTCO_wdt i40e iTCO_vendor_support wmi ipmi_si nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables ipv6 autofs4 2024-06-19T15:55:36.268181+02:00 wayofthedodo kernel: [2502369.672534] ---[ end trace 0000000000000000 ]--- 2024-06-19T15:55:36.278358+02:00 wayofthedodo kernel: [2502369.677557] RIP: 0010:svc_destroy+0xc9/0xf0 [sunrpc] 2024-06-19T15:55:36.278372+02:00 wayofthedodo kernel: [2502369.682931] Code: 00 00 00 be 01 00 00 00 e8 d4 f2 54 e1 41 3b 6d 74 72 bc 49 8b 7d 7c e8 95 40 1c e1 4c 89 e7 5b 5d 41 5c 41 5d e9 87 40 1c e1 <0f> 0b 48 8b 47 ec 48 c7 c7 f9 5a 15 a0 48 8b 70 20 e8 c1 87 01 e1 2024-06-19T15:55:36.297705+02:00 wayofthedodo kernel: [2502369.702288] RSP: 0018:ffffc9000e26bd60 EFLAGS: 00010206 2024-06-19T15:55:36.303328+02:00 wayofthedodo kernel: [2502369.707906] RAX: ffff88886064e130 RBX: ffff88886064e114 RCX: 0000000000000010 2024-06-19T15:55:36.310851+02:00 wayofthedodo kernel: [2502369.715430] RDX: ffff889092d73018 RSI: 0000000000000246 RDI: ffff88a03fc1cfc0 2024-06-19T15:55:36.325920+02:00 wayofthedodo kernel: [2502369.722960] RBP: 0000000000000040 R08: 000000000000000f R09: 0000000000000001 2024-06-19T15:55:36.325933+02:00 wayofthedodo kernel: [2502369.730483] R10: ffff889092d71000 R11: 0000000000000000 R12: ffff88886064e100 2024-06-19T15:55:36.340957+02:00 wayofthedodo kernel: [2502369.738015] R13: ffff88886064e114 R14: ffff88886064e100 R15: ffff8881061d6000 2024-06-19T15:55:36.340970+02:00 wayofthedodo kernel: [2502369.745537] FS: 00007f610ac30740(0000) GS:ffff88a03fd00000(0000) knlGS:0000000000000000 2024-06-19T15:55:36.349441+02:00 wayofthedodo kernel: [2502369.754015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-06-19T15:55:36.355576+02:00 wayofthedodo kernel: [2502369.760149] CR2: 000000000069adf8 CR3: 00000004ba14a002 CR4: 00000000007706e0 2024-06-19T15:55:36.363069+02:00 wayofthedodo kernel: [2502369.767681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2024-06-19T15:55:36.370634+02:00 wayofthedodo kernel: [2502369.775210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 2024-06-19T15:55:36.378121+02:00 wayofthedodo kernel: [2502369.782735] PKRU: 55555554 2024-06-19T15:55:36+02:00 wayofthedodo rpc.mountd[4578]: Caught signal 15, un-registering and exiting. Best Donald > > Thanks, > NeilBrown >
在 2024/6/27 20:53, Donald Buczek 写道: > On 9/20/23 00:11, NeilBrown wrote: >> On Wed, 20 Sep 2023, Chuck Lever wrote: >>> From: Chuck Lever <chuck.lever@oracle.com> >>> >>> There is no need to take down the whole system for these assertions. >>> >>> I'd rather not attempt a heroic save here, as some bug has occurred >>> that has left the transport data structures in an unknown state. >>> Just warn and then leak the left-over resources. >>> >>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> >>> Acked-by: Christian Brauner <brauner@kernel.org> >>> --- >>> net/sunrpc/svc.c | 9 +++++---- >>> 1 file changed, 5 insertions(+), 4 deletions(-) >>> >>> Changes since v1: >>> - Use WARN_ONCE() instead of pr_warn() >>> >>> diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c >>> index 587811a002c9..3237f7dfde1e 100644 >>> --- a/net/sunrpc/svc.c >>> +++ b/net/sunrpc/svc.c >>> @@ -575,11 +575,12 @@ svc_destroy(struct kref *ref) >>> timer_shutdown_sync(&serv->sv_temptimer); >>> >>> /* >>> - * The last user is gone and thus all sockets have to be destroyed to >>> - * the point. Check this. >>> + * Remaining transports at this point are not expected. >>> */ >>> - BUG_ON(!list_empty(&serv->sv_permsocks)); >>> - BUG_ON(!list_empty(&serv->sv_tempsocks)); >>> + WARN_ONCE(!list_empty(&serv->sv_permsocks), >>> + "SVC: permsocks remain for %s\n", serv->sv_program->pg_name); >>> + WARN_ONCE(!list_empty(&serv->sv_tempsocks), >>> + "SVC: tempsocks remain for %s\n", serv->sv_program->pg_name); >>> >>> cache_clean_deferred(serv); >>> >>> >> Reviewed-by: NeilBrown <neilb@suse.de> >> >> The stack trace might not be helpful, but this circumstance really >> really shouldn't happen so if it ever does, I think we really want as >> much context as practicable. > I've just wanted to leave a small note, that we've hit this recently (with 6.6.12, so BUG_ON). > > Before we've hit the BUG_ON, we've run some performance tests, which involved changing the number of nfsd threads multiple times. After a period without problems, these changes started to fail: > > 2024-06-19T15:52:22+02:00 wayofthedodo rpc.nfsd[4145]: error starting threads: errno 12 (Cannot allocate memory) > 2024-06-19T15:54:05+02:00 wayofthedodo rpc.nfsd[4393]: error starting threads: errno 12 (Cannot allocate memory) > 2024-06-19T15:54:19+02:00 wayofthedodo rpc.nfsd[4424]: error starting threads: errno 12 (Cannot allocate memory) > 2024-06-19T15:54:40+02:00 wayofthedodo rpc.nfsd[4476]: error starting threads: errno 12 (Cannot allocate memory) > 2024-06-19T15:55:18+02:00 wayofthedodo rpc.nfsd[4558]: error starting threads: errno 12 (Cannot allocate memory) > > Which is - looking at nfs-utils source - ENOMEM from the open or write of /proc/fs/nfsd/threads. ( Or a truncated write to that file, which nfs-utils would report with a random (unset) errno value, but I don't think this can happen with the nfsd filesystem ) > > Anyway. then 'systemctl restart nfsd' was attempted, which in our setting stops and starts mountd as the service process with Pre and Post commands to stop and start the nfsd: > > ExecStartPre=/usr/sbin/exportfs -ra > ExecStart=/usr/sbin/rpc.mountd --foreground --manage-gids > ExecStartPost=/usr/sbin/rpc.nfsd --lease-time 90 --grace-time 90 --no-nfs-version 3 8 > ExecStartPost=bash -c "(sleep 10;exportfs -r;sleep 20;exportfs -r;sleep 30;exportfs -r)&" > ExecReload=/usr/sbin/exportfs -ra > ExecStopPost=/usr/sbin/rpc.nfsd 0 ; /usr/sbin/exportfs -ua > Restart=always > > This resulted in: > > 2024-06-19T15:55:34+02:00 wayofthedodo rpc.mountd[807]: Caught signal 15, un-registering and exiting. > 2024-06-19T15:55:34.561970+02:00 wayofthedodo kernel: [2502367.958818] nfsd: last server has exited, flushing export cache > 2024-06-19T15:55:34+02:00 wayofthedodo rpc.mountd[4578]: Version 2.5.4 starting > 2024-06-19T15:55:35.870933+02:00 wayofthedodo kernel: [2502369.261987] NFSD: Using UMH upcall client tracking operations. > 2024-06-19T15:55:35.870949+02:00 wayofthedodo kernel: [2502369.268678] NFSD: starting 90-second grace period (net f0000000) > 2024-06-19T15:55:35.887646+02:00 wayofthedodo kernel: [2502369.285013] ------------[ cut here ]------------ > 2024-06-19T15:55:35.887662+02:00 wayofthedodo kernel: [2502369.291230] kernel BUG at net/sunrpc/svc.c:581! > 2024-06-19T15:55:36.268015+02:00 wayofthedodo kernel: [2502369.297008] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI > 2024-06-19T15:55:36.268028+02:00 wayofthedodo kernel: [2502369.303548] CPU: 9 PID: 4579 Comm: rpc.nfsd Not tainted 6.6.12.mx64.461 #1 > 2024-06-19T15:55:36.268031+02:00 wayofthedodo kernel: [2502369.311741] Hardware name: Dell Inc. PowerEdge T440/021KCD, BIOS 2.12.2 07/09/2021 > 2024-06-19T15:55:36.268034+02:00 wayofthedodo kernel: [2502369.320696] RIP: 0010:svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.268037+02:00 wayofthedodo kernel: [2502369.327474] Code: 00 00 00 be 01 00 00 00 e8 d4 f2 54 e1 41 3b 6d 74 72 bc 49 8b 7d 7c e8 95 40 1c e1 4c 89 e7 5b 5d 41 5c 41 5d e9 87 40 1c e1 <0f> 0b 48 8b 47 ec 48 c7 c7 f9 5a 15 a0 48 8b 70 20 e8 c1 87 01 e1 > 2024-06-19T15:55:36.268040+02:00 wayofthedodo kernel: [2502369.349863] RSP: 0018:ffffc9000e26bd60 EFLAGS: 00010206 > 2024-06-19T15:55:36.268043+02:00 wayofthedodo kernel: [2502369.356573] RAX: ffff88886064e130 RBX: ffff88886064e114 RCX: 0000000000000010 > 2024-06-19T15:55:36.268048+02:00 wayofthedodo kernel: [2502369.365173] RDX: ffff889092d73018 RSI: 0000000000000246 RDI: ffff88a03fc1cfc0 > 2024-06-19T15:55:36.268050+02:00 wayofthedodo kernel: [2502369.373879] RBP: 0000000000000040 R08: 000000000000000f R09: 0000000000000001 > 2024-06-19T15:55:36.268053+02:00 wayofthedodo kernel: [2502369.382474] R10: ffff889092d71000 R11: 0000000000000000 R12: ffff88886064e100 > 2024-06-19T15:55:36.268055+02:00 wayofthedodo kernel: [2502369.391115] R13: ffff88886064e114 R14: ffff88886064e100 R15: ffff8881061d6000 > 2024-06-19T15:55:36.268058+02:00 wayofthedodo kernel: [2502369.399730] FS: 00007f610ac30740(0000) GS:ffff88a03fd00000(0000) knlGS:0000000000000000 > 2024-06-19T15:55:36.268059+02:00 wayofthedodo kernel: [2502369.409410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > 2024-06-19T15:55:36.268062+02:00 wayofthedodo kernel: [2502369.416667] CR2: 000000000069adf8 CR3: 00000004ba14a002 CR4: 00000000007706e0 > 2024-06-19T15:55:36.268064+02:00 wayofthedodo kernel: [2502369.425524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > 2024-06-19T15:55:36.268066+02:00 wayofthedodo kernel: [2502369.434240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > 2024-06-19T15:55:36.268068+02:00 wayofthedodo kernel: [2502369.442880] PKRU: 55555554 > 2024-06-19T15:55:36.268070+02:00 wayofthedodo kernel: [2502369.447193] Call Trace: > 2024-06-19T15:55:36.268071+02:00 wayofthedodo kernel: [2502369.451211] <TASK> > 2024-06-19T15:55:36.268073+02:00 wayofthedodo kernel: [2502369.454982] ? die+0x36/0x90 > 2024-06-19T15:55:36.268075+02:00 wayofthedodo kernel: [2502369.459421] ? do_trap+0xda/0x100 > 2024-06-19T15:55:36.268077+02:00 wayofthedodo kernel: [2502369.464337] ? svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.268079+02:00 wayofthedodo kernel: [2502369.470479] ? do_error_trap+0x65/0x80 > 2024-06-19T15:55:36.268080+02:00 wayofthedodo kernel: [2502369.475857] ? svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.268082+02:00 wayofthedodo kernel: [2502369.481924] ? exc_invalid_op+0x50/0x70 > 2024-06-19T15:55:36.268084+02:00 wayofthedodo kernel: [2502369.487390] ? svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.268099+02:00 wayofthedodo kernel: [2502369.493402] ? asm_exc_invalid_op+0x1a/0x20 > 2024-06-19T15:55:36.268131+02:00 wayofthedodo kernel: [2502369.498494] ? svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.268134+02:00 wayofthedodo kernel: [2502369.504826] nfsd_svc+0x28c/0x3d0 [nfsd] > 2024-06-19T15:55:36.268136+02:00 wayofthedodo kernel: [2502369.510836] write_threads+0xe4/0x190 [nfsd] > 2024-06-19T15:55:36.268138+02:00 wayofthedodo kernel: [2502369.517184] ? __pfx_write_threads+0x10/0x10 [nfsd] > 2024-06-19T15:55:36.268140+02:00 wayofthedodo kernel: [2502369.524580] nfsctl_transaction_write+0x4a/0x80 [nfsd] > 2024-06-19T15:55:36.268141+02:00 wayofthedodo kernel: [2502369.531495] vfs_write+0xcf/0x450 > 2024-06-19T15:55:36.268143+02:00 wayofthedodo kernel: [2502369.535578] ksys_write+0x6f/0xf0 > 2024-06-19T15:55:36.268145+02:00 wayofthedodo kernel: [2502369.540415] do_syscall_64+0x43/0x90 > 2024-06-19T15:55:36.268146+02:00 wayofthedodo kernel: [2502369.545455] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 > 2024-06-19T15:55:36.268148+02:00 wayofthedodo kernel: [2502369.551988] RIP: 0033:0x7f610ad3aa20 > 2024-06-19T15:55:36.268151+02:00 wayofthedodo kernel: [2502369.557030] Code: 40 00 48 8b 15 e1 b3 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d c1 3b 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89 > 2024-06-19T15:55:36.268153+02:00 wayofthedodo kernel: [2502369.578504] RSP: 002b:00007fff4d8deaf8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 > 2024-06-19T15:55:36.268155+02:00 wayofthedodo kernel: [2502369.587720] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f610ad3aa20 > 2024-06-19T15:55:36.268157+02:00 wayofthedodo kernel: [2502369.596419] RDX: 0000000000000003 RSI: 000000000040d540 RDI: 0000000000000003 > 2024-06-19T15:55:36.268160+02:00 wayofthedodo kernel: [2502369.604613] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fff4d8de990 > 2024-06-19T15:55:36.268163+02:00 wayofthedodo kernel: [2502369.613258] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000040 > 2024-06-19T15:55:36.268165+02:00 wayofthedodo kernel: [2502369.621276] R13: 0000000000000001 R14: 000000000040e2a0 R15: 000000000040910e > 2024-06-19T15:55:36.268166+02:00 wayofthedodo kernel: [2502369.629927] </TASK> > 2024-06-19T15:55:36.268171+02:00 wayofthedodo kernel: [2502369.632849] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs i915 iosf_mbi drm_buddy drm_display_helper ttm intel_gtt video 8021q garp stp mrp llc x86_pkg_temp_thermal coretemp kvm_intel tg3 kvm irqbypass crc32c_intel wmi_bmof mgag200 i2c_algo_bit libphy iTCO_wdt i40e iTCO_vendor_support wmi ipmi_si nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables ipv6 autofs4 > 2024-06-19T15:55:36.268181+02:00 wayofthedodo kernel: [2502369.672534] ---[ end trace 0000000000000000 ]--- > 2024-06-19T15:55:36.278358+02:00 wayofthedodo kernel: [2502369.677557] RIP: 0010:svc_destroy+0xc9/0xf0 [sunrpc] > 2024-06-19T15:55:36.278372+02:00 wayofthedodo kernel: [2502369.682931] Code: 00 00 00 be 01 00 00 00 e8 d4 f2 54 e1 41 3b 6d 74 72 bc 49 8b 7d 7c e8 95 40 1c e1 4c 89 e7 5b 5d 41 5c 41 5d e9 87 40 1c e1 <0f> 0b 48 8b 47 ec 48 c7 c7 f9 5a 15 a0 48 8b 70 20 e8 c1 87 01 e1 > 2024-06-19T15:55:36.297705+02:00 wayofthedodo kernel: [2502369.702288] RSP: 0018:ffffc9000e26bd60 EFLAGS: 00010206 > 2024-06-19T15:55:36.303328+02:00 wayofthedodo kernel: [2502369.707906] RAX: ffff88886064e130 RBX: ffff88886064e114 RCX: 0000000000000010 > 2024-06-19T15:55:36.310851+02:00 wayofthedodo kernel: [2502369.715430] RDX: ffff889092d73018 RSI: 0000000000000246 RDI: ffff88a03fc1cfc0 > 2024-06-19T15:55:36.325920+02:00 wayofthedodo kernel: [2502369.722960] RBP: 0000000000000040 R08: 000000000000000f R09: 0000000000000001 > 2024-06-19T15:55:36.325933+02:00 wayofthedodo kernel: [2502369.730483] R10: ffff889092d71000 R11: 0000000000000000 R12: ffff88886064e100 > 2024-06-19T15:55:36.340957+02:00 wayofthedodo kernel: [2502369.738015] R13: ffff88886064e114 R14: ffff88886064e100 R15: ffff8881061d6000 > 2024-06-19T15:55:36.340970+02:00 wayofthedodo kernel: [2502369.745537] FS: 00007f610ac30740(0000) GS:ffff88a03fd00000(0000) knlGS:0000000000000000 > 2024-06-19T15:55:36.349441+02:00 wayofthedodo kernel: [2502369.754015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > 2024-06-19T15:55:36.355576+02:00 wayofthedodo kernel: [2502369.760149] CR2: 000000000069adf8 CR3: 00000004ba14a002 CR4: 00000000007706e0 > 2024-06-19T15:55:36.363069+02:00 wayofthedodo kernel: [2502369.767681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > 2024-06-19T15:55:36.370634+02:00 wayofthedodo kernel: [2502369.775210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > 2024-06-19T15:55:36.378121+02:00 wayofthedodo kernel: [2502369.782735] PKRU: 55555554 > 2024-06-19T15:55:36+02:00 wayofthedodo rpc.mountd[4578]: Caught signal 15, un-registering and exiting. > > Best > > Donald Hi I noticed that this problem still exists in 6.6 now and I also found it in 5.10. What's more, I found another UAF problem. They can be reproduced in the following way: 1) BUG_ON in svc_destroy mount /dev/sda /mnt/sda echo "/mnt *(rw,no_root_squash,fsid=0)" > /etc/exports echo "/mnt/sda *(rw,no_root_squash,fsid=1)" >> /etc/exports mount -t nfsd nfsd /proc/fs/nfsd systemctl restart nfs-server cd test/ sh threads_test.sh & ps aux | grep thread kill -9 2779 [root@localhost ~]# mount /dev/sda /mnt/sda [root@localhost ~]# echo "/mnt *(rw,no_root_squash,fsid=0)" > /etc/exports [root@localhost ~]# echo "/mnt/sda *(rw,no_root_squash,fsid=1)" >> /etc/exports [root@localhost ~]# mount -t nfsd nfsd /proc/fs/nfsd [root@localhost ~]# systemctl restart nfs-server [root@localhost ~]# cd test/ [root@localhost test]# sh threads_test.sh & [1] 2779 [root@localhost test]# ps aux | grep thread root 2 0.0 0.0 0 0 ? S 10:59 0:00 [kthreadd] root 2779 0.8 0.0 120052 1068 ttyS0 D 11:08 0:00 sh threads_test.sh root 2808 0.0 0.0 119468 884 ttyS0 S+ 11:08 0:00 grep --color=auto thread [root@localhost test]# kill -9 2779 [root@localhost test]# [ 521.533252] ------------[ cut here ]------------ [ 521.534078] kernel BUG at net/sunrpc/svc.c:570! [ 521.534786] invalid opcode: 0000 [#1] SMP KASAN [ 521.535492] CPU: 1 PID: 2779 Comm: sh Not tainted 5.10.0-00140-g606211a2593d-dirty #371 [ 521.536845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014 [ 521.538988] RIP: 0010:svc_destroy+0x118/0x160 [ 521.539677] Code: 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 41 48 8b bb 9c 00 00 00 e8 a4 9e 47 fe 4c 89 e7 5b 5d 41 5c e9 98 9e 47 fe <0f> 0b 0f 0b 48 c7 c7 c0 1e c1 88 [ 521.542643] RSP: 0018:ffffc90003d67ca8 EFLAGS: 00010297 [ 521.543544] RAX: ffff88810db3c018 RBX: ffff888105457514 RCX: 1ffff11020a8aeac [ 521.544714] RDX: 1ffff11020a8aea6 RSI: 0000000000000246 RDI: ffff88813632c100 [ 521.545841] RBP: ffff888105457530 R08: 0000000000000001 R09: fffff520007acf6a [ 521.546956] R10: 0000000000000003 R11: fffff520007acf69 R12: ffff888105457500 [ 521.548098] R13: ffff8882418c05c9 R14: ffff888105457500 R15: ffff8882418c05f0 [ 521.549234] FS: 00007fea0bba3700(0000) GS:ffff888136280000(0000) knlGS:0000000000000000 [ 521.550486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 521.551405] CR2: 00007efd6ca85000 CR3: 0000000450f80000 CR4: 00000000000006e0 [ 521.552509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 521.553654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 521.554666] Call Trace: [ 521.555031] ? __die_body+0x1b/0x60 [ 521.555534] ? die+0x2b/0x50 [ 521.555958] ? do_trap+0x1a1/0x260 [ 521.556460] ? svc_destroy+0x118/0x160 [ 521.556985] ? do_error_trap+0x8a/0xe0 [ 521.557512] ? svc_destroy+0x118/0x160 [ 521.558044] ? exc_invalid_op+0x4e/0x70 [ 521.558580] ? svc_destroy+0x118/0x160 [ 521.559116] ? asm_exc_invalid_op+0x12/0x20 [ 521.559701] ? svc_destroy+0x118/0x160 [ 521.560230] ? svc_destroy+0x58/0x160 [ 521.560744] nfsd_svc+0x601/0x950 [ 521.561219] ? _raw_spin_lock+0x7a/0xd0 [ 521.561754] write_threads+0x1ad/0x280 [ 521.562282] ? write_pool_threads+0x430/0x430 [ 521.562893] ? acpi_idle_enter_bm.isra.0+0x181/0x3e0 [ 521.563593] ? _raw_spin_lock_irq+0xd0/0xd0 [ 521.564191] ? _copy_from_user+0x4f/0x90 [ 521.564743] ? write_pool_threads+0x430/0x430 [ 521.565360] nfsctl_transaction_write+0xac/0x110 [ 521.566020] vfs_write+0x174/0x780 [ 521.566502] ksys_write+0xed/0x1c0 [ 521.567002] ? __ia32_sys_read+0xb0/0xb0 [ 521.567558] ? exit_to_user_mode_prepare+0x17/0x140 [ 521.568234] do_syscall_64+0x2d/0x40 [ 521.568736] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 521.569447] RIP: 0033:0x7fea0b292130 [ 521.569955] Code: 73 01 c3 48 8b 0d 58 ed 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d b9 45 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 84 [ 521.572520] RSP: 002b:00007ffc31276548 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 521.573562] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fea0b292130 [ 521.574551] RDX: 0000000000000002 RSI: 000055e8e1398240 RDI: 0000000000000001 [ 521.575545] RBP: 000055e8e1398240 R08: 000000000000000a R09: 00007fea0bba3700 [ 521.576534] R10: 000055e8e1397c20 R11: 0000000000000246 R12: 0000000000000002 [ 521.577549] R13: 0000000000000001 R14: 00007fea0b5625e0 R15: 00007fea0b55d8c0 [ 521.578567] Modules linked in: [ 521.579065] ---[ end trace 7e90504bd46025c7 ]--- [ 521.579727] RIP: 0010:svc_destroy+0x118/0x160 [ 521.580367] Code: 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 41 48 8b bb 9c 00 00 00 e8 a4 9e 47 fe 4c 89 e7 5b 5d 41 5c e9 98 9e 47 fe <0f> 0b 0f 0b 48 c7 c7 c0 1e c1 88 [ 521.582990] RSP: 0018:ffffc90003d67ca8 EFLAGS: 00010297 [ 521.583717] RAX: ffff88810db3c018 RBX: ffff888105457514 RCX: 1ffff11020a8aeac [ 521.584728] RDX: 1ffff11020a8aea6 RSI: 0000000000000246 RDI: ffff88813632c100 [ 521.585717] RBP: ffff888105457530 R08: 0000000000000001 R09: fffff520007acf6a [ 521.586736] R10: 0000000000000003 R11: fffff520007acf69 R12: ffff888105457500 [ 521.587727] R13: ffff8882418c05c9 R14: ffff888105457500 R15: ffff8882418c05f0 [ 521.588728] FS: 00007fea0bba3700(0000) GS:ffff888136280000(0000) knlGS:0000000000000000 [ 521.589863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 521.590673] CR2: 00007efd6ca85000 CR3: 0000000450f80000 CR4: 00000000000006e0 [ 521.591673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 521.592670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 521.593665] Kernel panic - not syncing: Fatal exception [ 521.594692] Kernel Offset: disabled [ 521.595234] ---[ end Kernel panic - not syncing: Fatal exception ]--- threads_test.sh: [root@localhost ~]# cat test/threads_test.sh #!/bin/bash while true do echo 1 > /proc/fs/nfsd/threads echo 0 > /proc/fs/nfsd/threads done [root@localhost ~]# 2) UAF in nfsd_create_serv mount -t nfsd nfsd /proc/fs/nfsd echo 1 > /proc/fs/nfsd/threads echo 0 > /proc/fs/nfsd/threads [root@localhost ~]# mount -t nfsd nfsd /proc/fs/nfsd [root@localhost ~]# echo 1 > /proc/fs/nfsd/threads [ 87.131454] svc: failed to register nfsdv3 RPC service (errno 111). [ 87.133288] svc: failed to register nfsaclv3 RPC service (errno 111). -bash: echo: write error: Connection refused [root@localhost ~]# echo 0 > /proc/fs/nfsd/threads [ 91.144429] ================================================================== [ 91.147024] BUG: KASAN: use-after-free in nfsd_create_serv+0x318/0x470 [ 91.149130] Write of size 4 at addr ffff888240817914 by task bash/2603 [ 91.151204] [ 91.151719] CPU: 6 PID: 2603 Comm: bash Not tainted 5.10.0-00140-g606211a2593d-dirty #371 [ 91.153868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014 [ 91.155806] Call Trace: [ 91.156184] dump_stack+0x7d/0xa3 [ 91.156676] print_address_description.constprop.0+0x1e/0x280 [ 91.157521] ? _raw_spin_lock_irqsave+0x80/0xe0 [ 91.158179] ? _raw_write_unlock_irqrestore+0x50/0x50 [ 91.158921] ? nfsd_create_serv+0x318/0x470 [ 91.159528] ? nfsd_create_serv+0x318/0x470 [ 91.160181] kasan_report.cold+0x67/0x7f [ 91.160755] ? nfsd_create_serv+0x318/0x470 [ 91.161391] check_memory_region+0x14d/0x1d0 [ 91.162024] nfsd_create_serv+0x318/0x470 [ 91.162609] nfsd_svc+0x198/0x950 [ 91.163109] ? _raw_spin_lock+0x7a/0xd0 [ 91.163670] write_threads+0x1ad/0x280 [ 91.164222] ? write_pool_threads+0x430/0x430 [ 91.164866] ? acpi_idle_enter_bm.isra.0+0x180/0x3e0 [ 91.165595] ? _raw_spin_lock_irq+0xd0/0xd0 [ 91.166210] ? _copy_from_user+0x4f/0x90 [ 91.166784] ? write_pool_threads+0x430/0x430 [ 91.167429] nfsctl_transaction_write+0xac/0x110 [ 91.168096] vfs_write+0x174/0x780 [ 91.168587] ksys_write+0xed/0x1c0 [ 91.169083] ? __ia32_sys_read+0xb0/0xb0 [ 91.169658] ? exit_to_user_mode_prepare+0x17/0x140 [ 91.170376] do_syscall_64+0x2d/0x40 [ 91.170899] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 91.171625] RIP: 0033:0x7f61f7394130 [ 91.172144] Code: 73 01 c3 48 8b 0d 58 ed 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d b9 45 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 84 [ 91.174775] RSP: 002b:00007ffc3ae06ae8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 91.175865] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f61f7394130 [ 91.176886] RDX: 0000000000000002 RSI: 000055720b66bc80 RDI: 0000000000000001 [ 91.177908] RBP: 000055720b66bc80 R08: 000000000000000a R09: 00007f61f7ca5700 [ 91.178904] R10: 000055720ba6b7c0 R11: 0000000000000246 R12: 0000000000000002 [ 91.179901] R13: 0000000000000001 R14: 00007f61f76645e0 R15: 00007f61f765f8c0 [ 91.180906] [ 91.181129] Allocated by task 2603: [ 91.181637] kasan_save_stack+0x1b/0x40 [ 91.182184] __kasan_kmalloc.constprop.0+0xb5/0xe0 [ 91.182863] __svc_create+0x53/0xab0 [ 91.183391] svc_create_pooled+0xa0/0x610 [ 91.183969] nfsd_create_serv+0x16b/0x470 [ 91.184546] nfsd_svc+0x198/0x950 [ 91.185033] write_threads+0x1ad/0x280 [ 91.185566] nfsctl_transaction_write+0xac/0x110 [ 91.186226] vfs_write+0x174/0x780 [ 91.186718] ksys_write+0xed/0x1c0 [ 91.187225] do_syscall_64+0x2d/0x40 [ 91.187730] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 91.188451] [ 91.188673] Freed by task 2603: [ 91.189131] kasan_save_stack+0x1b/0x40 [ 91.189673] kasan_set_track+0x1c/0x30 [ 91.190204] kasan_set_free_info+0x20/0x30 [ 91.190779] __kasan_slab_free+0x14a/0x180 [ 91.191364] kfree+0xac/0x6c0 [ 91.191784] nfsd_svc+0x601/0x950 [ 91.192252] write_threads+0x1ad/0x280 [ 91.192776] nfsctl_transaction_write+0xac/0x110 [ 91.193422] vfs_write+0x174/0x780 [ 91.193915] ksys_write+0xed/0x1c0 [ 91.194394] do_syscall_64+0x2d/0x40 [ 91.194904] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 91.195605] [ 91.195825] The buggy address belongs to the object at ffff888240817900 [ 91.195825] which belongs to the cache kmalloc-256 of size 256 [ 91.197579] The buggy address is located 20 bytes inside of [ 91.197579] 256-byte region [ffff888240817900, ffff888240817a00) [ 91.199189] The buggy address belongs to the page: [ 91.199860] page:ffffea0009020400 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x240810 [ 91.201165] head:ffffea0009020400 order:3 compound_mapcount:0 compound_pincount:0 [ 91.202207] flags: 0x6fffff80010200(slab|head|node=1|zone=2|lastcpupid=0x1fffff) [ 91.203237] raw: 006fffff80010200 ffffea000903ae08 ffff888240001270 ffff888100051900 [ 91.204300] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000 [ 91.205367] page dumped because: kasan: bad access detected [ 91.206136] [ 91.206359] Memory state around the buggy address: [ 91.207045] ffff888240817800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 91.208025] ffff888240817880: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 91.209012] >ffff888240817900: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 91.210015] ^ [ 91.210538] ffff888240817980: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 91.211520] ffff888240817a00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 91.212503] ================================================================== [ 91.213497] Disabling lock debugging due to kernel taint [ 91.215200] ------------[ cut here ]------------ [ 91.215854] refcount_t: addition on 0; use-after-free. [ 91.216635] WARNING: CPU: 6 PID: 2603 at lib/refcount.c:25 refcount_warn_saturate+0xdd/0x140 [ 91.217808] Modules linked in: [ 91.218254] CPU: 6 PID: 2603 Comm: bash Tainted: G B 5.10.0-00140-g606211a2593d-dirty #371 [ 91.219607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014 [ 91.221432] RIP: 0010:refcount_warn_saturate+0xdd/0x140 [ 91.222154] Code: 08 25 7d 09 01 e8 01 33 f2 00 0f 0b eb 9d 80 3d f7 24 7d 09 00 75 94 48 c7 c7 40 56 f2 83 c6 05 e7 24 7d 09 01 e8 e1 32 f2 00 <0f> 0b e9 7a ff ff ff 80 3d d1 27 [ 91.224696] RSP: 0018:ffffc90007ca7c78 EFLAGS: 00010282 [ 91.225425] RAX: 0000000000000000 RBX: ffff888240817914 RCX: 0000000000000000 [ 91.226407] RDX: 0000000000000004 RSI: 0000000000000008 RDI: fffff52000f94f81 [ 91.227397] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed1086fa700d [ 91.228383] R10: ffff888437d38067 R11: ffffed1086fa700c R12: 0000000000000000 [ 91.229381] R13: ffffffff8bd91c80 R14: ffff8881061ee5f0 R15: ffffffff8bd92280 [ 91.230391] FS: 00007f61f7ca5700(0000) GS:ffff888437d00000(0000) knlGS:0000000000000000 [ 91.231487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 91.232275] CR2: 000055720b6efe20 CR3: 0000000247283000 CR4: 00000000000006e0 [ 91.233268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 91.234271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 91.235208] Call Trace: [ 91.235542] ? __warn+0xc9/0x180 [ 91.235967] ? refcount_warn_saturate+0xdd/0x140 [ 91.236561] ? report_bug+0x1e5/0x270 [ 91.237045] ? handle_bug+0x41/0x80 [ 91.237504] ? exc_invalid_op+0x14/0x70 [ 91.238001] ? asm_exc_invalid_op+0x12/0x20 [ 91.238557] ? refcount_warn_saturate+0xdd/0x140 [ 91.239168] ? refcount_warn_saturate+0xdd/0x140 [ 91.239785] nfsd_create_serv+0x407/0x470 [ 91.240331] nfsd_svc+0x198/0x950 [ 91.240775] ? _raw_spin_lock+0x7a/0xd0 [ 91.241279] write_threads+0x1ad/0x280 [ 91.241784] ? write_pool_threads+0x430/0x430 [ 91.242394] ? acpi_idle_enter_bm.isra.0+0x180/0x3e0 [ 91.243042] ? _raw_spin_lock_irq+0xd0/0xd0 [ 91.243603] ? _copy_from_user+0x4f/0x90 [ 91.244130] ? write_pool_threads+0x430/0x430 [ 91.244700] nfsctl_transaction_write+0xac/0x110 [ 91.245319] vfs_write+0x174/0x780 [ 91.245772] ksys_write+0xed/0x1c0 [ 91.246230] ? __ia32_sys_read+0xb0/0xb0 [ 91.246761] ? exit_to_user_mode_prepare+0x17/0x140 [ 91.247428] do_syscall_64+0x2d/0x40 [ 91.247920] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 91.248600] RIP: 0033:0x7f61f7394130 [ 91.249083] Code: 73 01 c3 48 8b 0d 58 ed 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d b9 45 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 84 [ 91.251532] RSP: 002b:00007ffc3ae06ae8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 91.252554] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f61f7394130 [ 91.253498] RDX: 0000000000000002 RSI: 000055720b66bc80 RDI: 0000000000000001 [ 91.254434] RBP: 000055720b66bc80 R08: 000000000000000a R09: 00007f61f7ca5700 [ 91.255361] R10: 000055720ba6b7c0 R11: 0000000000000246 R12: 0000000000000002 [ 91.256269] R13: 0000000000000001 R14: 00007f61f76645e0 R15: 00007f61f765f8c0 [ 91.257180] ---[ end trace b515c6670aea0e34 ]--- [ 91.259080] general protection fault, maybe for address 0xffff88824f5ec740: 0000 [#1] SMP KASAN [ 91.260223] CPU: 6 PID: 2603 Comm: bash Tainted: G B W 5.10.0-00140-g606211a2593d-dirty #371 [ 91.261454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014 [ 91.263159] RIP: 0010:svc_register+0x108/0x1e0 [ 91.263732] Code: 89 e0 41 89 dc 44 89 eb 41 89 c5 48 8b 44 24 08 80 38 00 0f 85 83 00 00 00 45 89 e9 45 89 e0 44 89 f9 89 da 48 89 ee 4c 89 f7 <ff> 55 48 85 c0 0f 88 2a 40 19 03 [ 91.266112] RSP: 0018:ffffc90007ca79c0 EFLAGS: 00010286 [ 91.266771] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002 [ 91.267667] RDX: 0000000000000000 RSI: ffff88824f5ec740 RDI: ffffffff8bd91c80 [ 91.268574] RBP: ffff88824f5ec740 R08: 0000000000000011 R09: 0000000000000801 [ 91.269507] R10: ffffc90007ca7f50 R11: fffff52000f94ee0 R12: 0000000000000011 [ 91.270441] R13: 0000000000000801 R14: ffffffff8bd91c80 R15: 0000000000000002 [ 91.271381] FS: 00007f61f7ca5700(0000) GS:ffff888437d00000(0000) knlGS:0000000000000000 [ 91.272451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 91.273192] CR2: 000055720b6efe20 CR3: 0000000247283000 CR4: 00000000000006e0 [ 91.274107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 91.274996] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 91.275910] Call Trace: [ 91.276228] ? __die_body+0x1b/0x60 [ 91.276687] ? die_addr+0x43/0x70 [ 91.277112] ? exc_general_protection+0x198/0x2e0 [ 91.277710] ? asm_exc_general_protection+0x1e/0x30 [ 91.278345] ? svc_register+0x108/0x1e0 [ 91.278841] svc_setup_socket+0x6e6/0xcb0 [ 91.279377] ? memset+0x20/0x40 [ 91.279806] svc_create_socket+0x220/0x3f0 [ 91.280359] ? svc_setup_socket+0xcb0/0xcb0 [ 91.280934] ? kasan_set_track+0x1c/0x30 [ 91.281471] ? mutex_lock+0x8e/0xe0 [ 91.281946] ? __mutex_lock_slowpath+0x10/0x10 [ 91.282563] ? _raw_spin_lock+0x7a/0xd0 [ 91.283092] ? _raw_spin_lock_irq+0xd0/0xd0 [ 91.283665] ? alloc_workqueue+0x857/0xeb0 [ 91.284204] _svc_xprt_create+0x1f0/0x570 [ 91.284739] ? svc_add_new_perm_xprt+0x140/0x140 [ 91.285369] ? bucket_table_alloc.isra.0+0xf6/0x3f0 [ 91.285997] svc_xprt_create+0x36/0x90 [ 91.286481] nfsd_svc+0x792/0x950 [ 91.286922] write_threads+0x1ad/0x280 [ 91.287397] ? write_pool_threads+0x430/0x430 [ 91.287946] ? acpi_idle_enter_bm.isra.0+0x180/0x3e0 [ 91.288575] ? _raw_spin_lock_irq+0xd0/0xd0 [ 91.289110] ? _copy_from_user+0x4f/0x90 [ 91.289603] ? write_pool_threads+0x430/0x430 [ 91.290152] nfsctl_transaction_write+0xac/0x110 [ 91.290731] vfs_write+0x174/0x780 [ 91.291172] ksys_write+0xed/0x1c0 [ 91.291608] ? __ia32_sys_read+0xb0/0xb0 [ 91.292112] ? exit_to_user_mode_prepare+0x17/0x140 [ 91.292746] do_syscall_64+0x2d/0x40 [ 91.293208] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 91.293852] RIP: 0033:0x7f61f7394130 [ 91.294302] Code: 73 01 c3 48 8b 0d 58 ed 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d b9 45 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 84 [ 91.296652] RSP: 002b:00007ffc3ae06ae8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 91.297622] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f61f7394130 [ 91.298546] RDX: 0000000000000002 RSI: 000055720b66bc80 RDI: 0000000000000001 [ 91.299465] RBP: 000055720b66bc80 R08: 000000000000000a R09: 00007f61f7ca5700 [ 91.300392] R10: 000055720ba6b7c0 R11: 0000000000000246 R12: 0000000000000002 [ 91.301301] R13: 0000000000000001 R14: 00007f61f76645e0 R15: 00007f61f765f8c0 [ 91.302207] Modules linked in: [ 91.302674] ---[ end trace b515c6670aea0e35 ]--- [ 91.303281] RIP: 0010:svc_register+0x108/0x1e0 [ 91.303850] Code: 89 e0 41 89 dc 44 89 eb 41 89 c5 48 8b 44 24 08 80 38 00 0f 85 83 00 00 00 45 89 e9 45 89 e0 44 89 f9 89 da 48 89 ee 4c 89 f7 <ff> 55 48 85 c0 0f 88 2a 40 19 03 [ 91.306245] RSP: 0018:ffffc90007ca79c0 EFLAGS: 00010286 [ 91.306919] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002 [ 91.307798] RDX: 0000000000000000 RSI: ffff88824f5ec740 RDI: ffffffff8bd91c80 [ 91.308732] RBP: ffff88824f5ec740 R08: 0000000000000011 R09: 0000000000000801 [ 91.309658] R10: ffffc90007ca7f50 R11: fffff52000f94ee0 R12: 0000000000000011 [ 91.310564] R13: 0000000000000801 R14: ffffffff8bd91c80 R15: 0000000000000002 [ 91.311472] FS: 00007f61f7ca5700(0000) GS:ffff888437d00000(0000) knlGS:0000000000000000 [ 91.312483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 91.313207] CR2: 000055720b6efe20 CR3: 0000000247283000 CR4: 00000000000006e0 [ 91.314121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 91.315028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 91.319328] Kernel panic - not syncing: Fatal exception [ 91.320712] Kernel Offset: disabled [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") since this patch changes the behavior of the error path. I confirmed this by fixing both issues with the following changes: diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index ee5713fca187..05d4b463c16b 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred *cred) if (error < 0 && !nfsd_up_before) nfsd_shutdown_net(net); out_put: + if (error < 0) + nfsd_last_thread(net); /* Threads now hold service active */ if (xchg(&nn->keep_active, 0)) svc_put(serv); They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in nfsd_svc()") in mainline. Maybe it would be a good idea to push it to the LTS branches. Thanks. >> Thanks, >> NeilBrown >>
On Fri, Aug 23, 2024 at 11:35:28AM +0800, Li Lingfeng wrote: [ snipped ] > [ 91.319328] Kernel panic - not syncing: Fatal exception > [ 91.320712] Kernel Offset: disabled > [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate > nfsd_last_thread() from nfsd_put()") since this patch changes the behavior > of the error path. > > I confirmed this by fixing both issues with the following changes: > diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c > index ee5713fca187..05d4b463c16b 100644 > --- a/fs/nfsd/nfssvc.c > +++ b/fs/nfsd/nfssvc.c > @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred > *cred) > if (error < 0 && !nfsd_up_before) > nfsd_shutdown_net(net); > out_put: > + if (error < 0) > + nfsd_last_thread(net); > /* Threads now hold service active */ > if (xchg(&nn->keep_active, 0)) > svc_put(serv); > > They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in > nfsd_svc()") in mainline. > > Maybe it would be a good idea to push it to the LTS branches. To be clear, by "push it to LTS" I assume you mean apply bf32075? I have now applied commit bf32075256e9 ("NFSD: simplify error paths in nfsd_svc()") to nfsd-6.6.y, nfsd-5.15.y, and nfsd-5.10.y in my kernel.org git repo, for testing. https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git I will run these three against the usual NFSD CI today, but feel free to try them out yourself and report your results. Now unfortunately 6.1.y is still "special." It appears that commit 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") was reverted in that kernel, and the fix you mention here does not cleanly apply to v6.1.106. Based on some previous comments on this list, I think I need to fix up v6.1 LTS to be like the other three kernels, and then apply bf32075.
On Fri, Aug 23, 2024 at 03:15:04PM -0400, Chuck Lever wrote: > On Fri, Aug 23, 2024 at 11:35:28AM +0800, Li Lingfeng wrote: > > [ snipped ] > > > [ 91.319328] Kernel panic - not syncing: Fatal exception > > [ 91.320712] Kernel Offset: disabled > > [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate > > nfsd_last_thread() from nfsd_put()") since this patch changes the behavior > > of the error path. > > > > I confirmed this by fixing both issues with the following changes: > > diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c > > index ee5713fca187..05d4b463c16b 100644 > > --- a/fs/nfsd/nfssvc.c > > +++ b/fs/nfsd/nfssvc.c > > @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred > > *cred) > > if (error < 0 && !nfsd_up_before) > > nfsd_shutdown_net(net); > > out_put: > > + if (error < 0) > > + nfsd_last_thread(net); > > /* Threads now hold service active */ > > if (xchg(&nn->keep_active, 0)) > > svc_put(serv); > > > > They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in > > nfsd_svc()") in mainline. > > > > Maybe it would be a good idea to push it to the LTS branches. > > To be clear, by "push it to LTS" I assume you mean apply bf32075? > > I have now applied commit bf32075256e9 ("NFSD: simplify error paths > in nfsd_svc()") to nfsd-6.6.y, nfsd-5.15.y, and nfsd-5.10.y in my > kernel.org git repo, for testing. > > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git > > I will run these three against the usual NFSD CI today, but feel > free to try them out yourself and report your results. > > Now unfortunately 6.1.y is still "special." It appears that commit > 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") > was reverted in that kernel, and the fix you mention here does not > cleanly apply to v6.1.106. Based on some previous comments on this > list, I think I need to fix up v6.1 LTS to be like the other three > kernels, and then apply bf32075. OK, I've got a candidate for v6.1.y now, too. See the nfsd-6.1.y branch in the above git repo.
在 2024/8/24 3:15, Chuck Lever 写道: > On Fri, Aug 23, 2024 at 11:35:28AM +0800, Li Lingfeng wrote: > > [ snipped ] > >> [ 91.319328] Kernel panic - not syncing: Fatal exception >> [ 91.320712] Kernel Offset: disabled >> [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- >> >> Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate >> nfsd_last_thread() from nfsd_put()") since this patch changes the behavior >> of the error path. >> >> I confirmed this by fixing both issues with the following changes: >> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c >> index ee5713fca187..05d4b463c16b 100644 >> --- a/fs/nfsd/nfssvc.c >> +++ b/fs/nfsd/nfssvc.c >> @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred >> *cred) >> if (error < 0 && !nfsd_up_before) >> nfsd_shutdown_net(net); >> out_put: >> + if (error < 0) >> + nfsd_last_thread(net); >> /* Threads now hold service active */ >> if (xchg(&nn->keep_active, 0)) >> svc_put(serv); >> >> They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in >> nfsd_svc()") in mainline. >> >> Maybe it would be a good idea to push it to the LTS branches. > To be clear, by "push it to LTS" I assume you mean apply bf32075? Yes. And I've applied this patch to my own 5.10 repo and tested it. > > I have now applied commit bf32075256e9 ("NFSD: simplify error paths > in nfsd_svc()") to nfsd-6.6.y, nfsd-5.15.y, and nfsd-5.10.y in my > kernel.org git repo, for testing. > > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git > > I will run these three against the usual NFSD CI today, but feel > free to try them out yourself and report your results. > > Now unfortunately 6.1.y is still "special." It appears that commit > 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") > was reverted in that kernel, and the fix you mention here does not > cleanly apply to v6.1.106. Based on some previous comments on this > list, I think I need to fix up v6.1 LTS to be like the other three > kernels, and then apply bf32075. >
在 2024/8/24 3:15, Chuck Lever 写道: > On Fri, Aug 23, 2024 at 11:35:28AM +0800, Li Lingfeng wrote: > > [ snipped ] > >> [ 91.319328] Kernel panic - not syncing: Fatal exception >> [ 91.320712] Kernel Offset: disabled >> [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- >> >> Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate >> nfsd_last_thread() from nfsd_put()") since this patch changes the behavior >> of the error path. >> >> I confirmed this by fixing both issues with the following changes: >> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c >> index ee5713fca187..05d4b463c16b 100644 >> --- a/fs/nfsd/nfssvc.c >> +++ b/fs/nfsd/nfssvc.c >> @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred >> *cred) >> if (error < 0 && !nfsd_up_before) >> nfsd_shutdown_net(net); >> out_put: >> + if (error < 0) >> + nfsd_last_thread(net); >> /* Threads now hold service active */ >> if (xchg(&nn->keep_active, 0)) >> svc_put(serv); >> >> They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in >> nfsd_svc()") in mainline. >> >> Maybe it would be a good idea to push it to the LTS branches. > To be clear, by "push it to LTS" I assume you mean apply bf32075? > > I have now applied commit bf32075256e9 ("NFSD: simplify error paths > in nfsd_svc()") to nfsd-6.6.y, nfsd-5.15.y, and nfsd-5.10.y in my > kernel.org git repo, for testing. > > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git > > I will run these three against the usual NFSD CI today, but feel > free to try them out yourself and report your results. I have tested the two use cases I mentioned above on nfsd-5.10.y, nfsd-5.15.y, and nfsd-6.6.y, and all three versions no longer have issues. > Now unfortunately 6.1.y is still "special." It appears that commit > 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") > was reverted in that kernel, and the fix you mention here does not > cleanly apply to v6.1.106. Based on some previous comments on this > list, I think I need to fix up v6.1 LTS to be like the other three > kernels, and then apply bf32075. >
在 2024/8/24 3:59, Chuck Lever 写道: > On Fri, Aug 23, 2024 at 03:15:04PM -0400, Chuck Lever wrote: >> On Fri, Aug 23, 2024 at 11:35:28AM +0800, Li Lingfeng wrote: >> >> [ snipped ] >> >>> [ 91.319328] Kernel panic - not syncing: Fatal exception >>> [ 91.320712] Kernel Offset: disabled >>> [ 91.321189] ---[ end Kernel panic - not syncing: Fatal exception ]--- >>> >>> Both of them were introduced by commit 9f28a971ee9f ("nfsd: separate >>> nfsd_last_thread() from nfsd_put()") since this patch changes the behavior >>> of the error path. >>> >>> I confirmed this by fixing both issues with the following changes: >>> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c >>> index ee5713fca187..05d4b463c16b 100644 >>> --- a/fs/nfsd/nfssvc.c >>> +++ b/fs/nfsd/nfssvc.c >>> @@ -811,6 +811,8 @@ nfsd_svc(int nrservs, struct net *net, const struct cred >>> *cred) >>> if (error < 0 && !nfsd_up_before) >>> nfsd_shutdown_net(net); >>> out_put: >>> + if (error < 0) >>> + nfsd_last_thread(net); >>> /* Threads now hold service active */ >>> if (xchg(&nn->keep_active, 0)) >>> svc_put(serv); >>> >>> They have been fixed by commit bf32075256e9 ("NFSD: simplify error paths in >>> nfsd_svc()") in mainline. >>> >>> Maybe it would be a good idea to push it to the LTS branches. >> To be clear, by "push it to LTS" I assume you mean apply bf32075? >> >> I have now applied commit bf32075256e9 ("NFSD: simplify error paths >> in nfsd_svc()") to nfsd-6.6.y, nfsd-5.15.y, and nfsd-5.10.y in my >> kernel.org git repo, for testing. >> >> https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git >> >> I will run these three against the usual NFSD CI today, but feel >> free to try them out yourself and report your results. >> >> Now unfortunately 6.1.y is still "special." It appears that commit >> 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()") >> was reverted in that kernel, and the fix you mention here does not >> cleanly apply to v6.1.106. Based on some previous comments on this >> list, I think I need to fix up v6.1 LTS to be like the other three >> kernels, and then apply bf32075. > OK, I've got a candidate for v6.1.y now, too. See the nfsd-6.1.y > branch in the above git repo. I'm getting a compile error when trying to compile nfsd-6.1.y. The return value of fh_fill_pre_attrs is void, but nfsd_setattr attempts to assign its return value to a __be32 variable. I found out that the return type of fh_fill_pre_attrs was changed in a332018a91c4 ("nfsd: handle failure to collect pre/post-op attrs more sanely") which hasn't been added to nfsd-6.1.y yet. Is nfsd-6.1.y not ready yet? fs/nfsd/vfs.c: In function ‘nfsd_setattr’: fs/nfsd/vfs.c:536:6: error: void value not ignored as it ought to be 536 | err = fh_fill_pre_attrs(fhp); | ^ make[3]: *** [scripts/Makefile.build:250: fs/nfsd/vfs.o] Error 1 make[3]: *** Waiting for unfinished jobs.... LD [M] fs/ubifs/ubifs.o make[2]: *** [scripts/Makefile.build:503: fs/nfsd] Error 2 make[1]: *** [scripts/Makefile.build:503: fs] Error 2 make: *** [Makefile:2009: .] Error 2 >
diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 587811a002c9..3237f7dfde1e 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -575,11 +575,12 @@ svc_destroy(struct kref *ref) timer_shutdown_sync(&serv->sv_temptimer); /* - * The last user is gone and thus all sockets have to be destroyed to - * the point. Check this. + * Remaining transports at this point are not expected. */ - BUG_ON(!list_empty(&serv->sv_permsocks)); - BUG_ON(!list_empty(&serv->sv_tempsocks)); + WARN_ONCE(!list_empty(&serv->sv_permsocks), + "SVC: permsocks remain for %s\n", serv->sv_program->pg_name); + WARN_ONCE(!list_empty(&serv->sv_tempsocks), + "SVC: tempsocks remain for %s\n", serv->sv_program->pg_name); cache_clean_deferred(serv);