| Message ID | 20210312022737.44122-1-skhan@linuxfoundation.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | usbip: fix vhci races in connection tear down | expand |
On Thu, Mar 11, 2021 at 07:27:37PM -0700, Shuah Khan wrote: > vhci_shutdown_connection() references connection state (tcp_socket, > tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the > lock. > > Current connection tear down sequence: > Step 1: shutdown the socket > Step 2: stop rx thread and reset tcp_rx pointer > Step 3: stop tx thread and reset tcp_tx pointer > Step 4: Reset tcp_socket and sockfd > > There are several race windows between these steps. In addition, device > reset routine (vhci_device_reset) resets tcp_socket and sockfd holding > the lock. > > Fix these races: > - Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs > only once. > - Change attach_store() to initialize in_disconnect to false while > initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd) > - Change vhci_shutdown_connection() to check in_disconnect and bail > out if disconnect is in progress. > - Change vhci_shutdown_connection() to > -- hold lock to save connection state pointers and unlock. > -- Shutdown the socket and stop threads. > -- Hold lock to clear connection status and in_disconnect flag. > - Change vhci_device_reset() to reset tcp_socket and sockfd. > if !in_disconnect > > Tested syzbot and the reproducer did not trigger any issue. > > Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com > Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> > --- > drivers/usb/usbip/usbip_common.h | 1 + > drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++--------- > drivers/usb/usbip/vhci_sysfs.c | 4 +++ > 3 files changed, 45 insertions(+), 15 deletions(-) > diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c > index 3209b5ddd30c..c1917efe5737 100644 > --- a/drivers/usb/usbip/vhci_hcd.c > +++ b/drivers/usb/usbip/vhci_hcd.c > @@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev) > static void vhci_shutdown_connection(struct usbip_device *ud) > { > struct vhci_device *vdev = container_of(ud, struct vhci_device, ud); > + unsigned long flags; > + struct socket *socket; > + struct task_struct *tcp_rx = NULL; > + struct task_struct *tcp_tx = NULL; > + int sockfd = 0; > + > + spin_lock_irqsave(&ud->lock, flags); > + if (vdev->ud.in_disconnect) { > + pr_info("%s: Disconnect in progress for sockfd %d\n", > + __func__, ud->sockfd); Looks like you forgot to remove all you debug printks like this one before submitting. > + spin_unlock_irqrestore(&ud->lock, flags); > + return; > + } Johan
On 3/12/21 3:45 AM, Johan Hovold wrote: > On Thu, Mar 11, 2021 at 07:27:37PM -0700, Shuah Khan wrote: >> vhci_shutdown_connection() references connection state (tcp_socket, >> tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the >> lock. >> >> Current connection tear down sequence: >> Step 1: shutdown the socket >> Step 2: stop rx thread and reset tcp_rx pointer >> Step 3: stop tx thread and reset tcp_tx pointer >> Step 4: Reset tcp_socket and sockfd >> >> There are several race windows between these steps. In addition, device >> reset routine (vhci_device_reset) resets tcp_socket and sockfd holding >> the lock. >> >> Fix these races: >> - Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs >> only once. >> - Change attach_store() to initialize in_disconnect to false while >> initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd) >> - Change vhci_shutdown_connection() to check in_disconnect and bail >> out if disconnect is in progress. >> - Change vhci_shutdown_connection() to >> -- hold lock to save connection state pointers and unlock. >> -- Shutdown the socket and stop threads. >> -- Hold lock to clear connection status and in_disconnect flag. >> - Change vhci_device_reset() to reset tcp_socket and sockfd. >> if !in_disconnect >> >> Tested syzbot and the reproducer did not trigger any issue. >> >> Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com >> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> >> --- >> drivers/usb/usbip/usbip_common.h | 1 + >> drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++--------- >> drivers/usb/usbip/vhci_sysfs.c | 4 +++ >> 3 files changed, 45 insertions(+), 15 deletions(-) > >> diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c >> index 3209b5ddd30c..c1917efe5737 100644 >> --- a/drivers/usb/usbip/vhci_hcd.c >> +++ b/drivers/usb/usbip/vhci_hcd.c >> @@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev) >> static void vhci_shutdown_connection(struct usbip_device *ud) >> { >> struct vhci_device *vdev = container_of(ud, struct vhci_device, ud); >> + unsigned long flags; >> + struct socket *socket; >> + struct task_struct *tcp_rx = NULL; >> + struct task_struct *tcp_tx = NULL; >> + int sockfd = 0; >> + >> + spin_lock_irqsave(&ud->lock, flags); >> + if (vdev->ud.in_disconnect) { >> + pr_info("%s: Disconnect in progress for sockfd %d\n", >> + __func__, ud->sockfd); > > Looks like you forgot to remove all you debug printks like this one > before submitting. > Some printks were already in there and helped with debug. Yes I added a few more when I submitted for syzbot testing. I will clean them up i v2. thanks, -- Shuah
On 3/12/21 12:08 AM, Hillf Danton wrote: > On Thu, 11 Mar 2021 19:27:37 -0700 Shuah Khan wrote: >> vhci_shutdown_connection() references connection state (tcp_socket, >> tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the >> lock. >> >> Current connection tear down sequence: >> Step 1: shutdown the socket >> Step 2: stop rx thread and reset tcp_rx pointer >> Step 3: stop tx thread and reset tcp_tx pointer >> Step 4: Reset tcp_socket and sockfd >> >> There are several race windows between these steps. In addition, device >> reset routine (vhci_device_reset) resets tcp_socket and sockfd holding >> the lock. > > Can you specify the scenario where reset runs in race with teardown as > both are parts of usbip_work on a singlethread workqueue? >> Hmm. I can't think of one. I was concerned about any async paths that potentially interfere with shutdown. With vhci_shutdown_connection() being so relaxed with locking, this is a cautious approach on my part. I am also keeping in mind that this problem shows up in a limited scope fuzzing test that doesn't trigger any other normal paths that would be active if there is real device on the other side. As for the tcp_socket check in the reset routine, I am not positive what purpose it serves. I introduced the in_disconnect flag so shutdown and reset don't collide, in case I am missing some scenario in the normal path when we actually have a actual device attached. With the other locking and error path problems in addressed, both shutdown and reset could be made simpler. In any case, I think in_disconnect might be too big a hammer. I will redo the patch without it and also remove tcp_socket handling from the reset routine. I don't see USBIP_EH_RESET getting set without USBIP_EH_SHUTDOWN. thanks, -- Shuah
diff --git a/drivers/usb/usbip/usbip_common.h b/drivers/usb/usbip/usbip_common.h index d60ce17d3dd2..f6261c5a8c91 100644 --- a/drivers/usb/usbip/usbip_common.h +++ b/drivers/usb/usbip/usbip_common.h @@ -268,6 +268,7 @@ struct usbip_device { struct task_struct *tcp_rx; struct task_struct *tcp_tx; + bool in_disconnect; /* run device disconnect just once */ unsigned long event; wait_queue_head_t eh_waitq; diff --git a/drivers/usb/usbip/vhci_hcd.c b/drivers/usb/usbip/vhci_hcd.c index 3209b5ddd30c..c1917efe5737 100644 --- a/drivers/usb/usbip/vhci_hcd.c +++ b/drivers/usb/usbip/vhci_hcd.c @@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev) static void vhci_shutdown_connection(struct usbip_device *ud) { struct vhci_device *vdev = container_of(ud, struct vhci_device, ud); + unsigned long flags; + struct socket *socket; + struct task_struct *tcp_rx = NULL; + struct task_struct *tcp_tx = NULL; + int sockfd = 0; + + spin_lock_irqsave(&ud->lock, flags); + if (vdev->ud.in_disconnect) { + pr_info("%s: Disconnect in progress for sockfd %d\n", + __func__, ud->sockfd); + spin_unlock_irqrestore(&ud->lock, flags); + return; + } + vdev->ud.in_disconnect = true; + socket = ud->tcp_socket; + tcp_rx = vdev->ud.tcp_rx; + tcp_tx = vdev->ud.tcp_tx; + sockfd = ud->sockfd; + spin_unlock_irqrestore(&ud->lock, flags); /* need this? see stub_dev.c */ - if (ud->tcp_socket) { - pr_debug("shutdown tcp_socket %d\n", ud->sockfd); - kernel_sock_shutdown(ud->tcp_socket, SHUT_RDWR); + if (socket) { + pr_info("%s: shutdown tcp_socket %d\n", __func__, sockfd); + kernel_sock_shutdown(socket, SHUT_RDWR); } - /* kill threads related to this sdev */ - if (vdev->ud.tcp_rx) { - kthread_stop_put(vdev->ud.tcp_rx); - vdev->ud.tcp_rx = NULL; + /* kill threads related to this vdev */ + if (tcp_rx) { + pr_info("%s: stop rx thread\n", __func__); + kthread_stop_put(tcp_rx); } - if (vdev->ud.tcp_tx) { - kthread_stop_put(vdev->ud.tcp_tx); - vdev->ud.tcp_tx = NULL; + if (tcp_tx) { + pr_info("%s: stop tx thread\n", __func__); + kthread_stop_put(tcp_tx); } - pr_info("stop threads\n"); + spin_lock_irqsave(&ud->lock, flags); /* active connection is closed */ - if (vdev->ud.tcp_socket) { + if (ud->tcp_socket) { + vdev->ud.tcp_rx = NULL; + vdev->ud.tcp_tx = NULL; sockfd_put(vdev->ud.tcp_socket); vdev->ud.tcp_socket = NULL; vdev->ud.sockfd = -1; } - pr_info("release socket\n"); + vdev->ud.in_disconnect = false; + spin_unlock_irqrestore(&ud->lock, flags); + pr_info("%s: release socket\n", __func__); vhci_device_unlink_cleanup(vdev); @@ -1057,7 +1080,7 @@ static void vhci_shutdown_connection(struct usbip_device *ud) */ rh_port_disconnect(vdev); - pr_info("disconnect device\n"); + pr_info("%s: disconnect device\n", __func__); } static void vhci_device_reset(struct usbip_device *ud) @@ -1073,7 +1096,9 @@ static void vhci_device_reset(struct usbip_device *ud) usb_put_dev(vdev->udev); vdev->udev = NULL; - if (ud->tcp_socket) { + if (!vdev->ud.in_disconnect && ud->tcp_socket) { + pr_info("%s: release tcp_socket %d\n", + __func__, ud->sockfd); sockfd_put(ud->tcp_socket); ud->tcp_socket = NULL; ud->sockfd = -1; diff --git a/drivers/usb/usbip/vhci_sysfs.c b/drivers/usb/usbip/vhci_sysfs.c index c4b4256e5dad..c4457026d5ad 100644 --- a/drivers/usb/usbip/vhci_sysfs.c +++ b/drivers/usb/usbip/vhci_sysfs.c @@ -412,6 +412,10 @@ static ssize_t attach_store(struct device *dev, struct device_attribute *attr, vdev->ud.tcp_rx = tcp_rx; vdev->ud.tcp_tx = tcp_tx; vdev->ud.status = VDEV_ST_NOTASSIGNED; + + /* used to run socket shutdown just once */ + vdev->ud.in_disconnect = false; + usbip_kcov_handle_init(&vdev->ud); spin_unlock(&vdev->ud.lock);
vhci_shutdown_connection() references connection state (tcp_socket, tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the lock. Current connection tear down sequence: Step 1: shutdown the socket Step 2: stop rx thread and reset tcp_rx pointer Step 3: stop tx thread and reset tcp_tx pointer Step 4: Reset tcp_socket and sockfd There are several race windows between these steps. In addition, device reset routine (vhci_device_reset) resets tcp_socket and sockfd holding the lock. Fix these races: - Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs only once. - Change attach_store() to initialize in_disconnect to false while initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd) - Change vhci_shutdown_connection() to check in_disconnect and bail out if disconnect is in progress. - Change vhci_shutdown_connection() to -- hold lock to save connection state pointers and unlock. -- Shutdown the socket and stop threads. -- Hold lock to clear connection status and in_disconnect flag. - Change vhci_device_reset() to reset tcp_socket and sockfd. if !in_disconnect Tested syzbot and the reproducer did not trigger any issue. Reported-and-tested-by: syzbot+a93fba6d384346a761e3@syzkaller.appspotmail.com Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> --- drivers/usb/usbip/usbip_common.h | 1 + drivers/usb/usbip/vhci_hcd.c | 55 +++++++++++++++++++++++--------- drivers/usb/usbip/vhci_sysfs.c | 4 +++ 3 files changed, 45 insertions(+), 15 deletions(-)