Message ID | 20240815220437.69511-1-kuniyu@amazon.com (mailing list archive) |
---|---|
State | Accepted |
Commit | 807067bf014d4a3ae2cc55bd3de16f22a01eb580 |
Delegated to: | Netdev Maintainers |
Headers | show |
Series | [v1,net] kcm: Serialise kcm_sendmsg() for the same socket. | expand |
Hello Kuniyuki, On Fri, Aug 16, 2024 at 6:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > syzkaller reported UAF in kcm_release(). [0] > > The scenario is > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > by sk_stream_wait_memory() > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > and puts the skb to the write queue > > 4. Thread A faces an error and finally frees skb that is already in the > write queue > > 5. kcm_release() does double-free the skb in the write queue > > When a thread is building a MSG_MORE skb, another thread must not touch it. Thanks for the analysis. Since the empty skb (without payload) could cause such race and double-free issue, I wonder if we can clear the empty skb before waiting for memory, like what already implemented in tcp_recvmsg_locked(): commit 72bf4f1767f0386970dc04726dc5bc2e3991dc19 Author: Eric Dumazet <edumazet@google.com> Date: Thu Oct 19 11:24:57 2023 +0000 net: do not leave an empty skb in write queue Under memory stress conditions, tcp_sendmsg_locked() might call sk_stream_wait_memory(), thus releasing the socket lock. If a fresh skb has been allocated prior to this, we should not leave it in the write queue otherwise tcp_write_xmit() could panic. I'm not pretty sure if I understand correctly. Thanks, Jason
From: Jason Xing <kerneljasonxing@gmail.com> Date: Fri, 16 Aug 2024 10:56:19 +0800 > Hello Kuniyuki, > > On Fri, Aug 16, 2024 at 6:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > syzkaller reported UAF in kcm_release(). [0] > > > > The scenario is > > > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > > by sk_stream_wait_memory() > > > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > > and puts the skb to the write queue > > > > 4. Thread A faces an error and finally frees skb that is already in the > > write queue > > > > 5. kcm_release() does double-free the skb in the write queue > > > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > Thanks for the analysis. > > Since the empty skb (without payload) could cause such race and > double-free issue, I wonder if we can clear the empty skb before > waiting for memory, kcm->seq_skb is set when a part of data is copied to skb, so it's not empty. Also, seq_skb is cleared when queued to the write queue. The problem is one thread referencing kcm->seq_skb goes to sleep and another thread queues the skb to the write queue. ---8<--- if (eor) { bool not_busy = skb_queue_empty(&sk->sk_write_queue); if (head) { /* Message complete, queue it on send buffer */ __skb_queue_tail(&sk->sk_write_queue, head); kcm->seq_skb = NULL; KCM_STATS_INCR(kcm->stats.tx_msgs); } ... } else { /* Message not complete, save state */ partial_message: if (head) { kcm->seq_skb = head; kcm_tx_msg(head)->last_skb = skb; } ---8<---
On Fri, Aug 16, 2024 at 11:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > From: Jason Xing <kerneljasonxing@gmail.com> > Date: Fri, 16 Aug 2024 10:56:19 +0800 > > Hello Kuniyuki, > > > > On Fri, Aug 16, 2024 at 6:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > > > syzkaller reported UAF in kcm_release(). [0] > > > > > > The scenario is > > > > > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > > > > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > > > by sk_stream_wait_memory() > > > > > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > > > and puts the skb to the write queue > > > > > > 4. Thread A faces an error and finally frees skb that is already in the > > > write queue > > > > > > 5. kcm_release() does double-free the skb in the write queue > > > > > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > > > Thanks for the analysis. > > > > Since the empty skb (without payload) could cause such race and > > double-free issue, I wonder if we can clear the empty skb before > > waiting for memory, > > kcm->seq_skb is set when a part of data is copied to skb, so it's not > empty. Also, seq_skb is cleared when queued to the write queue. > > The problem is one thread referencing kcm->seq_skb goes to sleep and > another thread queues the skb to the write queue. > > ---8<--- > if (eor) { > bool not_busy = skb_queue_empty(&sk->sk_write_queue); > > if (head) { > /* Message complete, queue it on send buffer */ > __skb_queue_tail(&sk->sk_write_queue, head); > kcm->seq_skb = NULL; > KCM_STATS_INCR(kcm->stats.tx_msgs); > } > ... > } else { > /* Message not complete, save state */ > partial_message: > if (head) { > kcm->seq_skb = head; > kcm_tx_msg(head)->last_skb = skb; > } > ---8<--- Oh, I see the difference of handling error part after waiting for memory between tcp_sendmsg_locked and kcm_sendmsg: In kcm_sendmsg, it could kfree the skb which causes the issue while tcp doesn't. But I cannot help asking if that lock is a little bit heavy, please don't get me wrong, I'm not against it. In the meantime, I decided to take a deep look at the 'out_error' label part. Thanks, Jason
From: Jason Xing <kerneljasonxing@gmail.com> Date: Fri, 16 Aug 2024 11:36:35 +0800 > On Fri, Aug 16, 2024 at 11:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > From: Jason Xing <kerneljasonxing@gmail.com> > > Date: Fri, 16 Aug 2024 10:56:19 +0800 > > > Hello Kuniyuki, > > > > > > On Fri, Aug 16, 2024 at 6:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > > > > > syzkaller reported UAF in kcm_release(). [0] > > > > > > > > The scenario is > > > > > > > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > > > > > > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > > > > by sk_stream_wait_memory() > > > > > > > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > > > > and puts the skb to the write queue > > > > > > > > 4. Thread A faces an error and finally frees skb that is already in the > > > > write queue > > > > > > > > 5. kcm_release() does double-free the skb in the write queue > > > > > > > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > > > > > Thanks for the analysis. > > > > > > Since the empty skb (without payload) could cause such race and > > > double-free issue, I wonder if we can clear the empty skb before > > > waiting for memory, > > > > kcm->seq_skb is set when a part of data is copied to skb, so it's not > > empty. Also, seq_skb is cleared when queued to the write queue. > > > > The problem is one thread referencing kcm->seq_skb goes to sleep and > > another thread queues the skb to the write queue. > > > > ---8<--- > > if (eor) { > > bool not_busy = skb_queue_empty(&sk->sk_write_queue); > > > > if (head) { > > /* Message complete, queue it on send buffer */ > > __skb_queue_tail(&sk->sk_write_queue, head); > > kcm->seq_skb = NULL; > > KCM_STATS_INCR(kcm->stats.tx_msgs); > > } > > ... > > } else { > > /* Message not complete, save state */ > > partial_message: > > if (head) { > > kcm->seq_skb = head; > > kcm_tx_msg(head)->last_skb = skb; > > } > > ---8<--- > > Oh, I see the difference of handling error part after waiting for > memory between tcp_sendmsg_locked and kcm_sendmsg: > In kcm_sendmsg, it could kfree the skb which causes the issue while tcp doesn't. > > But I cannot help asking if that lock is a little bit heavy, please > don't get me wrong, I'm not against it. In the meantime, I decided to > take a deep look at the 'out_error' label part. I don't think the mutex is heavy because kcm_sendmsg() is already serialised with lock_sock().
On Fri, Aug 16, 2024 at 11:47 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > From: Jason Xing <kerneljasonxing@gmail.com> > Date: Fri, 16 Aug 2024 11:36:35 +0800 > > On Fri, Aug 16, 2024 at 11:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > > > From: Jason Xing <kerneljasonxing@gmail.com> > > > Date: Fri, 16 Aug 2024 10:56:19 +0800 > > > > Hello Kuniyuki, > > > > > > > > On Fri, Aug 16, 2024 at 6:05 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > > > > > > > syzkaller reported UAF in kcm_release(). [0] > > > > > > > > > > The scenario is > > > > > > > > > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > > > > > > > > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > > > > > by sk_stream_wait_memory() > > > > > > > > > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > > > > > and puts the skb to the write queue > > > > > > > > > > 4. Thread A faces an error and finally frees skb that is already in the > > > > > write queue > > > > > > > > > > 5. kcm_release() does double-free the skb in the write queue > > > > > > > > > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > > > > > > > Thanks for the analysis. > > > > > > > > Since the empty skb (without payload) could cause such race and > > > > double-free issue, I wonder if we can clear the empty skb before > > > > waiting for memory, > > > > > > kcm->seq_skb is set when a part of data is copied to skb, so it's not > > > empty. Also, seq_skb is cleared when queued to the write queue. > > > > > > The problem is one thread referencing kcm->seq_skb goes to sleep and > > > another thread queues the skb to the write queue. > > > > > > ---8<--- > > > if (eor) { > > > bool not_busy = skb_queue_empty(&sk->sk_write_queue); > > > > > > if (head) { > > > /* Message complete, queue it on send buffer */ > > > __skb_queue_tail(&sk->sk_write_queue, head); > > > kcm->seq_skb = NULL; > > > KCM_STATS_INCR(kcm->stats.tx_msgs); > > > } > > > ... > > > } else { > > > /* Message not complete, save state */ > > > partial_message: > > > if (head) { > > > kcm->seq_skb = head; > > > kcm_tx_msg(head)->last_skb = skb; > > > } > > > ---8<--- > > > > Oh, I see the difference of handling error part after waiting for > > memory between tcp_sendmsg_locked and kcm_sendmsg: > > In kcm_sendmsg, it could kfree the skb which causes the issue while tcp doesn't. > > > > But I cannot help asking if that lock is a little bit heavy, please > > don't get me wrong, I'm not against it. In the meantime, I decided to > > take a deep look at the 'out_error' label part. > > I don't think the mutex is heavy because kcm_sendmsg() is already > serialised with lock_sock(). It makes sense. I have to say that it was my concern. After digging into this part, sorry, I can't find a easy way to prevent double-free issues because: initially I was trying using seq_skb (something like that) as an indicator to reflect whether we are allowed to kfree the skb, but it doesn't work for all the cases. Supposing there are three threads, each of them can call kcm_sendmsg() and wait, which makes it more complicated. Let alone more threads access this function and try to grab the lock nearly concurrently. Making the whole process serialised can make life easier. For sure, introducing mutex locks can solve the issue. Thanks, Jason
On Fri, Aug 16, 2024 at 12:04 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > syzkaller reported UAF in kcm_release(). [0] > > The scenario is > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > by sk_stream_wait_memory() > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > and puts the skb to the write queue > > 4. Thread A faces an error and finally frees skb that is already in the > write queue > > 5. kcm_release() does double-free the skb in the write queue > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > Let's add a per-sk mutex and serialise kcm_sendmsg(). > > > Fixes: ab7ac4eb9832 ("kcm: Kernel Connection Multiplexor module") > Reported-by: syzbot+b72d86aa5df17ce74c60@syzkaller.appspotmail.com > Closes: https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60 > Tested-by: syzbot+b72d86aa5df17ce74c60@syzkaller.appspotmail.com > Signed-off-by: Kuniyuki Iwashima <kuniyu@amazon.com> > --- Reviewed-by: Eric Dumazet <edumazet@google.com> I wonder if anyone is using KCM....
From: Eric Dumazet <edumazet@google.com> Date: Mon, 19 Aug 2024 17:56:55 +0200 > On Fri, Aug 16, 2024 at 12:04 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > > > syzkaller reported UAF in kcm_release(). [0] > > > > The scenario is > > > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > > by sk_stream_wait_memory() > > > > 3. Thread B calls sendmsg() concurrently, finishes building kcm->seq_skb > > and puts the skb to the write queue > > > > 4. Thread A faces an error and finally frees skb that is already in the > > write queue > > > > 5. kcm_release() does double-free the skb in the write queue > > > > When a thread is building a MSG_MORE skb, another thread must not touch it. > > > > Let's add a per-sk mutex and serialise kcm_sendmsg(). > > > > > > Fixes: ab7ac4eb9832 ("kcm: Kernel Connection Multiplexor module") > > Reported-by: syzbot+b72d86aa5df17ce74c60@syzkaller.appspotmail.com > > Closes: https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60 > > Tested-by: syzbot+b72d86aa5df17ce74c60@syzkaller.appspotmail.com > > Signed-off-by: Kuniyuki Iwashima <kuniyu@amazon.com> > > --- > > Reviewed-by: Eric Dumazet <edumazet@google.com> > > I wonder if anyone is using KCM.... Same impression :) Maybe it's time to remove KCM. https://lore.kernel.org/netdev/CALx6S37hSfQWw3Rku8vsavn8ejk0fndRk+=-=73gU7G-RbnK8Q@mail.gmail.com/
Hello: This patch was applied to netdev/net.git (main) by Jakub Kicinski <kuba@kernel.org>: On Thu, 15 Aug 2024 15:04:37 -0700 you wrote: > syzkaller reported UAF in kcm_release(). [0] > > The scenario is > > 1. Thread A builds a skb with MSG_MORE and sets kcm->seq_skb. > > 2. Thread A resumes building skb from kcm->seq_skb but is blocked > by sk_stream_wait_memory() > > [...] Here is the summary with links: - [v1,net] kcm: Serialise kcm_sendmsg() for the same socket. https://git.kernel.org/netdev/net/c/807067bf014d You are awesome, thank you!
diff --git a/include/net/kcm.h b/include/net/kcm.h index 90279e5e09a5..441e993be634 100644 --- a/include/net/kcm.h +++ b/include/net/kcm.h @@ -70,6 +70,7 @@ struct kcm_sock { struct work_struct tx_work; struct list_head wait_psock_list; struct sk_buff *seq_skb; + struct mutex tx_mutex; u32 tx_stopped : 1; /* Don't use bit fields here, these are set under different locks */ diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index 2f191e50d4fc..d4118c796290 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -755,6 +755,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) !(msg->msg_flags & MSG_MORE) : !!(msg->msg_flags & MSG_EOR); int err = -EPIPE; + mutex_lock(&kcm->tx_mutex); lock_sock(sk); /* Per tcp_sendmsg this should be in poll */ @@ -926,6 +927,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) KCM_STATS_ADD(kcm->stats.tx_bytes, copied); release_sock(sk); + mutex_unlock(&kcm->tx_mutex); return copied; out_error: @@ -951,6 +953,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) sk->sk_write_space(sk); release_sock(sk); + mutex_unlock(&kcm->tx_mutex); return err; } @@ -1204,6 +1207,7 @@ static void init_kcm_sock(struct kcm_sock *kcm, struct kcm_mux *mux) spin_unlock_bh(&mux->lock); INIT_WORK(&kcm->tx_work, kcm_tx_work); + mutex_init(&kcm->tx_mutex); spin_lock_bh(&mux->rx_lock); kcm_rcv_ready(kcm);