diff mbox series

net: do not release sk in sk_wait_event

Message ID 20240815084907.167870-1-sunyiqixm@gmail.com (mailing list archive)
State Changes Requested
Delegated to: Netdev Maintainers
Headers show
Series net: do not release sk in sk_wait_event | expand

Checks

Context Check Description
netdev/series_format warning Single patches do not need cover letters; Target tree name not specified in the subject
netdev/tree_selection success Guessed tree name to be net-next, async
netdev/ynl success Generated files up to date; no warnings/errors; no diff in generated;
netdev/fixes_present success Fixes tag not required for -next series
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 42 this patch: 42
netdev/build_tools success Errors and warnings before: 0 this patch: 0
netdev/cc_maintainers success CCed 4 of 4 maintainers
netdev/build_clang success Errors and warnings before: 62 this patch: 62
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/deprecated_api success None detected
netdev/check_selftest success No net selftest shell script
netdev/verify_fixes success Fixes tag looks correct
netdev/build_allmodconfig_warn success Errors and warnings before: 2546 this patch: 2546
netdev/checkpatch success total: 0 errors, 0 warnings, 0 checks, 14 lines checked
netdev/build_clang_rust success No Rust files in patch. Skipping build
netdev/kdoc success Errors and warnings before: 4 this patch: 4
netdev/source_inline success Was 0 now: 0

Commit Message

sunyiqi Aug. 15, 2024, 8:49 a.m. UTC 
When investigating the kcm socket UAF which is also found by syzbot,
I found that the root cause of this problem is actually in
sk_wait_event.

In sk_wait_event, sk is released and relocked and called by
sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
ops function like *sendmsg which will lock the sk at the beginning.
But sk_stream_wait_memory releases sk unexpectedly and destroy
the thread safety. Finally it causes the kcm sk UAF.

If at the time when a thread(thread A) calls sk_stream_wait_memory
and the other thread(thread B) is waiting for lock in lock_sock,
thread B will successfully get the sk lock as thread A release sk lock
in sk_wait_event.

The thread B may change the sk which is not thread A expecting.

As a result, it will lead kernel to the unexpected behavior. Just like
the kcm sk UAF, which is actually cause by sk_wait_event in
sk_stream_wait_memory.

Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
in 2016 seems do not solved this problem. Is it necessary to release
sock in sk_wait_event? Or just delete it to make the protocol ops
thread-secure.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Link: https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60
Signed-off-by: sunyiqi <sunyiqixm@gmail.com>
---
 include/net/sock.h | 2 --
 1 file changed, 2 deletions(-)

Comments

Paolo Abeni Aug. 15, 2024, 10:03 a.m. UTC | #1 
On 8/15/24 10:49, sunyiqi wrote:
> When investigating the kcm socket UAF which is also found by syzbot,
> I found that the root cause of this problem is actually in
> sk_wait_event.
> 
> In sk_wait_event, sk is released and relocked and called by
> sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> ops function like *sendmsg which will lock the sk at the beginning.
> But sk_stream_wait_memory releases sk unexpectedly and destroy
> the thread safety. Finally it causes the kcm sk UAF.
> 
> If at the time when a thread(thread A) calls sk_stream_wait_memory
> and the other thread(thread B) is waiting for lock in lock_sock,
> thread B will successfully get the sk lock as thread A release sk lock
> in sk_wait_event.
> 
> The thread B may change the sk which is not thread A expecting.
> 
> As a result, it will lead kernel to the unexpected behavior. Just like
> the kcm sk UAF, which is actually cause by sk_wait_event in
> sk_stream_wait_memory.
> 
> Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> in 2016 seems do not solved this problem. Is it necessary to release
> sock in sk_wait_event? Or just delete it to make the protocol ops
> thread-secure.

As a I wrote previously, please describe the suspected race more 
clearly, with the exact calls sequence that lead to the UAF.

Releasing the socket lock is not enough to cause UAF.

Removing the release/lock pair in sk_wait_event() will break many 
protocols (e.g. TCP) as the stack will not be able to land packets in 
the receive queue while the socked lock is owned.

Cheers,

Paolo
sunyiqi Aug. 15, 2024, 10:23 a.m. UTC | #2 
On Thu, 15 Aug 2024 12:03:37 +0200, Paolo Abeni wrote:
> On 8/15/24 10:49, sunyiqi wrote:
> > When investigating the kcm socket UAF which is also found by syzbot,
> > I found that the root cause of this problem is actually in
> > sk_wait_event.
> > 
> > In sk_wait_event, sk is released and relocked and called by
> > sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> > ops function like *sendmsg which will lock the sk at the beginning.
> > But sk_stream_wait_memory releases sk unexpectedly and destroy
> > the thread safety. Finally it causes the kcm sk UAF.
> > 
> > If at the time when a thread(thread A) calls sk_stream_wait_memory
> > and the other thread(thread B) is waiting for lock in lock_sock,
> > thread B will successfully get the sk lock as thread A release sk lock
> > in sk_wait_event.
> > 
> > The thread B may change the sk which is not thread A expecting.
> > 
> > As a result, it will lead kernel to the unexpected behavior. Just like
> > the kcm sk UAF, which is actually cause by sk_wait_event in
> > sk_stream_wait_memory.
> > 
> > Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> > in 2016 seems do not solved this problem. Is it necessary to release
> > sock in sk_wait_event? Or just delete it to make the protocol ops
> > thread-secure.
> 
> As a I wrote previously, please describe the suspected race more 
> clearly, with the exact calls sequence that lead to the UAF.
> 
> Releasing the socket lock is not enough to cause UAF.

Thread A                 Thread B
kcm_sendmsg
 lock_sock               kcm_sendmsg
                          lock_sock (blocked & waiting)
 head = sk->seq_buf
 sk_stream_wait_memory
  sk_wait_event
   release_sock
                          lock_sock (get the lock)
                          head = sk->seq_buf
                          add head to sk->sk_write_queue
                          release_sock
   lock_sock              return
 err_out to free(head)
 release_sock
 return
// ...
kcm_release
 // ...
 __skb_queue_purge(&sk->sk_write_queue) // <--- UAF
 // ...

The repro can be downloaded here:
https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60
 
> Removing the release/lock pair in sk_wait_event() will break many 
> protocols (e.g. TCP) as the stack will not be able to land packets in 
> the receive queue while the socked lock is owned.
> 
> Cheers,
> 
> Paolo
> 

Also a question about it's protocol itself should carefully use low-level
kernel API encapsulation to ensure its thread-safety or kernel API should
guarantee thread-safety since sk_wait_event or sk_stream_wait_memory used
in many cases.

Sincerely,
Yiqi Sun
Jason Xing Aug. 15, 2024, 11:14 a.m. UTC | #3 
On Thu, Aug 15, 2024 at 5:50 PM sunyiqi <sunyiqixm@gmail.com> wrote:
>
> When investigating the kcm socket UAF which is also found by syzbot,
> I found that the root cause of this problem is actually in
> sk_wait_event.
>
> In sk_wait_event, sk is released and relocked and called by
> sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> ops function like *sendmsg which will lock the sk at the beginning.
> But sk_stream_wait_memory releases sk unexpectedly and destroy
> the thread safety. Finally it causes the kcm sk UAF.
>
> If at the time when a thread(thread A) calls sk_stream_wait_memory
> and the other thread(thread B) is waiting for lock in lock_sock,
> thread B will successfully get the sk lock as thread A release sk lock
> in sk_wait_event.
>
> The thread B may change the sk which is not thread A expecting.
>
> As a result, it will lead kernel to the unexpected behavior. Just like
> the kcm sk UAF, which is actually cause by sk_wait_event in
> sk_stream_wait_memory.
>
> Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> in 2016 seems do not solved this problem. Is it necessary to release
> sock in sk_wait_event? Or just delete it to make the protocol ops
> thread-secure.
>
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Link: https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60
> Signed-off-by: sunyiqi <sunyiqixm@gmail.com>
> ---
>  include/net/sock.h | 2 --
>  1 file changed, 2 deletions(-)
>
> diff --git a/include/net/sock.h b/include/net/sock.h
> index cce23ac4d514..08d3b204b019 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -1145,7 +1145,6 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
>
>  #define sk_wait_event(__sk, __timeo, __condition, __wait)              \
>         ({      int __rc, __dis = __sk->sk_disconnects;                 \
> -               release_sock(__sk);                                     \
>                 __rc = __condition;                                     \
>                 if (!__rc) {                                            \
>                         *(__timeo) = wait_woken(__wait,                 \
> @@ -1153,7 +1152,6 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
>                                                 *(__timeo));            \
>                 }                                                       \
>                 sched_annotate_sleep();                                 \
> -               lock_sock(__sk);                                        \

Are you sure that you want the socket lock to be held possibly for a
really long time even if it has to wait for the available memory,
which means, during this period, other threads trying to access the
lock will be blocked?

Thanks,
Jason
Kuniyuki Iwashima Aug. 15, 2024, 7:56 p.m. UTC | #4 
From: sunyiqi <sunyiqixm@gmail.com>
Date: Thu, 15 Aug 2024 18:23:29 +0800
> On Thu, 15 Aug 2024 12:03:37 +0200, Paolo Abeni wrote:
> > On 8/15/24 10:49, sunyiqi wrote:
> > > When investigating the kcm socket UAF which is also found by syzbot,
> > > I found that the root cause of this problem is actually in
> > > sk_wait_event.
> > > 
> > > In sk_wait_event, sk is released and relocked and called by
> > > sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> > > ops function like *sendmsg which will lock the sk at the beginning.
> > > But sk_stream_wait_memory releases sk unexpectedly and destroy
> > > the thread safety. Finally it causes the kcm sk UAF.
> > > 
> > > If at the time when a thread(thread A) calls sk_stream_wait_memory
> > > and the other thread(thread B) is waiting for lock in lock_sock,
> > > thread B will successfully get the sk lock as thread A release sk lock
> > > in sk_wait_event.
> > > 
> > > The thread B may change the sk which is not thread A expecting.
> > > 
> > > As a result, it will lead kernel to the unexpected behavior. Just like
> > > the kcm sk UAF, which is actually cause by sk_wait_event in
> > > sk_stream_wait_memory.
> > > 
> > > Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> > > in 2016 seems do not solved this problem. Is it necessary to release
> > > sock in sk_wait_event? Or just delete it to make the protocol ops
> > > thread-secure.
> > 
> > As a I wrote previously, please describe the suspected race more 
> > clearly, with the exact calls sequence that lead to the UAF.
> > 
> > Releasing the socket lock is not enough to cause UAF.
> 
> Thread A                 Thread B
> kcm_sendmsg
>  lock_sock               kcm_sendmsg
>                           lock_sock (blocked & waiting)
>  head = sk->seq_buf
>  sk_stream_wait_memory
>   sk_wait_event
>    release_sock
>                           lock_sock (get the lock)
>                           head = sk->seq_buf
>                           add head to sk->sk_write_queue
>                           release_sock
>    lock_sock              return
>  err_out to free(head)
>  release_sock
>  return
> // ...
> kcm_release
>  // ...
>  __skb_queue_purge(&sk->sk_write_queue) // <--- UAF
>  // ...
> 
> The repro can be downloaded here:
> https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60

When a thread is building a skb with MSG_MORE, another thread
must not touch it nor complete building it by queuing it to
write queue and setting NULL to kcm->seq_skb.

I think the correct fix is simply serialise them with mutex.

---8<---
diff --git a/include/net/kcm.h b/include/net/kcm.h
index 90279e5e09a5..441e993be634 100644
--- a/include/net/kcm.h
+++ b/include/net/kcm.h
@@ -70,6 +70,7 @@ struct kcm_sock {
 	struct work_struct tx_work;
 	struct list_head wait_psock_list;
 	struct sk_buff *seq_skb;
+	struct mutex tx_mutex;
 	u32 tx_stopped : 1;
 
 	/* Don't use bit fields here, these are set under different locks */
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index 2f191e50d4fc..d4118c796290 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -755,6 +755,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 		  !(msg->msg_flags & MSG_MORE) : !!(msg->msg_flags & MSG_EOR);
 	int err = -EPIPE;
 
+	mutex_lock(&kcm->tx_mutex);
 	lock_sock(sk);
 
 	/* Per tcp_sendmsg this should be in poll */
@@ -926,6 +927,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 	KCM_STATS_ADD(kcm->stats.tx_bytes, copied);
 
 	release_sock(sk);
+	mutex_unlock(&kcm->tx_mutex);
 	return copied;
 
 out_error:
@@ -951,6 +953,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 		sk->sk_write_space(sk);
 
 	release_sock(sk);
+	mutex_unlock(&kcm->tx_mutex);
 	return err;
 }
 
@@ -1204,6 +1207,7 @@ static void init_kcm_sock(struct kcm_sock *kcm, struct kcm_mux *mux)
 	spin_unlock_bh(&mux->lock);
 
 	INIT_WORK(&kcm->tx_work, kcm_tx_work);
+	mutex_init(&kcm->tx_mutex);
 
 	spin_lock_bh(&mux->rx_lock);
 	kcm_rcv_ready(kcm);
---8<---


We can allow another thread to complete building skb by the
following but it doesn't sound correct to me.

---8<---
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index 2f191e50d4fc..51f2409d6113 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -748,7 +748,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 {
 	struct sock *sk = sock->sk;
 	struct kcm_sock *kcm = kcm_sk(sk);
-	struct sk_buff *skb = NULL, *head = NULL;
+	struct sk_buff *skb = NULL, *head = NULL, *seq_skb;
 	size_t copy, copied = 0;
 	long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
 	int eor = (sock->type == SOCK_DGRAM) ?
@@ -763,6 +763,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 	if (sk->sk_err)
 		goto out_error;
 
+	seq_skb = kcm->seq_skb;
 	if (kcm->seq_skb) {
 		/* Previously opened message */
 		head = kcm->seq_skb;
@@ -888,6 +889,8 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 		err = sk_stream_wait_memory(sk, &timeo);
 		if (err)
 			goto out_error;
+		if (seq_skb && seq_skb != kcm->seq_skb)
+			goto out_stolen;
 	}
 
 	if (eor) {
@@ -943,7 +946,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
 		kfree_skb(head);
 		kcm->seq_skb = NULL;
 	}
-
+out_stolen:
 	err = sk_stream_error(sk, msg->msg_flags, err);
 
 	/* make sure we wake any epoll edge trigger waiter */
---8<---
Kuniyuki Iwashima Aug. 15, 2024, 10:07 p.m. UTC | #5 
From: Kuniyuki Iwashima <kuniyu@amazon.com>
Date: Thu, 15 Aug 2024 12:56:45 -0700
> From: sunyiqi <sunyiqixm@gmail.com>
> Date: Thu, 15 Aug 2024 18:23:29 +0800
> > On Thu, 15 Aug 2024 12:03:37 +0200, Paolo Abeni wrote:
> > > On 8/15/24 10:49, sunyiqi wrote:
> > > > When investigating the kcm socket UAF which is also found by syzbot,
> > > > I found that the root cause of this problem is actually in
> > > > sk_wait_event.
> > > > 
> > > > In sk_wait_event, sk is released and relocked and called by
> > > > sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> > > > ops function like *sendmsg which will lock the sk at the beginning.
> > > > But sk_stream_wait_memory releases sk unexpectedly and destroy
> > > > the thread safety. Finally it causes the kcm sk UAF.
> > > > 
> > > > If at the time when a thread(thread A) calls sk_stream_wait_memory
> > > > and the other thread(thread B) is waiting for lock in lock_sock,
> > > > thread B will successfully get the sk lock as thread A release sk lock
> > > > in sk_wait_event.
> > > > 
> > > > The thread B may change the sk which is not thread A expecting.
> > > > 
> > > > As a result, it will lead kernel to the unexpected behavior. Just like
> > > > the kcm sk UAF, which is actually cause by sk_wait_event in
> > > > sk_stream_wait_memory.
> > > > 
> > > > Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> > > > in 2016 seems do not solved this problem. Is it necessary to release
> > > > sock in sk_wait_event? Or just delete it to make the protocol ops
> > > > thread-secure.
> > > 
> > > As a I wrote previously, please describe the suspected race more 
> > > clearly, with the exact calls sequence that lead to the UAF.
> > > 
> > > Releasing the socket lock is not enough to cause UAF.
> > 
> > Thread A                 Thread B
> > kcm_sendmsg
> >  lock_sock               kcm_sendmsg
> >                           lock_sock (blocked & waiting)
> >  head = sk->seq_buf
> >  sk_stream_wait_memory
> >   sk_wait_event
> >    release_sock
> >                           lock_sock (get the lock)
> >                           head = sk->seq_buf
> >                           add head to sk->sk_write_queue
> >                           release_sock
> >    lock_sock              return
> >  err_out to free(head)
> >  release_sock
> >  return
> > // ...
> > kcm_release
> >  // ...
> >  __skb_queue_purge(&sk->sk_write_queue) // <--- UAF
> >  // ...
> > 
> > The repro can be downloaded here:
> > https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60
> 
> When a thread is building a skb with MSG_MORE, another thread
> must not touch it nor complete building it by queuing it to
> write queue and setting NULL to kcm->seq_skb.
> 
> I think the correct fix is simply serialise them with mutex.

FTR, syzbot was happy with the change below, so I posted it officially.
https://lore.kernel.org/netdev/20240815220437.69511-1-kuniyu@amazon.com/


> ---8<---
> diff --git a/include/net/kcm.h b/include/net/kcm.h
> index 90279e5e09a5..441e993be634 100644
> --- a/include/net/kcm.h
> +++ b/include/net/kcm.h
> @@ -70,6 +70,7 @@ struct kcm_sock {
>  	struct work_struct tx_work;
>  	struct list_head wait_psock_list;
>  	struct sk_buff *seq_skb;
> +	struct mutex tx_mutex;
>  	u32 tx_stopped : 1;
>  
>  	/* Don't use bit fields here, these are set under different locks */
> diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
> index 2f191e50d4fc..d4118c796290 100644
> --- a/net/kcm/kcmsock.c
> +++ b/net/kcm/kcmsock.c
> @@ -755,6 +755,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
>  		  !(msg->msg_flags & MSG_MORE) : !!(msg->msg_flags & MSG_EOR);
>  	int err = -EPIPE;
>  
> +	mutex_lock(&kcm->tx_mutex);
>  	lock_sock(sk);
>  
>  	/* Per tcp_sendmsg this should be in poll */
> @@ -926,6 +927,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
>  	KCM_STATS_ADD(kcm->stats.tx_bytes, copied);
>  
>  	release_sock(sk);
> +	mutex_unlock(&kcm->tx_mutex);
>  	return copied;
>  
>  out_error:
> @@ -951,6 +953,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
>  		sk->sk_write_space(sk);
>  
>  	release_sock(sk);
> +	mutex_unlock(&kcm->tx_mutex);
>  	return err;
>  }
>  
> @@ -1204,6 +1207,7 @@ static void init_kcm_sock(struct kcm_sock *kcm, struct kcm_mux *mux)
>  	spin_unlock_bh(&mux->lock);
>  
>  	INIT_WORK(&kcm->tx_work, kcm_tx_work);
> +	mutex_init(&kcm->tx_mutex);
>  
>  	spin_lock_bh(&mux->rx_lock);
>  	kcm_rcv_ready(kcm);
> ---8<---
Oliver Sang Aug. 21, 2024, 7:38 a.m. UTC | #6 
Hello,

kernel test robot noticed a -100.0% regression of stress-ng.sock.ops_per_sec on:


commit: 485a7c1b69390b4db80d68c086dc5b619465c9c6 ("[PATCH] net: do not release sk in sk_wait_event")
url: https://github.com/intel-lab-lkp/linux/commits/sunyiqi/net-do-not-release-sk-in-sk_wait_event/20240815-175112
base: https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git a9c60712d71ff07197b2982899b9db28ed548ded
patch link: https://lore.kernel.org/all/20240815084907.167870-1-sunyiqixm@gmail.com/
patch subject: [PATCH] net: do not release sk in sk_wait_event

testcase: stress-ng
test machine: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory
parameters:

	nr_threads: 100%
	testtime: 60s
	test: sock
	cpufreq_governor: performance




If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202408211524.546a8e56-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240821/202408211524.546a8e56-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  gcc-12/performance/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/lkp-spr-r02/sock/stress-ng/60s

commit: 
  a9c60712d7 ("Merge branch 'uapi-net-sched-cxgb4-fix-wflex-array-member-not-at-end-warning'")
  485a7c1b69 ("net: do not release sk in sk_wait_event")

a9c60712d71ff071 485a7c1b69390b4db80d68c086d 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      7918          +172.2%      21554        uptime.idle
 4.039e+08 ±  4%   +3375.4%  1.404e+10        cpuidle..time
   3139094 ± 54%     -65.0%    1097704        cpuidle..usage
  93316867           -99.6%     370101 ± 18%  numa-numastat.node0.local_node
  93427347           -99.4%     517330 ± 12%  numa-numastat.node0.numa_hit
  92697012           -99.6%     339810 ± 20%  numa-numastat.node1.local_node
  92828493           -99.5%     424633 ± 15%  numa-numastat.node1.numa_hit
     58497 ±  7%    -100.0%       7.17 ± 84%  perf-c2c.DRAM.local
      2382 ±  7%     -98.9%      27.17 ± 20%  perf-c2c.DRAM.remote
      6334 ±  6%     -99.5%      28.67 ± 18%  perf-c2c.HITM.local
    539.33 ±  5%     -96.8%      17.50 ± 31%  perf-c2c.HITM.remote
      6873 ±  6%     -99.3%      46.17 ± 18%  perf-c2c.HITM.total
      5.25 ± 10%   +1799.0%      99.75        vmstat.cpu.id
     91.82           -99.9%       0.14 ±  7%  vmstat.cpu.sy
    357.35           -99.2%       2.68 ± 50%  vmstat.procs.r
  27285633          -100.0%       2512 ±  4%  vmstat.system.cs
    529503           -95.4%      24285 ±  7%  vmstat.system.in
      3.05 ± 17%     +96.6       99.67        mpstat.cpu.all.idle%
      0.00 ±100%      +0.0        0.02 ± 19%  mpstat.cpu.all.iowait%
      0.48            -0.4        0.04 ±  3%  mpstat.cpu.all.irq%
     19.79           -19.8        0.00 ± 36%  mpstat.cpu.all.soft%
     73.65           -73.6        0.08        mpstat.cpu.all.sys%
      3.02            -2.8        0.18        mpstat.cpu.all.usr%
     12.67 ± 56%     -92.1%       1.00        mpstat.max_utilization.seconds
    100.00           -97.6%       2.35 ±  2%  mpstat.max_utilization_pct
    512298 ± 18%     -53.4%     238782 ±  5%  numa-meminfo.node0.Mapped
     24524 ± 39%     +62.3%      39800 ±  5%  numa-meminfo.node0.PageTables
    566127 ±  3%     -95.8%      24008 ±  5%  numa-meminfo.node1.Active
    566113 ±  3%     -95.8%      23992 ±  5%  numa-meminfo.node1.Active(anon)
   2747024 ± 31%     -83.3%     458096 ± 39%  numa-meminfo.node1.Inactive
   2746948 ± 31%     -83.3%     457980 ± 39%  numa-meminfo.node1.Inactive(anon)
    967060 ± 11%     -99.0%      10023 ±121%  numa-meminfo.node1.Mapped
     27090 ± 35%     -65.6%       9308 ± 17%  numa-meminfo.node1.PageTables
   2794844 ± 30%     -99.1%      26390 ±  6%  numa-meminfo.node1.Shmem
    569024 ±  2%     -95.0%      28197        meminfo.Active
    568987 ±  2%     -95.0%      28165        meminfo.Active(anon)
   6545198 ± 16%     -49.1%    3334716        meminfo.Cached
   8122714 ± 13%     -39.8%    4892326        meminfo.Committed_AS
   3879368 ± 24%     -66.5%    1297885 ± 12%  meminfo.Inactive
   3879138 ± 24%     -66.5%    1297660 ± 12%  meminfo.Inactive(anon)
   1477918 ±  2%     -83.2%     248731        meminfo.Mapped
  10781439 ±  8%     -28.0%    7762846        meminfo.Memused
   3461321 ± 30%     -92.8%     250853        meminfo.Shmem
  10898889 ±  8%     -15.6%    9202079        meminfo.max_used_kB
    834112          -100.0%       0.00        stress-ng.sock.byte_average_in_queue_length
    798960 ± 35%    +224.0%    2588527        stress-ng.sock.byte_average_out_queue_length
    173337          -100.0%      12.46        stress-ng.sock.messages_sent_per_sec
   4561553          -100.0%     448.00        stress-ng.sock.ops
     76023          -100.0%       7.34        stress-ng.sock.ops_per_sec
 8.575e+08          -100.0%      64.83 ± 14%  stress-ng.time.involuntary_context_switches
     96104            -4.0%      92291        stress-ng.time.minor_page_faults
     17471          -100.0%       1.00        stress-ng.time.percent_of_cpu_this_job_got
     10191          -100.0%       0.71 ±  3%  stress-ng.time.system_time
    334.28           -99.9%       0.17 ±  5%  stress-ng.time.user_time
 8.622e+08          -100.0%       2163        stress-ng.time.voluntary_context_switches
    128351 ± 18%     -54.0%      59054 ±  5%  numa-vmstat.node0.nr_mapped
      6176 ± 40%     +60.3%       9903 ±  5%  numa-vmstat.node0.nr_page_table_pages
  93396523           -99.4%     517565 ± 12%  numa-vmstat.node0.numa_hit
  93286044           -99.6%     370336 ± 18%  numa-vmstat.node0.numa_local
    140292 ±  3%     -95.7%       5974 ±  6%  numa-vmstat.node1.nr_active_anon
    686202 ± 31%     -83.3%     114574 ± 39%  numa-vmstat.node1.nr_inactive_anon
    240443 ± 12%     -98.9%       2638 ±117%  numa-vmstat.node1.nr_mapped
      6791 ± 35%     -65.8%       2321 ± 17%  numa-vmstat.node1.nr_page_table_pages
    696959 ± 30%     -99.1%       6577 ±  6%  numa-vmstat.node1.nr_shmem
    140290 ±  3%     -95.7%       5974 ±  6%  numa-vmstat.node1.nr_zone_active_anon
    686199 ± 31%     -83.3%     114573 ± 39%  numa-vmstat.node1.nr_zone_inactive_anon
  92804117           -99.5%     423580 ± 15%  numa-vmstat.node1.numa_hit
  92672638           -99.6%     338761 ± 20%  numa-vmstat.node1.numa_local
    140679 ±  2%     -94.9%       7154        proc-vmstat.nr_active_anon
   1632998 ± 16%     -49.0%     833351        proc-vmstat.nr_file_pages
    968473 ± 23%     -66.7%     322808 ± 12%  proc-vmstat.nr_inactive_anon
    367618 ±  2%     -83.2%      61923        proc-vmstat.nr_mapped
    862027 ± 30%     -92.8%      62383        proc-vmstat.nr_shmem
     38965            -6.8%      36332        proc-vmstat.nr_slab_reclaimable
    124313            -6.0%     116811        proc-vmstat.nr_slab_unreclaimable
    140679 ±  2%     -94.9%       7154        proc-vmstat.nr_zone_active_anon
    968473 ± 23%     -66.7%     322808 ± 12%  proc-vmstat.nr_zone_inactive_anon
    246757 ± 18%     -94.8%      12714 ± 44%  proc-vmstat.numa_hint_faults
    131080 ± 12%     -99.4%     810.83 ± 79%  proc-vmstat.numa_hint_faults_local
 1.861e+08           -99.5%     944616        proc-vmstat.numa_hit
 1.859e+08           -99.6%     712564        proc-vmstat.numa_local
     21455 ± 29%     -44.5%      11903 ± 44%  proc-vmstat.numa_pages_migrated
    704523 ± 13%     -91.2%      62163 ± 44%  proc-vmstat.numa_pte_updates
 1.471e+09           -99.9%    1226836 ±  3%  proc-vmstat.pgalloc_normal
   1170858 ±  4%     -51.4%     569424        proc-vmstat.pgfault
 1.469e+09           -99.9%    1228672 ±  6%  proc-vmstat.pgfree
     21455 ± 29%     -44.5%      11903 ± 44%  proc-vmstat.pgmigrate_success
      1.51           -36.8%       0.95 ±  4%  perf-stat.i.MPKI
 9.383e+10           -99.4%  6.083e+08        perf-stat.i.branch-instructions
      0.43            +3.4        3.79        perf-stat.i.branch-miss-rate%
 3.712e+08           -91.8%   30507919        perf-stat.i.branch-misses
     19.35           -13.1        6.22 ±  6%  perf-stat.i.cache-miss-rate%
 6.833e+08           -99.8%    1669451 ± 10%  perf-stat.i.cache-misses
 3.385e+09           -99.6%   12178757        perf-stat.i.cache-references
  28327876          -100.0%       2265        perf-stat.i.context-switches
      1.35           +29.9%       1.75        perf-stat.i.cpi
 6.377e+11           -99.6%  2.637e+09        perf-stat.i.cpu-cycles
      1052 ±  7%     -71.3%     301.76 ±  2%  perf-stat.i.cpu-migrations
      1429 ±  6%    +394.8%       7073 ± 13%  perf-stat.i.cycles-between-cache-misses
 4.857e+11           -99.4%  2.988e+09        perf-stat.i.instructions
      0.76           +10.1%       0.84        perf-stat.i.ipc
    125.84           -98.2%       2.24 ±  4%  perf-stat.i.metric.K/sec
     17986 ±  6%     -59.8%       7222 ±  2%  perf-stat.i.minor-faults
     17986 ±  6%     -59.8%       7222 ±  2%  perf-stat.i.page-faults
      1.43           -61.5%       0.55 ± 11%  perf-stat.overall.MPKI
      0.39            +4.6        5.02        perf-stat.overall.branch-miss-rate%
     20.31            -6.8       13.48 ± 10%  perf-stat.overall.cache-miss-rate%
      1.32           -33.0%       0.88        perf-stat.overall.cpi
    925.76           +75.9%       1628 ± 10%  perf-stat.overall.cycles-between-cache-misses
      0.76           +49.3%       1.13        perf-stat.overall.ipc
  9.15e+10           -99.3%  5.959e+08        perf-stat.ps.branch-instructions
 3.613e+08           -91.7%   29926582        perf-stat.ps.branch-misses
 6.756e+08           -99.8%    1608044 ± 10%  perf-stat.ps.cache-misses
 3.326e+09           -99.6%   11926187        perf-stat.ps.cache-references
  27798409          -100.0%       2219        perf-stat.ps.context-switches
 6.254e+11           -99.6%  2.588e+09        perf-stat.ps.cpu-cycles
    941.05 ±  8%     -68.5%     296.14 ±  2%  perf-stat.ps.cpu-migrations
 4.739e+11           -99.4%  2.928e+09        perf-stat.ps.instructions
     16959 ±  4%     -59.0%       6953 ±  2%  perf-stat.ps.minor-faults
     16960 ±  4%     -59.0%       6953 ±  2%  perf-stat.ps.page-faults
   2.9e+13           -99.4%  1.818e+11        perf-stat.total.instructions
   5204371          -100.0%       1604 ±  7%  sched_debug.cfs_rq:/.avg_vruntime.avg
   7682820 ± 15%     -99.6%      30824 ± 23%  sched_debug.cfs_rq:/.avg_vruntime.max
   4090321 ±  7%    -100.0%     143.87 ± 33%  sched_debug.cfs_rq:/.avg_vruntime.min
    255159 ± 25%     -98.6%       3519 ± 11%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.90 ±  2%     -93.8%       0.06 ± 18%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.50          -100.0%       0.00        sched_debug.cfs_rq:/.h_nr_running.min
      0.34 ±  2%     -32.5%       0.23 ± 10%  sched_debug.cfs_rq:/.h_nr_running.stddev
      2382 ± 11%   +1019.8%      26682 ±189%  sched_debug.cfs_rq:/.load.stddev
     30048 ± 68%    +181.8%      84679 ±  7%  sched_debug.cfs_rq:/.load_avg.max
      3973 ± 74%    +194.0%      11682 ± 32%  sched_debug.cfs_rq:/.load_avg.stddev
   5204371          -100.0%       1604 ±  7%  sched_debug.cfs_rq:/.min_vruntime.avg
   7682820 ± 15%     -99.6%      30823 ± 23%  sched_debug.cfs_rq:/.min_vruntime.max
   4090321 ±  7%    -100.0%     143.87 ± 33%  sched_debug.cfs_rq:/.min_vruntime.min
    255159 ± 25%     -98.6%       3519 ± 11%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.53           -89.4%       0.06 ± 18%  sched_debug.cfs_rq:/.nr_running.avg
      0.50          -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.min
      0.12 ±  5%     +99.4%       0.23 ± 10%  sched_debug.cfs_rq:/.nr_running.stddev
      5.33 ± 67%    +173.4%      14.56 ± 32%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    247.25 ± 13%    +121.9%     548.67 ±  3%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     32.60 ± 40%    +156.1%      83.51 ± 16%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      5.33 ± 67%    +173.4%      14.56 ± 32%  sched_debug.cfs_rq:/.removed.util_avg.avg
    247.25 ± 13%    +121.9%     548.67 ±  3%  sched_debug.cfs_rq:/.removed.util_avg.max
     32.60 ± 40%    +156.1%      83.51 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    936.50           -83.3%     156.14 ±  8%  sched_debug.cfs_rq:/.runnable_avg.avg
      2094 ±  3%     -46.0%       1130 ±  7%  sched_debug.cfs_rq:/.runnable_avg.max
    586.33 ± 17%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.min
    188.89 ±  5%     +32.9%     251.07 ±  6%  sched_debug.cfs_rq:/.runnable_avg.stddev
    585.27           -73.5%     155.25 ±  8%  sched_debug.cfs_rq:/.util_avg.avg
    438.83 ± 22%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.min
    126.75 ±  7%     +97.0%     249.71 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
    652.42           -98.7%       8.73 ± 55%  sched_debug.cfs_rq:/.util_est.avg
      1409 ±  7%     -62.1%     533.50 ± 10%  sched_debug.cfs_rq:/.util_est.max
    400.25 ±  6%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est.min
    147.08 ±  9%     -57.9%      61.85 ± 25%  sched_debug.cfs_rq:/.util_est.stddev
    519373           +76.2%     915125        sched_debug.cpu.avg_idle.avg
   1258453 ±  8%     +28.1%    1611863 ± 21%  sched_debug.cpu.avg_idle.max
    193107 ±  5%     -12.7%     168533 ±  5%  sched_debug.cpu.avg_idle.stddev
     69255           -43.3%      39244        sched_debug.cpu.clock.avg
     69321           -43.4%      39260        sched_debug.cpu.clock.max
     69168           -43.3%      39220        sched_debug.cpu.clock.min
     42.19 ±  9%     -80.6%       8.20 ±  6%  sched_debug.cpu.clock.stddev
     62909           -37.9%      39085        sched_debug.cpu.clock_task.avg
     65633           -40.2%      39228        sched_debug.cpu.clock_task.max
     48891           -48.9%      24973 ±  2%  sched_debug.cpu.clock_task.min
      1059            -9.4%     959.90 ±  3%  sched_debug.cpu.clock_task.stddev
      3521           -94.1%     206.83 ± 19%  sched_debug.cpu.curr->pid.avg
      6093 ±  4%     -36.7%       3854        sched_debug.cpu.curr->pid.max
      1825 ± 45%    -100.0%       0.00        sched_debug.cpu.curr->pid.min
    582.09 ±  8%     +44.5%     841.37 ±  9%  sched_debug.cpu.curr->pid.stddev
    625107 ±  6%     +34.8%     842597 ± 24%  sched_debug.cpu.max_idle_balance_cost.max
     11913 ± 39%    +148.1%      29559 ± 44%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  6%     -72.2%       0.00 ± 17%  sched_debug.cpu.next_balance.stddev
      0.90 ±  2%     -93.7%       0.06 ± 20%  sched_debug.cpu.nr_running.avg
      0.50          -100.0%       0.00        sched_debug.cpu.nr_running.min
      0.33 ±  2%     -30.6%       0.23 ± 11%  sched_debug.cpu.nr_running.stddev
   3783489          -100.0%     601.41        sched_debug.cpu.nr_switches.avg
   4086725           -99.7%      12429 ±  9%  sched_debug.cpu.nr_switches.max
   1967335 ± 16%    -100.0%     115.67 ± 11%  sched_debug.cpu.nr_switches.min
    225133 ±  6%     -99.5%       1176 ±  5%  sched_debug.cpu.nr_switches.stddev
     69172           -43.3%      39230        sched_debug.cpu_clk
     68115           -44.0%      38175        sched_debug.ktime
     70010           -42.8%      40067        sched_debug.sched_clk
     73.23 ±  2%     -73.2        0.00        perf-profile.calltrace.cycles-pp.stress_sock
     49.51           -49.5        0.00        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     47.66           -47.7        0.00        perf-profile.calltrace.cycles-pp.tcp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.41           -44.4        0.00        perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     38.68           -38.7        0.00        perf-profile.calltrace.cycles-pp.recv.stress_sock
     38.10           -38.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.stress_sock
     38.02           -38.0        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.stress_sock
     37.20           -37.2        0.00        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
     36.42           -36.4        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.stress_sock
     36.27           -36.3        0.00        perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     35.94           -35.9        0.00        perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     35.87           -35.9        0.00        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     35.06           -35.1        0.00        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
     33.43 ±  5%     -33.4        0.00        perf-profile.calltrace.cycles-pp.__send.stress_sock
     32.80 ±  5%     -32.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.stress_sock
     32.72 ±  5%     -32.7        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.stress_sock
     29.87 ±  5%     -29.9        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.stress_sock
     23.84 ±  8%     -23.8        0.00        perf-profile.calltrace.cycles-pp.__send
     22.48 ±  8%     -22.5        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
     22.43 ±  8%     -22.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     19.73 ±  8%     -19.7        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     18.54           -18.5        0.00        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     18.49           -18.5        0.00        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
     18.27           -18.3        0.00        perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
     17.36           -17.4        0.00        perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
     16.87           -16.9        0.00        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip
     16.86           -16.9        0.00        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto.__x64_sys_sendto
     16.78           -16.8        0.00        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.handle_softirqs.do_softirq
     16.70           -16.7        0.00        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto
     16.21           -16.2        0.00        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.handle_softirqs
     14.79           -14.8        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     14.37           -14.4        0.00        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     14.10           -14.1        0.00        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     14.10           -14.1        0.00        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     13.28           -13.3        0.00        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     13.14           -13.1        0.00        perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto.__x64_sys_sendto
     13.12           -13.1        0.00        perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto
     13.04           -13.0        0.00        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
     12.72           -12.7        0.00        perf-profile.calltrace.cycles-pp.alloc_pages_mpol_noprof.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg
     12.70           -12.7        0.00        perf-profile.calltrace.cycles-pp.__alloc_pages_noprof.alloc_pages_mpol_noprof.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked
     12.65           -12.7        0.00        perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_noprof.alloc_pages_mpol_noprof.skb_page_frag_refill.sk_page_frag_refill
     12.39           -12.4        0.00        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
     12.21           -12.2        0.00        perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages_noprof.alloc_pages_mpol_noprof.skb_page_frag_refill
     12.10           -12.1        0.00        perf-profile.calltrace.cycles-pp.__rmqueue_pcplist.rmqueue.get_page_from_freelist.__alloc_pages_noprof.alloc_pages_mpol_noprof
     12.06           -12.1        0.00        perf-profile.calltrace.cycles-pp.rmqueue_bulk.__rmqueue_pcplist.rmqueue.get_page_from_freelist.__alloc_pages_noprof
     11.92 ±  2%     -11.9        0.00        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
     11.72           -11.7        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rmqueue_bulk.__rmqueue_pcplist.rmqueue.get_page_from_freelist
     11.68           -11.7        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.rmqueue_bulk.__rmqueue_pcplist.rmqueue
     11.40 ±  2%     -11.4        0.00        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      9.77            -9.8        0.00        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      9.08            -9.1        0.00        perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      8.90            -8.9        0.00        perf-profile.calltrace.cycles-pp.skb_attempt_defer_free.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      8.86            -8.9        0.00        perf-profile.calltrace.cycles-pp.skb_release_data.skb_attempt_defer_free.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      8.71            -8.7        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.free_pcppages_bulk.free_unref_page_commit.free_unref_page.skb_release_data
      8.67            -8.7        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.free_pcppages_bulk.free_unref_page_commit.free_unref_page
      8.64            -8.6        0.00        perf-profile.calltrace.cycles-pp.free_unref_page.skb_release_data.skb_attempt_defer_free.tcp_recvmsg_locked.tcp_recvmsg
      8.48            -8.5        0.00        perf-profile.calltrace.cycles-pp.free_unref_page_commit.free_unref_page.skb_release_data.skb_attempt_defer_free.tcp_recvmsg_locked
      8.41            -8.4        0.00        perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
      8.33            -8.3        0.00        perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_commit.free_unref_page.skb_release_data.skb_attempt_defer_free
      8.13            -8.1        0.00        perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
      6.78            -6.8        0.00        perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      6.51            -6.5        0.00        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      6.46            -6.5        0.00        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      6.41            -6.4        0.00        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      5.78            -5.8        0.00        perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      5.50            -5.5        0.00        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      5.50            -5.5        0.00        perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      5.27            -5.3        0.00        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      5.10            -5.1        0.00        perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      0.00            +0.5        0.53 ±  7%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main
      0.00            +0.5        0.53 ±  7%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main
      0.00            +0.6        0.64 ± 16%  perf-profile.calltrace.cycles-pp.next_uptodate_folio.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
      0.00            +0.7        0.67 ± 15%  perf-profile.calltrace.cycles-pp.__do_set_cpus_allowed.__set_cpus_allowed_ptr_locked.__set_cpus_allowed_ptr.__sched_setaffinity.sched_setaffinity
      0.00            +0.7        0.69 ± 12%  perf-profile.calltrace.cycles-pp.rcu_gp_fqs_loop.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +0.7        0.70 ± 12%  perf-profile.calltrace.cycles-pp.ct_idle_exit.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +0.7        0.71 ± 14%  perf-profile.calltrace.cycles-pp.sched_balance_find_dst_cpu.select_task_rq_fair.sched_exec.bprm_execve.do_execveat_common
      0.00            +0.7        0.71 ± 14%  perf-profile.calltrace.cycles-pp.sched_balance_find_dst_group.sched_balance_find_dst_cpu.select_task_rq_fair.sched_exec.bprm_execve
      0.00            +0.7        0.71 ± 14%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.sched_exec.bprm_execve.do_execveat_common.__x64_sys_execve
      0.00            +0.7        0.71 ± 15%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.71 ± 15%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00            +0.7        0.72 ± 14%  perf-profile.calltrace.cycles-pp.sched_exec.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      0.00            +0.7        0.72 ±  9%  perf-profile.calltrace.cycles-pp.show_stat.seq_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00            +0.7        0.75 ± 19%  perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.76 ±  8%  perf-profile.calltrace.cycles-pp.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +0.8        0.77 ± 19%  perf-profile.calltrace.cycles-pp.__set_cpus_allowed_ptr_locked.__set_cpus_allowed_ptr.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity
      0.00            +0.8        0.77 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
      0.00            +0.8        0.77 ± 13%  perf-profile.calltrace.cycles-pp.kernel_clone.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
      0.00            +0.8        0.78 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
      0.00            +0.8        0.78 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__vfork
      0.00            +0.8        0.78 ± 15%  perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.78 ± 15%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.8        0.80 ± 18%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.8        0.80 ± 21%  perf-profile.calltrace.cycles-pp.update_rq_clock_task._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.8        0.81 ± 18%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.8        0.84 ± 17%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.8        0.84 ± 17%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.9        0.85 ± 18%  perf-profile.calltrace.cycles-pp.sched_tick.update_process_times.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.9        0.89 ± 18%  perf-profile.calltrace.cycles-pp.write
      0.00            +0.9        0.90 ± 12%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.exit_mm.do_exit.do_group_exit
      0.00            +0.9        0.90 ± 12%  perf-profile.calltrace.cycles-pp.mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
      0.00            +0.9        0.90 ± 12%  perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
      0.00            +0.9        0.92 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_restart_sched_tick.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.9        0.93 ± 35%  perf-profile.calltrace.cycles-pp.read.readn.perf_evsel__read.read_counters.process_interval
      0.00            +0.9        0.94 ±  9%  perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      0.00            +1.0        0.96 ± 16%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
      0.00            +1.0        0.96 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +1.0        0.97 ± 13%  perf-profile.calltrace.cycles-pp.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity
      0.00            +1.0        0.97 ± 12%  perf-profile.calltrace.cycles-pp.__vfork
      0.00            +1.0        0.97 ± 10%  perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.00            +1.0        0.97 ± 17%  perf-profile.calltrace.cycles-pp.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm
      0.00            +1.0        0.99 ± 35%  perf-profile.calltrace.cycles-pp.readn.perf_evsel__read.read_counters.process_interval.dispatch_events
      0.00            +1.0        1.02 ± 15%  perf-profile.calltrace.cycles-pp.move_queued_task.migration_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread
      0.00            +1.0        1.03 ± 25%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      0.00            +1.0        1.04 ± 24%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.05 ±  8%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.06 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.07 ± 25%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.free_pcppages_bulk.decay_pcp_high.refresh_cpu_vm_stats
      0.00            +1.1        1.08 ± 24%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.free_pcppages_bulk.decay_pcp_high.refresh_cpu_vm_stats.vmstat_update
      0.00            +1.1        1.10 ± 12%  perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      0.00            +1.1        1.10 ± 10%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.00            +1.1        1.10 ± 36%  perf-profile.calltrace.cycles-pp.perf_evsel__read.read_counters.process_interval.dispatch_events.cmd_stat
      0.00            +1.1        1.12 ± 10%  perf-profile.calltrace.cycles-pp.seq_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.13 ± 51%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      0.00            +1.1        1.14 ± 14%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +1.2        1.15 ± 16%  perf-profile.calltrace.cycles-pp.migration_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +1.2        1.18 ± 26%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.readn.evsel__read_counter
      0.00            +1.2        1.18 ± 26%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read.readn.evsel__read_counter.read_counters
      0.00            +1.2        1.18 ± 16%  perf-profile.calltrace.cycles-pp.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +1.2        1.19 ± 22%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.00            +1.2        1.21 ± 10%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.22 ± 49%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.start_kernel
      0.00            +1.2        1.22 ± 24%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.22 ± 10%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.2        1.25 ± 17%  perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.26 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.26 ±  8%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64
      0.00            +1.3        1.26 ±  8%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.27 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.3        1.28 ± 22%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.decay_pcp_high.refresh_cpu_vm_stats.vmstat_update.process_one_work
      0.00            +1.3        1.29 ± 23%  perf-profile.calltrace.cycles-pp.read.readn.evsel__read_counter.read_counters.process_interval
      0.00            +1.3        1.30 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      0.00            +1.3        1.31 ± 24%  perf-profile.calltrace.cycles-pp.readn.evsel__read_counter.read_counters.process_interval.dispatch_events
      0.00            +1.3        1.31 ± 23%  perf-profile.calltrace.cycles-pp.decay_pcp_high.refresh_cpu_vm_stats.vmstat_update.process_one_work.worker_thread
      0.00            +1.3        1.32 ± 17%  perf-profile.calltrace.cycles-pp.tick_nohz_stop_tick.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.3        1.34 ± 13%  perf-profile.calltrace.cycles-pp.setlocale
      0.00            +1.3        1.35 ± 16%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.4        1.40 ± 49%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.4        1.42 ± 19%  perf-profile.calltrace.cycles-pp._raw_spin_lock.raw_spin_rq_lock_nested._nohz_idle_balance.do_idle.cpu_startup_entry
      0.00            +1.4        1.42 ± 19%  perf-profile.calltrace.cycles-pp.raw_spin_rq_lock_nested._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.4        1.44 ± 49%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.6        1.56 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.6        1.59 ±  7%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe._Fork
      0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      0.00            +1.7        1.66 ± 19%  perf-profile.calltrace.cycles-pp.mm_init.alloc_bprm.do_execveat_common.__x64_sys_execve.do_syscall_64
      0.00            +1.7        1.70 ±  4%  perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.8        1.79 ±  9%  perf-profile.calltrace.cycles-pp.__set_cpus_allowed_ptr.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64
      0.00            +1.8        1.81 ± 10%  perf-profile.calltrace.cycles-pp._Fork
      0.00            +1.9        1.85 ± 10%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +1.9        1.85 ± 20%  perf-profile.calltrace.cycles-pp.update_process_times.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.9        1.90 ± 37%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.readn
      0.00            +1.9        1.91 ± 13%  perf-profile.calltrace.cycles-pp.alloc_bprm.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.9        1.92 ± 24%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_domains
      0.00            +1.9        1.92 ± 10%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +2.1        2.14 ± 10%  perf-profile.calltrace.cycles-pp.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.1        2.14 ± 15%  perf-profile.calltrace.cycles-pp.evsel__read_counter.read_counters.process_interval.dispatch_events.cmd_stat
      0.00            +2.2        2.16 ± 21%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_domains.handle_softirqs
      0.00            +2.2        2.19 ± 13%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +2.2        2.20 ± 20%  perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_domains.handle_softirqs.__irq_exit_rcu
      0.00            +2.2        2.20 ± 13%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      0.00            +2.3        2.31 ± 11%  perf-profile.calltrace.cycles-pp.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.common_startup_64
      0.00            +2.4        2.36 ± 45%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
      0.00            +2.4        2.38 ± 13%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +2.4        2.39 ± 16%  perf-profile.calltrace.cycles-pp.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +2.4        2.43 ± 33%  perf-profile.calltrace.cycles-pp.refresh_cpu_vm_stats.vmstat_update.process_one_work.worker_thread.kthread
      0.00            +2.5        2.45 ± 32%  perf-profile.calltrace.cycles-pp.vmstat_update.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +2.5        2.48 ± 13%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.00            +2.5        2.50 ± 12%  perf-profile.calltrace.cycles-pp.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next
      0.00            +2.6        2.63 ± 18%  perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_domains.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +2.8        2.76 ± 16%  perf-profile.calltrace.cycles-pp.sched_balance_domains.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +2.8        2.85 ± 16%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +3.0        2.99 ±  6%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      0.00            +3.0        2.99 ± 14%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters
      0.00            +3.0        2.99 ± 14%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval
      0.00            +3.1        3.10 ±  5%  perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      0.00            +3.1        3.10 ±  6%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      0.00            +3.2        3.18 ±  8%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +3.3        3.28 ± 12%  perf-profile.calltrace.cycles-pp.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events
      0.00            +3.3        3.29 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +3.3        3.30 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      0.00            +3.4        3.38 ±  8%  perf-profile.calltrace.cycles-pp.read
      0.00            +3.8        3.80 ± 15%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +3.9        3.86 ±  5%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.9        3.92 ± 15%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +3.9        3.94 ± 11%  perf-profile.calltrace.cycles-pp.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events.cmd_stat
      0.00            +4.3        4.28 ± 15%  perf-profile.calltrace.cycles-pp._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      0.00            +4.7        4.67 ± 10%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +5.0        5.02 ± 34%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +5.0        5.03 ±  8%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.balance_fair
      0.00            +5.1        5.08 ±  8%  perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.balance_fair.__schedule
      0.00            +5.2        5.17 ± 12%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle
      0.00            +5.2        5.22 ±  8%  perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_newidle.balance_fair.__schedule.schedule
      0.00            +5.3        5.28 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +5.3        5.30 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +5.6        5.58 ±  8%  perf-profile.calltrace.cycles-pp.sched_balance_newidle.balance_fair.__schedule.schedule.smpboot_thread_fn
      0.00            +5.6        5.60 ±  8%  perf-profile.calltrace.cycles-pp.balance_fair.__schedule.schedule.smpboot_thread_fn.kthread
      0.00            +5.9        5.93 ± 31%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +6.1        6.10 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +6.2        6.17 ±  7%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +6.4        6.40 ±  3%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +6.4        6.41 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +6.4        6.41 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.00            +6.4        6.41 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      0.00            +6.4        6.42 ±  3%  perf-profile.calltrace.cycles-pp.execve
      0.00            +7.5        7.52 ±  7%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +7.9        7.86 ± 16%  perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat.run_builtin
      0.00            +8.0        8.02 ± 15%  perf-profile.calltrace.cycles-pp.dispatch_events.cmd_stat.run_builtin.main
      0.00            +8.0        8.02 ± 15%  perf-profile.calltrace.cycles-pp.process_interval.dispatch_events.cmd_stat.run_builtin.main
      0.00            +8.0        8.03 ± 15%  perf-profile.calltrace.cycles-pp.cmd_stat.run_builtin.main
      0.00            +8.1        8.09 ± 29%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.tmigr_handle_remote_up.tmigr_handle_remote.handle_softirqs
      0.00            +8.6        8.56 ± 14%  perf-profile.calltrace.cycles-pp.main
      0.00            +8.6        8.56 ± 14%  perf-profile.calltrace.cycles-pp.run_builtin.main
      0.00            +8.7        8.65 ± 28%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.tmigr_handle_remote_up.tmigr_handle_remote.handle_softirqs.__irq_exit_rcu
      0.00            +9.1        9.15 ± 26%  perf-profile.calltrace.cycles-pp.tmigr_handle_remote_up.tmigr_handle_remote.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +9.5        9.51 ± 26%  perf-profile.calltrace.cycles-pp.tmigr_handle_remote.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +10.3       10.30 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle_xstate.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00           +14.5       14.53 ± 19%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00           +14.6       14.65 ± 19%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00           +15.0       15.04 ± 14%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
      0.00           +15.0       15.04 ± 14%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      0.00           +15.0       15.04 ± 14%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      0.00           +20.4       20.41 ± 15%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00           +21.0       20.95 ± 15%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00           +34.3       34.34 ± 10%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00           +35.0       34.99 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00           +38.9       38.87 ±  9%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      0.00           +47.2       47.16 ±  7%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      0.00           +47.2       47.18 ±  7%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
      0.00           +47.2       47.18 ±  7%  perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
      0.00           +49.5       49.54 ±  5%  perf-profile.calltrace.cycles-pp.common_startup_64
     73.23 ±  2%     -73.2        0.00        perf-profile.children.cycles-pp.stress_sock
     94.70           -68.3       26.36 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     94.39           -68.1       26.30 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     57.83           -57.8        0.00        perf-profile.children.cycles-pp.__send
     49.62           -49.6        0.00        perf-profile.children.cycles-pp.__x64_sys_sendto
     49.58           -49.6        0.00        perf-profile.children.cycles-pp.__sys_sendto
     47.91           -47.9        0.00        perf-profile.children.cycles-pp.tcp_sendmsg
     44.49           -44.5        0.00        perf-profile.children.cycles-pp.tcp_sendmsg_locked
     40.87           -40.9        0.00        perf-profile.children.cycles-pp.recv
     37.26           -37.3        0.00        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     37.23           -37.2        0.00        perf-profile.children.cycles-pp.__sys_recvfrom
     36.30           -36.3        0.00        perf-profile.children.cycles-pp.sock_recvmsg
     35.96           -36.0        0.00        perf-profile.children.cycles-pp.inet_recvmsg
     35.91           -35.9        0.00        perf-profile.children.cycles-pp.tcp_recvmsg
     35.11           -35.1        0.00        perf-profile.children.cycles-pp.tcp_recvmsg_locked
     25.74           -25.7        0.00        perf-profile.children.cycles-pp.__tcp_transmit_skb
     23.73           -23.7        0.00        perf-profile.children.cycles-pp.__ip_queue_xmit
     22.26           -22.3        0.00        perf-profile.children.cycles-pp.ip_finish_output2
     21.67           -21.7        0.00        perf-profile.children.cycles-pp.__dev_queue_xmit
     19.22           -19.2        0.00        perf-profile.children.cycles-pp.__local_bh_enable_ip
     18.94           -18.9        0.00        perf-profile.children.cycles-pp.do_softirq
     18.86           -18.9        0.00        perf-profile.children.cycles-pp.tcp_write_xmit
     21.53           -18.6        2.98 ± 21%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     17.87           -17.8        0.04 ±152%  perf-profile.children.cycles-pp.net_rx_action
     17.34           -17.3        0.04 ±152%  perf-profile.children.cycles-pp.__napi_poll
     17.28           -17.3        0.00        perf-profile.children.cycles-pp.process_backlog
     16.98           -17.0        0.00        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     16.68           -16.7        0.00        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     15.23           -15.2        0.00        perf-profile.children.cycles-pp.ip_local_deliver_finish
     14.80           -14.8        0.00        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     14.57           -14.6        0.00        perf-profile.children.cycles-pp.tcp_v4_rcv
     13.69           -13.7        0.00        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     13.32           -13.3        0.00        perf-profile.children.cycles-pp.tcp_rcv_established
     13.16           -13.2        0.00        perf-profile.children.cycles-pp.sk_page_frag_refill
     13.13           -13.1        0.00        perf-profile.children.cycles-pp.skb_page_frag_refill
     12.66           -12.4        0.25 ± 20%  perf-profile.children.cycles-pp.get_page_from_freelist
     12.73           -12.3        0.40 ± 33%  perf-profile.children.cycles-pp.alloc_pages_mpol_noprof
     12.71           -12.3        0.38 ± 33%  perf-profile.children.cycles-pp.__alloc_pages_noprof
     12.22           -12.0        0.16 ± 41%  perf-profile.children.cycles-pp.rmqueue
     12.08           -12.0        0.11 ± 54%  perf-profile.children.cycles-pp.rmqueue_bulk
     12.10           -12.0        0.15 ± 43%  perf-profile.children.cycles-pp.__rmqueue_pcplist
     10.47           -10.5        0.00        perf-profile.children.cycles-pp.skb_release_data
      9.75            -9.6        0.11 ± 51%  perf-profile.children.cycles-pp.free_unref_page
      9.57            -9.4        0.13 ± 41%  perf-profile.children.cycles-pp.free_unref_page_commit
     20.82            -9.3       11.48 ± 19%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      8.92            -8.9        0.00        perf-profile.children.cycles-pp.skb_attempt_defer_free
      7.56            -7.6        0.00        perf-profile.children.cycles-pp.tcp_data_queue
      9.40            -7.1        2.32 ± 33%  perf-profile.children.cycles-pp.free_pcppages_bulk
      6.52            -6.5        0.00        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      6.47            -6.5        0.00        perf-profile.children.cycles-pp.__skb_datagram_iter
      6.45            -6.5        0.00        perf-profile.children.cycles-pp.sk_wait_data
      6.30 ±  2%      -6.3        0.00        perf-profile.children.cycles-pp.__sk_mem_schedule
      6.22            -6.2        0.00        perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      6.74            -5.9        0.83 ± 25%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      5.80            -5.8        0.00        perf-profile.children.cycles-pp.wait_woken
      5.53            -5.4        0.11 ± 67%  perf-profile.children.cycles-pp.schedule_timeout
      5.51            -5.3        0.20 ± 29%  perf-profile.children.cycles-pp._copy_to_iter
      4.38            -4.0        0.33 ± 32%  perf-profile.children.cycles-pp.__wake_up_sync_key
      4.25            -3.9        0.35 ± 27%  perf-profile.children.cycles-pp.__wake_up_common
      4.13            -3.0        1.10 ± 24%  perf-profile.children.cycles-pp.try_to_wake_up
      2.80 ±  8%      -2.7        0.12 ± 49%  perf-profile.children.cycles-pp.__mod_memcg_state
      2.35            -1.8        0.53 ± 24%  perf-profile.children.cycles-pp.ttwu_do_activate
      8.89            -1.5        7.34 ±  9%  perf-profile.children.cycles-pp.schedule
      1.67            -1.4        0.29 ± 24%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.38            -1.1        0.32 ± 46%  perf-profile.children.cycles-pp.__check_object_size
      1.32            -1.0        0.31 ± 34%  perf-profile.children.cycles-pp.update_curr
      1.07 ±  4%      -1.0        0.07 ± 83%  perf-profile.children.cycles-pp.try_charge_memcg
      1.82            -0.9        0.88 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.81            -0.9        0.88 ± 16%  perf-profile.children.cycles-pp.dequeue_task_fair
      1.20            -0.9        0.27 ± 48%  perf-profile.children.cycles-pp.switch_fpu_return
      1.12            -0.9        0.22 ± 23%  perf-profile.children.cycles-pp.__rseq_handle_notify_resume
      1.04            -0.9        0.18 ± 55%  perf-profile.children.cycles-pp.check_heap_object
      0.99            -0.9        0.14 ± 60%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      1.05            -0.8        0.22 ± 43%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.64            -0.8        0.83 ± 16%  perf-profile.children.cycles-pp.update_load_avg
      0.81            -0.8        0.04 ±103%  perf-profile.children.cycles-pp.__fdget
      0.92            -0.7        0.24 ± 43%  perf-profile.children.cycles-pp.read_tsc
      0.89 ±  2%      -0.6        0.25 ± 36%  perf-profile.children.cycles-pp.prepare_task_switch
      0.70            -0.5        0.18 ± 17%  perf-profile.children.cycles-pp.rseq_ip_fixup
      0.57            -0.5        0.10 ± 62%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.52 ±  2%      -0.5        0.05 ±104%  perf-profile.children.cycles-pp.put_prev_entity
      0.71            -0.4        0.26 ± 36%  perf-profile.children.cycles-pp.__switch_to
      0.66            -0.4        0.24 ± 22%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.47            -0.4        0.07 ±116%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.44            -0.4        0.06 ±101%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.61 ± 10%      -0.4        0.24 ± 40%  perf-profile.children.cycles-pp.select_task_rq
      0.42            -0.4        0.05 ±114%  perf-profile.children.cycles-pp.pick_eevdf
      0.54            -0.4        0.17 ± 36%  perf-profile.children.cycles-pp.__update_load_avg_se
      1.85            -0.3        1.51 ± 15%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.62 ±  2%      -0.3        0.32 ± 29%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.53            -0.3        0.23 ± 14%  perf-profile.children.cycles-pp.__switch_to_asm
      0.40            -0.3        0.14 ±  7%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.40            -0.3        0.14 ±  7%  perf-profile.children.cycles-pp.perf_mmap__push
      0.34 ±  4%      -0.3        0.08 ± 77%  perf-profile.children.cycles-pp.update_cfs_group
      0.53 ±  2%      -0.3        0.28 ± 30%  perf-profile.children.cycles-pp.sched_clock
      0.37            -0.2        0.14 ± 52%  perf-profile.children.cycles-pp.rseq_get_rseq_cs
      0.75 ± 12%      -0.2        0.53 ±  7%  perf-profile.children.cycles-pp.__cmd_record
      0.45            -0.2        0.24 ± 46%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.64            -0.2        0.44 ± 23%  perf-profile.children.cycles-pp.set_next_entity
      0.25 ±  2%      -0.2        0.06 ± 47%  perf-profile.children.cycles-pp.shmem_file_write_iter
      0.24            -0.2        0.06 ± 45%  perf-profile.children.cycles-pp.generic_perform_write
      0.89            -0.2        0.73 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.22 ±  2%      -0.1        0.07 ± 80%  perf-profile.children.cycles-pp.__put_user_8
      0.32            -0.1        0.18 ± 30%  perf-profile.children.cycles-pp.__get_user_8
      0.20 ±  2%      -0.1        0.07 ± 90%  perf-profile.children.cycles-pp.avg_vruntime
      0.11            -0.1        0.04 ±100%  perf-profile.children.cycles-pp.remove_wait_queue
      0.12 ±  4%      -0.1        0.07 ± 49%  perf-profile.children.cycles-pp.place_entity
      0.06 ±  8%      +0.1        0.12 ± 24%  perf-profile.children.cycles-pp.mm_cid_get
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.unlink_file_vma_batch_final
      0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.vm_area_dup
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp._copy_to_user
      0.00            +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.diskstats_show
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.00            +0.1        0.10 ± 26%  perf-profile.children.cycles-pp.kcpustat_cpu_fetch
      0.00            +0.1        0.10 ± 29%  perf-profile.children.cycles-pp.sched_balance_trigger
      0.00            +0.1        0.10 ± 42%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.__mbrtowc
      0.00            +0.1        0.11 ± 48%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      0.07 ±  6%      +0.1        0.19 ± 29%  perf-profile.children.cycles-pp.task_work_run
      0.09 ±  4%      +0.1        0.20 ± 23%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.__kmalloc_noprof
      0.00            +0.1        0.12 ± 33%  perf-profile.children.cycles-pp.__libc_fork
      0.00            +0.1        0.12 ± 32%  perf-profile.children.cycles-pp.devkmsg_read
      0.00            +0.1        0.12 ± 41%  perf-profile.children.cycles-pp.tmigr_quick_check
      0.00            +0.1        0.13 ± 41%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.1        0.13 ± 35%  perf-profile.children.cycles-pp.fput
      0.00            +0.1        0.13 ± 24%  perf-profile.children.cycles-pp.node_read_numastat
      0.00            +0.1        0.13 ± 40%  perf-profile.children.cycles-pp.perf_event_ctx_lock_nested
      0.00            +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.complete_signal
      0.00            +0.1        0.13 ± 49%  perf-profile.children.cycles-pp.free_unref_folios
      0.00            +0.1        0.13 ± 29%  perf-profile.children.cycles-pp.i40e_service_task
      0.40 ±  2%      +0.1        0.53 ±  7%  perf-profile.children.cycles-pp.cmd_record
      0.00            +0.1        0.13 ± 53%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown_vmflags
      0.00            +0.1        0.13 ± 42%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.00            +0.1        0.13 ± 23%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.00            +0.1        0.13 ± 34%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.00            +0.1        0.14 ± 20%  perf-profile.children.cycles-pp.update_group_capacity
      0.00            +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.delayed_vfree_work
      0.00            +0.1        0.14 ± 58%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.vfree
      0.00            +0.1        0.14 ± 30%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.00            +0.1        0.14 ± 44%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.06            +0.1        0.20 ± 30%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.00            +0.1        0.14 ± 37%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.14 ± 24%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.00            +0.1        0.14 ± 50%  perf-profile.children.cycles-pp.lru_add_drain
      0.00            +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.perf_stat_process_counter
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.acpi_ev_sci_xrupt_handler
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.acpi_irq
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.irq_thread_fn
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.00            +0.2        0.15 ± 12%  perf-profile.children.cycles-pp.do_notify_parent
      0.00            +0.2        0.15 ± 20%  perf-profile.children.cycles-pp.process_counters
      0.00            +0.2        0.15 ± 31%  perf-profile.children.cycles-pp.security_file_permission
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.acpi_ex_extract_from_field
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.acpi_ex_read_data_from_field
      0.00            +0.2        0.15 ± 33%  perf-profile.children.cycles-pp.acpi_ex_resolve_node_to_value
      0.00            +0.2        0.16 ± 53%  perf-profile.children.cycles-pp.__get_unmapped_area
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.2        0.16 ± 33%  perf-profile.children.cycles-pp.irq_thread
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp.vma_modify
      0.00            +0.2        0.16 ± 31%  perf-profile.children.cycles-pp.folio_remove_rmap_ptes
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.acpi_ev_address_space_dispatch
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.acpi_ex_access_region
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.acpi_ex_field_datum_io
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.acpi_ex_system_memory_space_handler
      0.00            +0.2        0.16 ± 32%  perf-profile.children.cycles-pp.acpi_ds_evaluate_name_path
      0.00            +0.2        0.16 ± 49%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.00            +0.2        0.16 ± 26%  perf-profile.children.cycles-pp.filemap_read
      0.00            +0.2        0.16 ± 51%  perf-profile.children.cycles-pp.mas_split
      0.00            +0.2        0.16 ± 26%  perf-profile.children.cycles-pp.open64
      0.00            +0.2        0.16 ± 12%  perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp._compound_head
      0.00            +0.2        0.16 ± 32%  perf-profile.children.cycles-pp.__pmd_alloc
      0.00            +0.2        0.16 ± 27%  perf-profile.children.cycles-pp.__x64_sys_close
      0.00            +0.2        0.16 ± 17%  perf-profile.children.cycles-pp.exit_notify
      0.00            +0.2        0.16 ± 22%  perf-profile.children.cycles-pp.seq_printf
      0.00            +0.2        0.17 ± 50%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.00            +0.2        0.17 ± 25%  perf-profile.children.cycles-pp.leave_mm
      0.00            +0.2        0.17 ± 62%  perf-profile.children.cycles-pp.strncpy_from_user
      0.00            +0.2        0.17 ± 35%  perf-profile.children.cycles-pp.wait_task_zombie
      0.00            +0.2        0.18 ± 21%  perf-profile.children.cycles-pp.__send_signal_locked
      0.00            +0.2        0.18 ± 25%  perf-profile.children.cycles-pp.cgroup_rstat_flush
      0.00            +0.2        0.18 ± 25%  perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
      0.00            +0.2        0.18 ± 25%  perf-profile.children.cycles-pp.flush_memcg_stats_dwork
      0.00            +0.2        0.18 ± 36%  perf-profile.children.cycles-pp.slab_show
      0.00            +0.2        0.18 ± 36%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +0.2        0.18 ± 46%  perf-profile.children.cycles-pp.rcu_report_qs_rnp
      0.00            +0.2        0.18 ± 42%  perf-profile.children.cycles-pp.__close
      0.00            +0.2        0.19 ± 40%  perf-profile.children.cycles-pp.__do_fault
      0.00            +0.2        0.19 ± 49%  perf-profile.children.cycles-pp.mas_wr_bnode
      0.00            +0.2        0.19 ± 38%  perf-profile.children.cycles-pp.perf_poll
      0.00            +0.2        0.19 ± 44%  perf-profile.children.cycles-pp.mas_find
      0.00            +0.2        0.19 ± 24%  perf-profile.children.cycles-pp.__mutex_lock
      0.00            +0.2        0.19 ± 31%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.00            +0.2        0.19 ± 26%  perf-profile.children.cycles-pp.alloc_anon_folio
      0.00            +0.2        0.20 ± 49%  perf-profile.children.cycles-pp.copy_p4d_range
      0.00            +0.2        0.20 ± 49%  perf-profile.children.cycles-pp.copy_page_range
      0.00            +0.2        0.20 ± 58%  perf-profile.children.cycles-pp.mod_objcg_state
      0.00            +0.2        0.20 ± 36%  perf-profile.children.cycles-pp.__do_wait
      0.00            +0.2        0.20 ± 49%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.05            +0.2        0.25 ± 30%  perf-profile.children.cycles-pp.get_jiffies_update
      0.00            +0.2        0.20 ± 56%  perf-profile.children.cycles-pp.copy_string_kernel
      0.00            +0.2        0.20 ± 30%  perf-profile.children.cycles-pp.shift_arg_pages
      0.00            +0.2        0.20 ± 28%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.00            +0.2        0.20 ± 40%  perf-profile.children.cycles-pp.mas_wr_node_store
      0.00            +0.2        0.20 ± 25%  perf-profile.children.cycles-pp.acpi_ds_exec_end_op
      0.00            +0.2        0.20 ± 42%  perf-profile.children.cycles-pp.mas_walk
      0.00            +0.2        0.20 ± 33%  perf-profile.children.cycles-pp.fstatat64
      0.00            +0.2        0.20 ± 20%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.2        0.20 ± 45%  perf-profile.children.cycles-pp.folio_batch_move_lru
      0.00            +0.2        0.21 ± 33%  perf-profile.children.cycles-pp.strnlen_user
      0.00            +0.2        0.21 ± 28%  perf-profile.children.cycles-pp.kmem_cache_alloc_lru_noprof
      0.00            +0.2        0.21 ± 47%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.2        0.22 ± 47%  perf-profile.children.cycles-pp.open_last_lookups
      0.00            +0.2        0.22 ± 46%  perf-profile.children.cycles-pp.dput
      0.00            +0.2        0.22 ± 25%  perf-profile.children.cycles-pp.sync_regs
      0.00            +0.2        0.22 ± 18%  perf-profile.children.cycles-pp.__memcpy
      0.00            +0.2        0.22 ± 34%  perf-profile.children.cycles-pp.lock_vma_under_rcu
      0.00            +0.2        0.22 ± 45%  perf-profile.children.cycles-pp.schedule_tail
      0.00            +0.2        0.22 ± 21%  perf-profile.children.cycles-pp.local_clock_noinstr
      0.00            +0.2        0.22 ± 49%  perf-profile.children.cycles-pp.step_into
      0.00            +0.2        0.22 ± 18%  perf-profile.children.cycles-pp.menu_reflect
      0.00            +0.2        0.23 ± 34%  perf-profile.children.cycles-pp.do_wait
      0.00            +0.2        0.23 ± 36%  perf-profile.children.cycles-pp.get_cpu_sleep_time_us
      0.00            +0.2        0.23 ± 35%  perf-profile.children.cycles-pp.note_gp_changes
      0.00            +0.2        0.23 ± 48%  perf-profile.children.cycles-pp.__get_user_pages
      0.00            +0.2        0.23 ± 41%  perf-profile.children.cycles-pp.hrtimer_try_to_cancel
      0.00            +0.2        0.23 ± 23%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.2        0.23 ± 45%  perf-profile.children.cycles-pp.getname_flags
      0.00            +0.2        0.23 ± 33%  perf-profile.children.cycles-pp.alloc_empty_file
      0.00            +0.2        0.24 ± 36%  perf-profile.children.cycles-pp.__d_alloc
      0.00            +0.2        0.24 ± 23%  perf-profile.children.cycles-pp.create_elf_tables
      0.00            +0.2        0.24 ± 52%  perf-profile.children.cycles-pp.get_user_pages_remote
      0.00            +0.2        0.24 ± 30%  perf-profile.children.cycles-pp.get_idle_time
      0.00            +0.2        0.24 ± 30%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.00            +0.2        0.24 ± 30%  perf-profile.children.cycles-pp.kernel_wait4
      0.00            +0.2        0.24 ± 47%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.2        0.24 ± 34%  perf-profile.children.cycles-pp.hrtimer_cancel
      0.00            +0.2        0.24 ± 37%  perf-profile.children.cycles-pp.mutex_unlock
      0.00            +0.2        0.24 ± 14%  perf-profile.children.cycles-pp.vsnprintf
      0.00            +0.2        0.24 ± 40%  perf-profile.children.cycles-pp.set_pte_range
      0.00            +0.2        0.25 ± 42%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.00            +0.2        0.25 ± 27%  perf-profile.children.cycles-pp.rw_verify_area
      0.00            +0.3        0.25 ± 21%  perf-profile.children.cycles-pp.do_poll
      0.00            +0.3        0.25 ± 39%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.00            +0.3        0.26 ± 29%  perf-profile.children.cycles-pp.do_open_execat
      0.00            +0.3        0.26 ± 30%  perf-profile.children.cycles-pp.dyntick_save_progress_counter
      0.00            +0.3        0.26 ± 39%  perf-profile.children.cycles-pp.mas_wr_store_entry
      0.00            +0.3        0.26 ± 17%  perf-profile.children.cycles-pp.down_write
      0.00            +0.3        0.26 ± 41%  perf-profile.children.cycles-pp.memchr_inv
      0.00            +0.3        0.26 ± 19%  perf-profile.children.cycles-pp.acpi_ps_parse_loop
      0.00            +0.3        0.26 ± 37%  perf-profile.children.cycles-pp.mprotect_fixup
      0.00            +0.3        0.26 ± 18%  perf-profile.children.cycles-pp.vma_complete
      0.00            +0.3        0.26 ± 30%  perf-profile.children.cycles-pp.unmap_region
      0.00            +0.3        0.27 ± 34%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.00            +0.3        0.27 ± 32%  perf-profile.children.cycles-pp.__vmalloc_node_range_noprof
      0.00            +0.3        0.27 ± 15%  perf-profile.children.cycles-pp.acpi_ps_parse_aml
      0.00            +0.3        0.27 ± 29%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.3        0.27 ± 28%  perf-profile.children.cycles-pp.perf_evlist__poll
      0.00            +0.3        0.27 ± 42%  perf-profile.children.cycles-pp.__slab_free
      0.00            +0.3        0.28 ± 23%  perf-profile.children.cycles-pp._IO_fwrite
      0.00            +0.3        0.28 ± 17%  perf-profile.children.cycles-pp.acpi_ev_asynch_execute_gpe_method
      0.00            +0.3        0.28 ± 17%  perf-profile.children.cycles-pp.acpi_ns_evaluate
      0.00            +0.3        0.28 ± 17%  perf-profile.children.cycles-pp.acpi_ps_execute_method
      0.00            +0.3        0.28 ± 37%  perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.00            +0.3        0.28 ± 18%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.3        0.28 ± 18%  perf-profile.children.cycles-pp.__tmigr_cpu_activate
      0.00            +0.3        0.28 ± 40%  perf-profile.children.cycles-pp.tmigr_inactive_up
      0.00            +0.3        0.29 ± 27%  perf-profile.children.cycles-pp.alloc_thread_stack_node
      0.00            +0.3        0.29 ± 37%  perf-profile.children.cycles-pp.dev_attr_show
      0.00            +0.3        0.29 ± 37%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.3        0.29 ± 49%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.00            +0.3        0.29 ± 28%  perf-profile.children.cycles-pp.all_vm_events
      0.00            +0.3        0.30 ± 18%  perf-profile.children.cycles-pp.setup_arg_pages
      0.00            +0.3        0.30 ± 30%  perf-profile.children.cycles-pp.__vm_munmap
      0.00            +0.3        0.30 ± 29%  perf-profile.children.cycles-pp.affinity__set
      0.00            +0.3        0.30 ± 48%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.3        0.30 ± 40%  perf-profile.children.cycles-pp.get_arg_page
      0.00            +0.3        0.30 ± 50%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.3        0.30 ± 14%  perf-profile.children.cycles-pp.acpi_os_execute_deferred
      0.00            +0.3        0.30 ± 34%  perf-profile.children.cycles-pp.do_open
      0.00            +0.3        0.30 ± 28%  perf-profile.children.cycles-pp.perf_evlist__poll_thread
      0.00            +0.3        0.30 ± 32%  perf-profile.children.cycles-pp.poll_idle
      0.00            +0.3        0.31 ± 28%  perf-profile.children.cycles-pp.vfs_statx
      0.00            +0.3        0.31 ± 14%  perf-profile.children.cycles-pp.wp_page_copy
      0.00            +0.3        0.31 ± 25%  perf-profile.children.cycles-pp.mm_init_cid
      0.00            +0.3        0.31 ± 19%  perf-profile.children.cycles-pp.should_we_balance
      0.00            +0.3        0.31 ± 24%  perf-profile.children.cycles-pp.wait4
      0.00            +0.3        0.31 ± 37%  perf-profile.children.cycles-pp.__pte_offset_map_lock
      0.00            +0.3        0.32 ± 40%  perf-profile.children.cycles-pp.__fdget_pos
      0.00            +0.3        0.32 ± 18%  perf-profile.children.cycles-pp.__x64_sys_poll
      0.00            +0.3        0.32 ± 18%  perf-profile.children.cycles-pp.do_sys_poll
      0.00            +0.3        0.32 ± 24%  perf-profile.children.cycles-pp.tmigr_cpu_activate
      0.00            +0.3        0.32 ± 33%  perf-profile.children.cycles-pp.wake_up_q
      0.00            +0.3        0.32 ± 20%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.3        0.32 ± 19%  perf-profile.children.cycles-pp.mutex_lock
      0.00            +0.3        0.32 ± 39%  perf-profile.children.cycles-pp.vma_prepare
      0.06 ±  9%      +0.3        0.38 ± 45%  perf-profile.children.cycles-pp.tmigr_requires_handle_remote
      0.00            +0.3        0.32 ±  8%  perf-profile.children.cycles-pp.__evlist__disable
      0.00            +0.3        0.32 ± 23%  perf-profile.children.cycles-pp.path_lookupat
      0.00            +0.3        0.33 ± 40%  perf-profile.children.cycles-pp.__hrtimer_start_range_ns
      0.00            +0.3        0.33 ± 28%  perf-profile.children.cycles-pp.tmigr_cpu_deactivate
      0.00            +0.3        0.33 ± 34%  perf-profile.children.cycles-pp.seq_read
      0.00            +0.3        0.33 ± 34%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.00            +0.3        0.33 ± 34%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.00            +0.3        0.33 ± 26%  perf-profile.children.cycles-pp.dup_task_struct
      0.00            +0.3        0.33 ± 21%  perf-profile.children.cycles-pp.filename_lookup
      0.00            +0.3        0.33 ± 25%  perf-profile.children.cycles-pp.vfs_fstatat
      0.00            +0.3        0.33 ± 36%  perf-profile.children.cycles-pp.lookup_fast
      0.00            +0.3        0.34 ± 21%  perf-profile.children.cycles-pp.__poll
      0.00            +0.3        0.34 ± 27%  perf-profile.children.cycles-pp.__perf_event_read_value
      0.00            +0.3        0.34 ± 36%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.3        0.35 ± 26%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.00            +0.4        0.35 ± 31%  perf-profile.children.cycles-pp.copy_strings
      0.00            +0.4        0.36 ± 17%  perf-profile.children.cycles-pp.cpu_util
      0.00            +0.4        0.36 ± 12%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.4        0.36 ± 12%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.4        0.36 ± 18%  perf-profile.children.cycles-pp.free_pgtables
      0.00            +0.4        0.37 ± 35%  perf-profile.children.cycles-pp.__tlb_batch_free_encoded_pages
      0.00            +0.4        0.37 ± 35%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.00            +0.4        0.37 ± 31%  perf-profile.children.cycles-pp.kick_pool
      0.00            +0.4        0.37 ± 33%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.00            +0.4        0.37 ± 27%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.00            +0.4        0.38 ± 36%  perf-profile.children.cycles-pp.folios_put_refs
      0.00            +0.4        0.38 ± 31%  perf-profile.children.cycles-pp.kfree
      0.00            +0.4        0.38 ± 25%  perf-profile.children.cycles-pp.tmigr_update_events
      0.00            +0.4        0.38 ± 28%  perf-profile.children.cycles-pp.cpu_stop_queue_work
      0.00            +0.4        0.38 ± 22%  perf-profile.children.cycles-pp.tmigr_handle_remote_cpu
      0.00            +0.4        0.39 ± 27%  perf-profile.children.cycles-pp.d_alloc
      0.00            +0.4        0.40 ± 52%  perf-profile.children.cycles-pp.__perf_read_group_add
      0.00            +0.4        0.40 ± 23%  perf-profile.children.cycles-pp.__do_sys_newfstatat
      0.00            +0.4        0.40 ± 17%  perf-profile.children.cycles-pp.load_elf_interp
      0.00            +0.4        0.40 ± 19%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.00            +0.4        0.41 ± 22%  perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      0.00            +0.4        0.41 ± 14%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.00            +0.4        0.42 ± 16%  perf-profile.children.cycles-pp.__x64_sys_exit
      0.00            +0.4        0.42 ± 21%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.42            +0.4        0.85 ± 22%  perf-profile.children.cycles-pp.__cond_resched
      0.00            +0.4        0.44 ± 31%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.00            +0.4        0.44 ± 27%  perf-profile.children.cycles-pp.dup_mmap
      0.31            +0.4        0.76 ± 12%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.5        0.46 ± 20%  perf-profile.children.cycles-pp.need_update
      0.00            +0.5        0.47 ± 26%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.00            +0.5        0.47 ± 20%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.07 ± 38%      +0.5        0.55 ± 16%  perf-profile.children.cycles-pp._find_next_and_bit
      0.00            +0.5        0.49 ±104%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.5        0.50 ± 20%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.00            +0.5        0.50 ± 25%  perf-profile.children.cycles-pp.sched_balance_softirq
      0.00            +0.5        0.50 ± 42%  perf-profile.children.cycles-pp.__perf_event_read
      0.00            +0.5        0.51 ± 22%  perf-profile.children.cycles-pp.rcu_do_batch
      0.00            +0.5        0.51 ± 28%  perf-profile.children.cycles-pp.quiet_vmstat
      0.10 ±  5%      +0.5        0.61 ± 36%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.5        0.52 ± 26%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.00            +0.5        0.53 ± 13%  perf-profile.children.cycles-pp.zap_present_ptes
      0.00            +0.5        0.53 ± 36%  perf-profile.children.cycles-pp.hrtimer_start_range_ns
      0.00            +0.5        0.54 ± 10%  perf-profile.children.cycles-pp.evlist__id2evsel
      0.00            +0.5        0.54 ± 21%  perf-profile.children.cycles-pp.__collapse_huge_page_copy
      0.00            +0.5        0.54 ± 20%  perf-profile.children.cycles-pp.do_anonymous_page
      0.00            +0.5        0.55 ± 28%  perf-profile.children.cycles-pp.__lookup_slow
      0.00            +0.5        0.55 ± 21%  perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
      0.00            +0.6        0.55 ± 24%  perf-profile.children.cycles-pp.__queue_work
      0.00            +0.6        0.56 ± 22%  perf-profile.children.cycles-pp.call_timer_fn
      0.00            +0.6        0.56 ± 22%  perf-profile.children.cycles-pp.__mmdrop
      0.00            +0.6        0.56 ± 14%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.6        0.56 ± 23%  perf-profile.children.cycles-pp.fold_vm_numa_events
      0.00            +0.6        0.57 ± 14%  perf-profile.children.cycles-pp.force_qs_rnp
      0.30            +0.6        0.86 ± 17%  perf-profile.children.cycles-pp.ksys_write
      0.28            +0.6        0.85 ± 18%  perf-profile.children.cycles-pp.vfs_write
      0.00            +0.6        0.58 ± 17%  perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +0.6        0.59 ± 37%  perf-profile.children.cycles-pp.generic_exec_single
      0.00            +0.6        0.60 ± 20%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.6        0.60 ± 17%  perf-profile.children.cycles-pp.vmstat_start
      0.00            +0.6        0.61 ±  7%  perf-profile.children.cycles-pp.__mmap
      0.00            +0.6        0.61 ± 13%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.00            +0.6        0.61 ± 39%  perf-profile.children.cycles-pp.smp_call_function_single
      0.00            +0.6        0.61 ± 25%  perf-profile.children.cycles-pp.__wait_for_common
      0.00            +0.6        0.61 ± 19%  perf-profile.children.cycles-pp.collapse_huge_page
      0.00            +0.6        0.61 ± 37%  perf-profile.children.cycles-pp.perf_event_read
      0.00            +0.6        0.62 ± 17%  perf-profile.children.cycles-pp.__run_timers
      0.00            +0.6        0.62 ± 19%  perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
      0.00            +0.6        0.62 ± 19%  perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
      0.00            +0.6        0.62 ± 10%  perf-profile.children.cycles-pp.zap_pte_range
      0.00            +0.6        0.63 ± 19%  perf-profile.children.cycles-pp.khugepaged
      0.00            +0.6        0.64 ±  9%  perf-profile.children.cycles-pp.zap_pmd_range
      0.32 ±  2%      +0.6        0.96 ± 18%  perf-profile.children.cycles-pp.write
      0.00            +0.7        0.65 ± 17%  perf-profile.children.cycles-pp.perf_event_mmap
      0.00            +0.7        0.65 ± 17%  perf-profile.children.cycles-pp.perf_event_mmap_event
      0.00            +0.7        0.67 ±  9%  perf-profile.children.cycles-pp.unmap_page_range
      0.00            +0.7        0.67 ± 15%  perf-profile.children.cycles-pp.__do_set_cpus_allowed
      0.00            +0.7        0.69 ± 12%  perf-profile.children.cycles-pp.rcu_gp_fqs_loop
      0.00            +0.7        0.71 ± 20%  perf-profile.children.cycles-pp.__split_vma
      0.00            +0.7        0.71 ± 15%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.00            +0.7        0.72 ± 13%  perf-profile.children.cycles-pp.__open64_nocancel
      0.00            +0.7        0.72 ± 14%  perf-profile.children.cycles-pp.sched_exec
      0.00            +0.7        0.72 ± 14%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.7        0.72 ± 10%  perf-profile.children.cycles-pp.unmap_vmas
      0.00            +0.7        0.72 ±  9%  perf-profile.children.cycles-pp.show_stat
      0.00            +0.7        0.75 ± 19%  perf-profile.children.cycles-pp.pipe_write
      0.00            +0.8        0.76 ±  8%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.00            +0.8        0.77 ± 19%  perf-profile.children.cycles-pp.__set_cpus_allowed_ptr_locked
      0.00            +0.8        0.77 ± 13%  perf-profile.children.cycles-pp.__x64_sys_vfork
      0.00            +0.8        0.78 ± 15%  perf-profile.children.cycles-pp.pipe_read
      0.00            +0.8        0.78 ± 33%  perf-profile.children.cycles-pp.__percpu_counter_init_many
      0.00            +0.8        0.79 ± 14%  perf-profile.children.cycles-pp.wake_up_new_task
      0.00            +0.8        0.83 ± 12%  perf-profile.children.cycles-pp.__get_next_timer_interrupt
      0.00            +0.8        0.83 ± 19%  perf-profile.children.cycles-pp.rcu_core
      0.14 ±  3%      +0.8        0.97 ± 14%  perf-profile.children.cycles-pp.sched_tick
      0.00            +0.9        0.86 ± 16%  perf-profile.children.cycles-pp.walk_component
      0.00            +0.9        0.89 ± 67%  perf-profile.children.cycles-pp.tick_do_update_jiffies64
      0.00            +0.9        0.94 ± 56%  perf-profile.children.cycles-pp.drain_zone_pages
      0.00            +1.0        0.96 ± 16%  perf-profile.children.cycles-pp.dup_mm
      0.00            +1.0        0.97 ± 17%  perf-profile.children.cycles-pp.exec_mmap
      0.00            +1.0        0.98 ± 13%  perf-profile.children.cycles-pp.__vfork
      0.00            +1.0        0.98 ± 14%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      0.00            +1.0        0.98 ± 11%  perf-profile.children.cycles-pp.tick_nohz_restart_sched_tick
      0.00            +1.0        0.99 ± 15%  perf-profile.children.cycles-pp.next_uptodate_folio
      0.00            +1.0        1.00 ± 18%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +1.0        1.02 ± 15%  perf-profile.children.cycles-pp.move_queued_task
      0.53 ± 12%      +1.0        1.56 ±  6%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +1.0        1.05 ± 17%  perf-profile.children.cycles-pp.affine_move_task
      0.00            +1.0        1.05 ± 14%  perf-profile.children.cycles-pp.elf_load
      0.00            +1.1        1.06 ± 23%  perf-profile.children.cycles-pp.link_path_walk
      0.00            +1.1        1.08 ± 16%  perf-profile.children.cycles-pp.do_vmi_munmap
      0.00            +1.1        1.10 ± 12%  perf-profile.children.cycles-pp.begin_new_exec
      0.00            +1.1        1.11 ± 36%  perf-profile.children.cycles-pp.perf_evsel__read
      0.00            +1.1        1.15 ± 11%  perf-profile.children.cycles-pp.exit_mm
      0.00            +1.2        1.15 ± 16%  perf-profile.children.cycles-pp.migration_cpu_stop
      0.00            +1.2        1.16 ± 14%  perf-profile.children.cycles-pp.intel_idle
      0.23 ±  2%      +1.2        1.40 ± 16%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.00            +1.2        1.18 ± 16%  perf-profile.children.cycles-pp.cpu_stopper_thread
      0.00            +1.3        1.29 ± 17%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +1.3        1.29 ±  7%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.00            +1.3        1.29 ±  7%  perf-profile.children.cycles-pp.do_group_exit
      0.00            +1.3        1.31 ± 10%  perf-profile.children.cycles-pp.sched_balance_update_blocked_averages
      0.00            +1.3        1.31 ± 23%  perf-profile.children.cycles-pp.decay_pcp_high
      0.00            +1.3        1.32 ± 25%  perf-profile.children.cycles-pp.__memset
      0.00            +1.3        1.34 ± 13%  perf-profile.children.cycles-pp.setlocale
      0.00            +1.4        1.35 ± 17%  perf-profile.children.cycles-pp.tick_nohz_stop_tick
      0.00            +1.4        1.36 ± 28%  perf-profile.children.cycles-pp.perf_read
      0.00            +1.4        1.38 ± 16%  perf-profile.children.cycles-pp.tick_nohz_idle_stop_tick
      0.00            +1.4        1.39 ±  6%  perf-profile.children.cycles-pp.update_sg_wakeup_stats
      0.00            +1.4        1.40 ± 13%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.00            +1.4        1.42 ±  7%  perf-profile.children.cycles-pp.sched_balance_find_dst_group
      0.00            +1.4        1.44 ±  7%  perf-profile.children.cycles-pp.sched_balance_find_dst_cpu
      0.00            +1.5        1.45 ± 48%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.5        1.48 ± 48%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +1.5        1.48 ± 10%  perf-profile.children.cycles-pp.filemap_map_pages
      0.00            +1.5        1.48 ± 18%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +1.6        1.55 ± 14%  perf-profile.children.cycles-pp.copy_process
      0.00            +1.6        1.55 ± 28%  perf-profile.children.cycles-pp.pcpu_alloc_noprof
      0.00            +1.6        1.55 ± 95%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.00            +1.6        1.55 ± 95%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +1.6        1.55 ± 95%  perf-profile.children.cycles-pp.drm_fbdev_shmem_helper_fb_dirty
      0.00            +1.6        1.58 ± 14%  perf-profile.children.cycles-pp.exit_mmap
      0.00            +1.6        1.58 ± 13%  perf-profile.children.cycles-pp.mmput
      0.00            +1.6        1.59 ± 12%  perf-profile.children.cycles-pp.__do_sys_clone
      0.00            +1.6        1.60 ± 10%  perf-profile.children.cycles-pp.do_read_fault
      0.00            +1.7        1.71 ±  4%  perf-profile.children.cycles-pp.do_exit
      0.00            +1.8        1.78 ±  8%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.12 ±  3%      +1.8        1.92 ±  8%  perf-profile.children.cycles-pp.x64_sys_call
      0.00            +1.8        1.82 ±  9%  perf-profile.children.cycles-pp._Fork
      0.26 ±  4%      +1.8        2.09 ± 18%  perf-profile.children.cycles-pp.update_process_times
      0.00            +1.8        1.85 ± 18%  perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
      0.00            +1.9        1.86 ±  9%  perf-profile.children.cycles-pp.path_openat
      0.00            +1.9        1.86 ±  9%  perf-profile.children.cycles-pp.do_fault
      0.00            +1.9        1.87 ±  9%  perf-profile.children.cycles-pp.__set_cpus_allowed_ptr
      0.00            +1.9        1.89 ±  6%  perf-profile.children.cycles-pp.__x64_sys_openat
      0.00            +1.9        1.89 ±  9%  perf-profile.children.cycles-pp.do_filp_open
      0.00            +1.9        1.91 ± 13%  perf-profile.children.cycles-pp.alloc_bprm
      0.00            +2.0        1.96 ±  7%  perf-profile.children.cycles-pp.do_sys_openat2
      0.00            +2.1        2.13 ± 24%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +2.1        2.15 ± 15%  perf-profile.children.cycles-pp.evsel__read_counter
      0.00            +2.2        2.16 ±  8%  perf-profile.children.cycles-pp.seq_read_iter
      0.00            +2.2        2.18 ± 18%  perf-profile.children.cycles-pp.mm_init
      0.00            +2.2        2.24 ±  7%  perf-profile.children.cycles-pp.mmap_region
      0.00            +2.2        2.24 ±  9%  perf-profile.children.cycles-pp.__sched_setaffinity
      0.00            +2.3        2.31 ± 27%  perf-profile.children.cycles-pp.readn
      0.00            +2.4        2.36 ±  8%  perf-profile.children.cycles-pp.kernel_clone
      0.00            +2.4        2.36 ± 45%  perf-profile.children.cycles-pp.rest_init
      0.00            +2.4        2.36 ± 45%  perf-profile.children.cycles-pp.start_kernel
      0.00            +2.4        2.36 ± 45%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.00            +2.4        2.36 ± 45%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.28 ±  4%      +2.4        2.69 ± 14%  perf-profile.children.cycles-pp.tick_nohz_handler
      0.00            +2.4        2.42 ± 12%  perf-profile.children.cycles-pp.menu_select
      0.00            +2.4        2.43 ±  5%  perf-profile.children.cycles-pp.do_mmap
      0.00            +2.5        2.45 ± 32%  perf-profile.children.cycles-pp.vmstat_update
      0.00            +2.6        2.57 ± 31%  perf-profile.children.cycles-pp.refresh_cpu_vm_stats
      0.00            +2.6        2.62 ± 11%  perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
      0.00            +2.7        2.67 ±  4%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.29 ±  4%      +2.9        3.17 ± 13%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +3.0        2.99 ±  6%  perf-profile.children.cycles-pp.load_elf_binary
      0.00            +3.1        3.10 ±  5%  perf-profile.children.cycles-pp.search_binary_handler
      0.00            +3.1        3.10 ±  6%  perf-profile.children.cycles-pp.exec_binprm
      0.00            +3.1        3.12 ±  8%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +3.2        3.23 ±  8%  perf-profile.children.cycles-pp.handle_mm_fault
      0.00            +3.4        3.36 ±  8%  perf-profile.children.cycles-pp.sched_balance_domains
      0.00            +3.5        3.46 ±  9%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.00            +3.5        3.46 ±  9%  perf-profile.children.cycles-pp.exc_page_fault
      0.41 ±  2%      +3.8        4.17 ± 13%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.42 ±  2%      +3.9        4.28 ± 12%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.46 ±  4%      +3.9        4.33 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +3.9        3.91 ±  4%  perf-profile.children.cycles-pp.bprm_execve
      0.00            +4.0        4.00 ±  8%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.00            +4.1        4.14 ± 11%  perf-profile.children.cycles-pp.evlist_cpu_iterator__next
      0.00            +4.9        4.89 ±  6%  perf-profile.children.cycles-pp.vfs_read
      0.01 ±223%      +5.0        5.02 ± 34%  perf-profile.children.cycles-pp.process_one_work
      0.00            +5.2        5.24 ±  7%  perf-profile.children.cycles-pp.ksys_read
      0.00            +5.4        5.40 ± 19%  perf-profile.children.cycles-pp._nohz_idle_balance
      0.00            +5.6        5.61 ±  8%  perf-profile.children.cycles-pp.balance_fair
      0.00            +5.7        5.68 ±  6%  perf-profile.children.cycles-pp.read
      0.01 ±223%      +5.9        5.93 ± 31%  perf-profile.children.cycles-pp.worker_thread
      0.00            +6.0        5.96 ± 11%  perf-profile.children.cycles-pp.sched_setaffinity
      0.00            +6.4        6.42 ±  3%  perf-profile.children.cycles-pp.execve
      0.00            +6.4        6.44 ±  3%  perf-profile.children.cycles-pp.do_execveat_common
      0.00            +6.5        6.46 ±  3%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.00            +6.5        6.54 ±  9%  perf-profile.children.cycles-pp.sched_balance_newidle
      0.00            +7.5        7.52 ±  7%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.00            +7.9        7.86 ± 16%  perf-profile.children.cycles-pp.read_counters
      0.00            +7.9        7.90 ±  9%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.00            +8.0        8.02 ± 15%  perf-profile.children.cycles-pp.dispatch_events
      0.00            +8.0        8.02 ± 15%  perf-profile.children.cycles-pp.process_interval
      0.00            +8.0        8.03 ± 15%  perf-profile.children.cycles-pp.cmd_stat
      0.40            +8.2        8.56 ± 14%  perf-profile.children.cycles-pp.main
      0.40            +8.2        8.56 ± 14%  perf-profile.children.cycles-pp.run_builtin
      0.00            +8.4        8.38 ±  8%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +8.5        8.46 ±  8%  perf-profile.children.cycles-pp.sched_balance_find_src_group
      0.33            +8.6        8.96 ± 26%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +9.2        9.17 ± 26%  perf-profile.children.cycles-pp.tmigr_handle_remote_up
      0.00            +9.2        9.20 ±  8%  perf-profile.children.cycles-pp.sched_balance_rq
      0.00            +9.5        9.53 ± 25%  perf-profile.children.cycles-pp.tmigr_handle_remote
      0.00           +10.5       10.45 ± 10%  perf-profile.children.cycles-pp.intel_idle_xstate
      0.01 ±223%     +15.0       15.04 ± 14%  perf-profile.children.cycles-pp.kthread
      0.01 ±223%     +15.3       15.28 ± 14%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±223%     +15.3       15.33 ± 14%  perf-profile.children.cycles-pp.ret_from_fork_asm
      0.00           +15.3       15.34 ± 18%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.45 ±  2%     +20.6       21.08 ± 15%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.50 ±  2%     +21.1       21.62 ± 14%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.01 ±223%     +35.4       35.45 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.01 ±223%     +35.5       35.48 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter
      0.02 ±146%     +40.1       40.10 ±  8%  perf-profile.children.cycles-pp.cpuidle_idle_call
      0.03 ±144%     +47.1       47.18 ±  7%  perf-profile.children.cycles-pp.start_secondary
      0.03 ±148%     +49.5       49.53 ±  5%  perf-profile.children.cycles-pp.do_idle
      0.03 ±145%     +49.5       49.54 ±  5%  perf-profile.children.cycles-pp.common_startup_64
      0.03 ±145%     +49.5       49.54 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
     20.82            -9.3       11.47 ± 19%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      5.48            -5.3        0.14 ± 49%  perf-profile.self.cycles-pp._copy_to_iter
      2.56 ±  9%      -2.5        0.06 ± 73%  perf-profile.self.cycles-pp.__mod_memcg_state
      1.64            -1.4        0.28 ± 22%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      1.15            -0.9        0.29 ± 37%  perf-profile.self.cycles-pp.__schedule
      0.98            -0.8        0.14 ± 60%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.79            -0.7        0.04 ±103%  perf-profile.self.cycles-pp.__fdget
      0.88            -0.6        0.24 ± 43%  perf-profile.self.cycles-pp.read_tsc
      0.56            -0.5        0.09 ± 53%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.56            -0.5        0.10 ± 62%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.54            -0.4        0.09 ± 51%  perf-profile.self.cycles-pp.check_heap_object
      0.68            -0.4        0.26 ± 34%  perf-profile.self.cycles-pp.__switch_to
      0.60            -0.4        0.20 ± 49%  perf-profile.self.cycles-pp.kmem_cache_free
      0.41 ±  2%      -0.4        0.06 ±101%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.49            -0.3        0.16 ± 33%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.41            -0.3        0.08 ± 82%  perf-profile.self.cycles-pp.prepare_task_switch
      0.44            -0.3        0.13 ± 48%  perf-profile.self.cycles-pp.update_curr
      0.36            -0.3        0.05 ± 75%  perf-profile.self.cycles-pp.do_syscall_64
      0.52            -0.3        0.23 ± 14%  perf-profile.self.cycles-pp.__switch_to_asm
      0.32 ±  2%      -0.3        0.05 ±103%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.32            -0.3        0.06 ±108%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.32 ±  3%      -0.2        0.08 ± 77%  perf-profile.self.cycles-pp.update_cfs_group
      0.28            -0.2        0.05 ±114%  perf-profile.self.cycles-pp.pick_eevdf
      0.29            -0.2        0.08 ± 81%  perf-profile.self.cycles-pp.handle_softirqs
      0.42            -0.2        0.22 ± 53%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.67            -0.2        0.50 ± 15%  perf-profile.self.cycles-pp.update_load_avg
      0.26            -0.2        0.10 ± 50%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.21            -0.1        0.07 ± 73%  perf-profile.self.cycles-pp.schedule
      0.30            -0.1        0.18 ± 30%  perf-profile.self.cycles-pp.__get_user_8
      0.24            -0.1        0.12 ± 62%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.24 ±  5%      -0.1        0.12 ± 57%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.05            +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.mm_cid_get
      0.00            +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.09 ± 29%  perf-profile.self.cycles-pp.sched_balance_trigger
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.set_pte_range
      0.00            +0.1        0.10 ± 26%  perf-profile.self.cycles-pp.kcpustat_cpu_fetch
      0.00            +0.1        0.10 ± 42%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.1        0.11 ± 21%  perf-profile.self.cycles-pp.__mbrtowc
      0.00            +0.1        0.12 ± 29%  perf-profile.self.cycles-pp.__kmalloc_noprof
      0.00            +0.1        0.12 ± 41%  perf-profile.self.cycles-pp.tmigr_quick_check
      0.08            +0.1        0.20 ± 23%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +0.1        0.12 ± 54%  perf-profile.self.cycles-pp.mmap_region
      0.00            +0.1        0.13 ± 37%  perf-profile.self.cycles-pp.read
      0.00            +0.1        0.13 ± 42%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.1        0.13 ± 34%  perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.00            +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.update_group_capacity
      0.00            +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.timekeeping_advance
      0.00            +0.1        0.14 ± 36%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.1        0.14 ± 24%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.00            +0.1        0.15 ± 56%  perf-profile.self.cycles-pp.sched_balance_update_blocked_averages
      0.00            +0.2        0.15 ± 18%  perf-profile.self.cycles-pp.tick_nohz_idle_got_tick
      0.00            +0.2        0.15 ± 36%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.00            +0.2        0.15 ± 14%  perf-profile.self.cycles-pp.pipe_read
      0.00            +0.2        0.15 ± 31%  perf-profile.self.cycles-pp.acpi_ex_system_memory_space_handler
      0.00            +0.2        0.16 ± 42%  perf-profile.self.cycles-pp._compound_head
      0.00            +0.2        0.16 ± 34%  perf-profile.self.cycles-pp.vfs_read
      0.00            +0.2        0.16 ± 35%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.2        0.16 ± 46%  perf-profile.self.cycles-pp.need_update
      0.00            +0.2        0.17 ± 46%  perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      0.00            +0.2        0.18 ± 30%  perf-profile.self.cycles-pp.affinity__set
      0.00            +0.2        0.18 ± 36%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.2        0.18 ± 45%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.2        0.18 ± 35%  perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.00            +0.2        0.19 ± 40%  perf-profile.self.cycles-pp.evlist_cpu_iterator__next
      0.00            +0.2        0.19 ± 39%  perf-profile.self.cycles-pp.zap_present_ptes
      0.00            +0.2        0.20 ± 49%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.05            +0.2        0.25 ± 30%  perf-profile.self.cycles-pp.get_jiffies_update
      0.00            +0.2        0.20 ± 28%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.00            +0.2        0.20 ± 42%  perf-profile.self.cycles-pp.mas_walk
      0.00            +0.2        0.20 ± 34%  perf-profile.self.cycles-pp.strnlen_user
      0.00            +0.2        0.20 ± 20%  perf-profile.self.cycles-pp.__memcpy
      0.00            +0.2        0.21 ± 47%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.2        0.22 ± 25%  perf-profile.self.cycles-pp.sync_regs
      0.00            +0.2        0.22 ± 49%  perf-profile.self.cycles-pp.refresh_cpu_vm_stats
      0.00            +0.2        0.22 ± 29%  perf-profile.self.cycles-pp.__tmigr_cpu_activate
      0.00            +0.2        0.22 ± 18%  perf-profile.self.cycles-pp.down_write
      0.00            +0.2        0.22 ± 13%  perf-profile.self.cycles-pp._find_next_bit
      0.00            +0.2        0.23 ± 38%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.2        0.23 ± 36%  perf-profile.self.cycles-pp.mutex_unlock
      0.00            +0.3        0.25 ± 13%  perf-profile.self.cycles-pp.filemap_map_pages
      0.00            +0.3        0.26 ± 30%  perf-profile.self.cycles-pp.dyntick_save_progress_counter
      0.00            +0.3        0.26 ± 41%  perf-profile.self.cycles-pp.memchr_inv
      0.00            +0.3        0.27 ± 26%  perf-profile.self.cycles-pp._IO_fwrite
      0.00            +0.3        0.27 ± 42%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.3        0.28 ± 34%  perf-profile.self.cycles-pp.sched_balance_newidle
      0.00            +0.3        0.28 ± 35%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.00            +0.3        0.28 ± 28%  perf-profile.self.cycles-pp.tmigr_handle_remote
      0.00            +0.3        0.28 ± 32%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +0.3        0.28 ± 29%  perf-profile.self.cycles-pp.evsel__read_counter
      0.00            +0.3        0.28 ± 34%  perf-profile.self.cycles-pp.kfree
      0.00            +0.3        0.29 ± 28%  perf-profile.self.cycles-pp.all_vm_events
      0.00            +0.3        0.30 ± 28%  perf-profile.self.cycles-pp.mm_init_cid
      0.00            +0.3        0.30 ± 26%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.3        0.30 ± 33%  perf-profile.self.cycles-pp.poll_idle
      0.00            +0.3        0.30 ± 26%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.3        0.30 ± 18%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.3        0.31 ± 22%  perf-profile.self.cycles-pp.cpu_util
      0.00            +0.3        0.31 ± 39%  perf-profile.self.cycles-pp.__fdget_pos
      0.00            +0.3        0.32 ± 22%  perf-profile.self.cycles-pp.sched_balance_rq
      0.00            +0.3        0.32 ± 19%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.00            +0.3        0.34 ± 37%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.3        0.34 ± 37%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.00            +0.3        0.34 ± 24%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.70            +0.4        1.09 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.07 ± 35%      +0.4        0.50 ± 21%  perf-profile.self.cycles-pp._find_next_and_bit
      0.31            +0.5        0.76 ± 21%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.5        0.48 ± 28%  perf-profile.self.cycles-pp.mm_init
      0.00            +0.5        0.50 ± 17%  perf-profile.self.cycles-pp.evlist__id2evsel
      0.00            +0.5        0.52 ± 24%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.00            +0.5        0.53 ± 27%  perf-profile.self.cycles-pp.read_counters
      0.00            +0.6        0.56 ± 23%  perf-profile.self.cycles-pp.fold_vm_numa_events
      0.00            +0.6        0.58 ± 17%  perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +0.6        0.60 ± 20%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.7        0.75 ± 27%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.9        0.90 ± 11%  perf-profile.self.cycles-pp.next_uptodate_folio
      0.00            +0.9        0.91 ± 15%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +1.0        0.96 ± 23%  perf-profile.self.cycles-pp._nohz_idle_balance
      0.14 ±  3%      +1.1        1.26 ± 16%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.00            +1.2        1.16 ± 14%  perf-profile.self.cycles-pp.intel_idle
      0.00            +1.3        1.29 ±  9%  perf-profile.self.cycles-pp.update_sg_wakeup_stats
      0.00            +1.3        1.30 ± 25%  perf-profile.self.cycles-pp.__memset
      0.00            +1.4        1.44 ± 17%  perf-profile.self.cycles-pp.idle_cpu
      0.45 ±  4%      +2.4        2.90 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +6.2        6.20 ±  5%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.00           +10.4       10.43 ± 10%  perf-profile.self.cycles-pp.intel_idle_xstate




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
sunyiqi Aug. 27, 2024, 8:59 a.m. UTC | #7 
On Thu, 15 Aug 2024 19:14:12 Jason Xing <kerneljasonxing@gmail.com> wrote:
> On Thu, Aug 15, 2024 at 5:50 PM sunyiqi <sunyiqixm@gmail.com> wrote:
> >
> > When investigating the kcm socket UAF which is also found by syzbot,
> > I found that the root cause of this problem is actually in
> > sk_wait_event.
> >
> > In sk_wait_event, sk is released and relocked and called by
> > sk_stream_wait_memory. Protocols like tcp, kcm, etc., called it in some
> > ops function like *sendmsg which will lock the sk at the beginning.
> > But sk_stream_wait_memory releases sk unexpectedly and destroy
> > the thread safety. Finally it causes the kcm sk UAF.
> >
> > If at the time when a thread(thread A) calls sk_stream_wait_memory
> > and the other thread(thread B) is waiting for lock in lock_sock,
> > thread B will successfully get the sk lock as thread A release sk lock
> > in sk_wait_event.
> >
> > The thread B may change the sk which is not thread A expecting.
> >
> > As a result, it will lead kernel to the unexpected behavior. Just like
> > the kcm sk UAF, which is actually cause by sk_wait_event in
> > sk_stream_wait_memory.
> >
> > Previous commit d9dc8b0f8b4e ("net: fix sleeping for sk_wait_event()")
> > in 2016 seems do not solved this problem. Is it necessary to release
> > sock in sk_wait_event? Or just delete it to make the protocol ops
> > thread-secure.
> >
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Link: https://syzkaller.appspot.com/bug?extid=b72d86aa5df17ce74c60
> > Signed-off-by: sunyiqi <sunyiqixm@gmail.com>
> > ---
> >  include/net/sock.h | 2 --
> >  1 file changed, 2 deletions(-)
> >
> > diff --git a/include/net/sock.h b/include/net/sock.h
> > index cce23ac4d514..08d3b204b019 100644
> > --- a/include/net/sock.h
> > +++ b/include/net/sock.h
> > @@ -1145,7 +1145,6 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
> >
> >  #define sk_wait_event(__sk, __timeo, __condition, __wait)              \
> >         ({      int __rc, __dis = __sk->sk_disconnects;                 \
> > -               release_sock(__sk);                                     \
> >                 __rc = __condition;                                     \
> >                 if (!__rc) {                                            \
> >                         *(__timeo) = wait_woken(__wait,                 \
> > @@ -1153,7 +1152,6 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
> >                                                 *(__timeo));            \
> >                 }                                                       \
> >                 sched_annotate_sleep();                                 \
> > -               lock_sock(__sk);                                        \
> 
> Are you sure that you want the socket lock to be held possibly for a
> really long time even if it has to wait for the available memory,
> which means, during this period, other threads trying to access the
> lock will be blocked?
> 
> Thanks,
> Jason

You're correct. Sorry for my poor knowledge of kernel. But I think kcm socket
UAF vulnerability is not the last one which use kernel API correctly(?) but
result in vulnerability.

Though the kcm developer can fix(and already fixed) this issue by adding more
lock in kcm sock, but, just the same question, what's the root cause of this
issue? Is kcm socket code or sk_wait_event release the sock?

Thanks,
Yiqi Sun
diff mbox series

Patch

diff --git a/include/net/sock.h b/include/net/sock.h
index cce23ac4d514..08d3b204b019 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1145,7 +1145,6 @@  static inline void sock_rps_reset_rxhash(struct sock *sk)
 
 #define sk_wait_event(__sk, __timeo, __condition, __wait)		\
 	({	int __rc, __dis = __sk->sk_disconnects;			\
-		release_sock(__sk);					\
 		__rc = __condition;					\
 		if (!__rc) {						\
 			*(__timeo) = wait_woken(__wait,			\
@@ -1153,7 +1152,6 @@  static inline void sock_rps_reset_rxhash(struct sock *sk)
 						*(__timeo));		\
 		}							\
 		sched_annotate_sleep();					\
-		lock_sock(__sk);					\
 		__rc = __dis == __sk->sk_disconnects ? __condition : -EPIPE; \
 		__rc;							\
 	})