[RFC,net-next,v3,0/8] netconsole: Add support for CPU population

Message

Breno Leitao Jan. 24, 2025, 3:16 p.m. UTC

The current implementation of netconsole sends all log messages in
parallel, which can lead to an intermixed and interleaved output on the
receiving side. This makes it challenging to demultiplex the messages
and attribute them to their originating CPUs.

As a result, users and developers often struggle to effectively analyze
and debug the parallel log output received through netconsole.

Example of a message got from produciton hosts:

	------------[ cut here ]------------
	------------[ cut here ]------------
	refcount_t: saturated; leaking memory.
	WARNING: CPU: 2 PID: 1613668 at lib/refcount.c:22 refcount_warn_saturate+0x5e/0xe0
	refcount_t: addition on 0; use-after-free.
	WARNING: CPU: 26 PID: 4139916 at lib/refcount.c:25 refcount_warn_saturate+0x7d/0xe0
	Modules linked in: bpf_preload(E) vhost_net(E) tun(E) vhost(E)

This series of patches introduces a new feature to the netconsole
subsystem that allows the automatic population of the CPU number in the
userdata field for each log message. This enhancement provides several
benefits:

* Improved demultiplexing of parallel log output: When multiple CPUs are
  sending messages concurrently, the added CPU number in the userdata
  makes it easier to differentiate and attribute the messages to their
  originating CPUs.

* Better visibility into message sources: The CPU number information
  gives users and developers more insight into which specific CPU a
  particular log message came from, which can be valuable for debugging
  and analysis.

The changes in this series are as follows Patches::

Patch "consolidate send buffers into netconsole_target struct"
=================================================

Move the static buffers to netconsole target, from static declaration
in send_msg_no_fragmentation() and send_msg_fragmented().

Patch "netconsole: Rename userdata to extradata"
=================================================
Create the a concept of extradata, which encompasses the concept of
userdata and the upcoming sysdatao

Sysdata is a new concept being added, which is basically fields that are
populated by the kernel. At this time only the CPU#, but, there is a
desire to add current task name, kernel release version, etc.

Patch "netconsole: Helper to count number of used entries"
===========================================================
Create a simple helper to count number of entries in extradata. I am
separating this in a function since it will need to count userdata and
sysdata. For instance, when the user adds an extra userdata, we need to
check if there is space, counting the previous data entries (from
userdata and cpu data)

Patch "Introduce configfs helpers for sysdata features"
======================================================
Create the concept of sysdata feature in the netconsole target, and
create the configfs helpers to enable the bit in nt->sysdata

Patch "Include sysdata in extradata entry count"
================================================
Add the concept of sysdata when counting for available space in the
buffer. This will protect users from creating new userdata/sysdata if
there is no more space

Patch "netconsole: add support for sysdata and CPU population"
===============================================================
This is the core patch. Basically add a new option to enable automatic
CPU number population in the netconsole userdata Provides a new "cpu_nr"
sysfs attribute to control this feature

Patch "netconsole: selftest: test CPU number auto-population"
=============================================================
Expands the existing netconsole selftest to verify the CPU number
auto-population functionality Ensures the received netconsole messages
contain the expected "cpu=<CPU>" entry in the message. Test different
permutation with userdata

Patch "netconsole: docs: Add documentation for CPU number auto-population"
=============================================================================
Updates the netconsole documentation to explain the new CPU number
auto-population feature Provides instructions on how to enable and use
the feature

I believe these changes will be a valuable addition to the netconsole
subsystem, enhancing its usefulness for kernel developers and users.

Signed-off-by: Breno Leitao <[e-mail]>
---
Changes in v3:
- Moved the buffer into netconsole_target, avoiding static functions in
  the send path (Jakub).
- Fix a documentation error (Randy Dunlap)
- Created a function that handle all the extradata, consolidating it in
  a single place (Jakub)
- Split the patch even more, trying to simplify the review.
- Link to v2: https://lore.kernel.org/r/[e-mail]

Changes in v2:
- Create the concept of extradata and sysdata. This will make the design
  easier to understand, and the code easier to read.
  * Basically extradata encompasses userdata and the new sysdata.
    Userdata originates from user, and sysdata originates in kernel.
- Improved the test to send from a very specific CPU, which can be
  checked to be correct on the other side, as suggested by Jakub.
- Fixed a bug where CPU # was populated at the wrong place
- Link to v1: https://lore.kernel.org/r/[e-mail]

---
Breno Leitao (8):
      netconsole: consolidate send buffers into netconsole_target struct
      netconsole: Rename userdata to extradata
      netconsole: Helper to count number of used entries
      netconsole: Introduce configfs helpers for sysdata features
      netconsole: Include sysdata in extradata entry count
      netconsole: add support for sysdata and CPU population
      netconsole: selftest: test for sysdata CPU
      netconsole: docs: Add documentation for CPU number auto-population

 Documentation/networking/netconsole.rst            |  45 ++++
 drivers/net/netconsole.c                           | 260 ++++++++++++++++-----
 tools/testing/selftests/drivers/net/Makefile       |   1 +
 .../selftests/drivers/net/lib/sh/lib_netcons.sh    |  17 ++
 .../selftests/drivers/net/netcons_sysdata.sh       | 167 +++++++++++++
 5 files changed, 426 insertions(+), 64 deletions(-)
---
base-commit: fa10c9f5aa705deb43fd65623508074bee942764
change-id: 20241108-netcon_cpu-ce3917e88f4b

Best regards,

Comments

Andrew Lunn Jan. 24, 2025, 4:02 p.m. UTC | #1

On Fri, Jan 24, 2025 at 07:16:39AM -0800, Breno Leitao wrote:
> The current implementation of netconsole sends all log messages in
> parallel, which can lead to an intermixed and interleaved output on the
> receiving side. This makes it challenging to demultiplex the messages
> and attribute them to their originating CPUs.
> 
> As a result, users and developers often struggle to effectively analyze
> and debug the parallel log output received through netconsole.

I know very little about consoles and netconsle, so this is probably a
silly question:

Why is this a netconsole problem, and not a generic console problem?

Can other console types also send in parallel? Do they have the same
issue of intermixing?

	Andrew

Breno Leitao Jan. 27, 2025, 9:52 a.m. UTC | #2

Hello Andrew,

On Fri, Jan 24, 2025 at 05:02:26PM +0100, Andrew Lunn wrote:
> On Fri, Jan 24, 2025 at 07:16:39AM -0800, Breno Leitao wrote:
> > The current implementation of netconsole sends all log messages in
> > parallel, which can lead to an intermixed and interleaved output on the
> > receiving side. This makes it challenging to demultiplex the messages
> > and attribute them to their originating CPUs.
> > 
> > As a result, users and developers often struggle to effectively analyze
> > and debug the parallel log output received through netconsole.
> 
> I know very little about consoles and netconsle, so this is probably a
> silly question:
> 
> Why is this a netconsole problem, and not a generic console problem?

This issue isn't inherent to netconsole. To provide more context and
clarity, let me take a step back and revisit the history of this
discussion, where the idea of adding enriched format originated.

Initially, Calvin proposed adding similar messages, such as the kernel
release version information to messages via printk, but this approach
was deemed inappropriate. The discussions could be found the following
link:

https://lore.kernel.org/all/510[e-mail]/

Later, we shifted to implementing such enriched messages in netconsole,
which proved to be a less intrusive solution. I implemented the release
append in netconsole, effectively addressing Calvin's original concern.

https://lore.kernel.org/all/[e-mail]/

The release append proved to be very useful, the concept evolved
further during discussions at Linux Plumbers Conference, where we
developed the userdata feature, where any userspace data/text can append
any message that flies together with the message.

https://www.youtube.com/watch?v=ILTqn1EYIXQ

This functionality has become *extremely* valuable for hyperscale
environments, leading to current efforts to expand its capabilities
- specifically by adding CPU information and, in future updates, the
current task name.

For instance, at meta, we append service name that is running when
"something happen" (warning, crash, etc) in the kernel. That helps to
narrow down and categorize issues very easily.

> Can other console types also send in parallel? Do they have the same
> issue of intermixing?

Interpreting logs is straightforward when dealing with a single machine.

However, the complexity increases exponentially when managing a large
number of servers and processing logs to gather metrics on systems,
kernels, and more.

For instance, let's come back to appending the kernel version. When
working with a single kernel/host, identifying the kernel version for
a host is simple. If a warning message appears, you can easily attribute
it to that specific kernel version. 

In contrast, with millions of servers running multiple kernel versions
and releases, the challenge lies in accurately mapping warnings to their
corresponding kernel versions and releases, that is why having the
kernel release together with the message make the mapping easy.

Thanks for your time reading it and the discussion,
--breno