mbox series

[00/37] cpu/hotplug, x86: Reworked parallel CPU bringup

Message ID 20230414225551.858160935@linutronix.de (mailing list archive)
Headers show
Series cpu/hotplug, x86: Reworked parallel CPU bringup | expand

Message

Thomas Gleixner April 14, 2023, 11:44 p.m. UTC
Hi!

This is a complete rework of the parallel bringup patch series (V17)

    https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@bytedance.com

to address the issues which were discovered in review:

 1) The X86 microcode loader serialization requirement

    https://lore.kernel.org/lkml/87v8iirxun.ffs@tglx

    Microcode loading on HT enabled X86 CPUs requires that the microcode is
    loaded on the primary thread. The sibling thread(s) must be in
    quiescent state; either looping in a place which is aware of potential
    changes by the microcode update (see late loading) or in fully quiescent
    state, i.e. waiting for INIT/SIPI.

    This is required by hardware/firmware on Intel. Aside of that it's a
    vendor independent software correctness issue. Assume the following
    sequence:

    CPU1.0	  	      CPU1.1
    			      CPUID($A)
    Load microcode.
    Changes CPUID($A, $B)
    			      CPUID($B)

    CPU1.1 makes a decision on $A and $B which might be inconsistent due
    to the microcode update.

    The solution for this is to bringup the primary threads first and after
    that the siblings. Loading microcode on the siblings is a NOOP on Intel
    and on AMD it is guaranteed to only modify thread local state.

    This ensures that the APs can load microcode before reaching the alive
    synchronization point w/o doing any further x86 specific
    synchronization between the core siblings.

 2) The general design issues discussed in V16

    https://lore.kernel.org/lkml/87pm8y6yme.ffs@tglx

    The previous parallel bringup patches just glued this mechanism into
    the existing code without a deeper analysis of the synchronization
    mechanisms and without generalizing it so that the control logic is
    mostly in the core code and not made an architecture specific tinker
    space.

    Much of that had been pointed out 2 years ago in the discussions about
    the early versions of parallel bringup already.


The series is based on:

  git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip x86/apic

and also available from git:

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug


Background
----------

The reason why people are interested in parallel bringup is to shorten
the (kexec) reboot time of cloud servers to reduce the downtime of the
VM tenants. There are obviously other interesting use cases for this
like VM startup time, embedded devices...

The current fully serialized bringup does the following per AP:

    1) Prepare callbacks (allocate, intialize, create threads)
    2) Kick the AP alive (e.g. INIT/SIPI on x86)
    3) Wait for the AP to report alive state
    4) Let the AP continue through the atomic bringup
    5) Let the AP run the threaded bringup to full online state

There are two significant delays:

    #3 The time for an AP to report alive state in start_secondary() on x86
       has been measured in the range between 350us and 3.5ms depending on
       vendor and CPU type, BIOS microcode size etc.

    #4 The atomic bringup does the microcode update. This has been measured
       to take up to ~8ms on the primary threads depending on the microcode
       patch size to apply.

On a two socket SKL server with 56 cores (112 threads) the boot CPU spends
on current mainline about 800ms busy waiting for the APs to come up and
apply microcode. That's more than 80% of the actual onlining procedure.

By splitting the actual bringup mechanism into two parts this can be
reduced to waiting for the first AP to report alive or if the system is
large enough the first AP is already waiting when the boot CPU finished the
wake-up of the last AP.


The actual solution comes in several parts
------------------------------------------

 1) [P 1-2] General cleanups (init annotations, kernel doc...)

 2) [P 3] The obvious

    Avoid pointless delay calibration when TSC is synchronized across
    sockets. That removes a whopping 100ms delay for the first CPU of a
    socket. This is an improvement independent of parallel bringup and had
    been discussed two years ago already.

 2) [P 3-6] Removal of the CPU0 hotplug hack.

    This was added 11 years ago with the promise to make this a real
    hardware mechanism, but that never materialized. As physical CPU
    hotplug is not really supported and the physical unplugging of CPU0
    never materialized there is no reason to keep this cruft around. It's
    just maintenance ballast for no value and the removal makes
    implementing the parallel bringup feature way simpler.

 3) [P 7-16] Cleanup of the existing bringup mechanism:

     a) Code reorganisation so that the general hotplug specific code is
        in smpboot.c and not sprinkled all over the place

     b) Decouple MTRR/PAT initialization from smp_callout_mask to prepare
        for replacing that mask with a hotplug core code synchronization
        mechanism.

     c) Make TSC synchronization function call based so that the control CPU
        does not have to busy wait for nothing if synchronization is not
        required.

     d) Remove the smp_callin_mask synchronization point as its not longer
        required due to #3c.

     e) Rework the sparse_irq_lock held region in the core code so that the
        next polling synchronization point in the x86 code can be removed to.

     f) Due to #3e it's not longer required to spin wait for the AP to set
        it's online bit.  Remove wait_cpu_online() and the XENPV
        counterpart. So the control CPU can directly wait for the online
        idle completion by the AP and free the control CPU up for other
        work.

     This reduces the synchronization points in the x86 code to one, which
     is the AP alive one. This synchronization will be moved to core
     infrastructure in the next section.

 4) [P 17-27] Replace the disconnected CPU state tracking

    The extra CPU state tracking which is used by a few architectures is
    completely separate from the CPU hotplug core code.

    Replacing it by a variant integrated in the core hotplug machinery
    allows to reduce architecture specific code and provides a generic
    synchronization mechanism for (parallel) CPU bringup/teardown.

    - Convert x86 over and replace the AP alive synchronization on x86 with
      the core variant which removes the remaining x86 hotplug
      synchronization masks.

    - Convert the other architectures usage and remove the old interface
      and code.

 5) [P 28-30] Split the bringup into two steps

    First step invokes the wakeup function on the BP, e.g. SIPI/STARTUP on
    x86. The second one waits on the BP for the AP to report alive and
    releases it for the complete onlining.

    As the hotplug state machine allows partial bringup this allows later
    to kick all APs alive in a first iteration and then bring them up
    completely one by one afterwards.

 6) [P 31] Switch the primary thread detection to a cpumask

    This makes the parallel bringup a simple cpumask based mechanism
    without tons of conditionals and checks for primary threads.

 7) [P 32] Implement the parallel bringup core code

    The parallel bringup looks like this:
    
      1) Bring up the primary SMT threads to the CPUHP_KICK_AP_ALIVE step
      	 one by one

      2) Bring up the primary SMT threads to the CPUHP_ONLINE step one by
      	 one

      3) Bring up the secondary SMT threads to the CPUHP_KICK_AP_ALIVE
      	 step one by one

      4) Bring up the secondary SMT threads to the CPUHP_ONLINE
      	 step one by one

    In case that SMT is not supported this is obviously reduced to step #1
    and #2.

 8) [P 33-37] Prepare X86 for parallel bringup and enable it


Caveats
-------

The non X86 changes have been all compile tested. Boot and runtime
testing has only be done on a few real hardware platforms and qemu as
available. That definitely needs some help from the people who have
these systems at their fingertips.


Results and analysis
--------------------

Here are numbers for a dual socket SKL 56 cores/ 112 threads machine.  All
numbers in milliseconds. The time measured is the time which the cpu_up()
call takes for each CPU and phase. It's not exact as the system is already
scheduling, handling interrupts and soft interrupts, which is obviously
skewing the picture slightly.

Baseline tip tree x86/apic branch.

		total      avg/CPU          min          max
total  :      912.081        8.217        3.720      113.271

The max of 100ms is due to the silly delay calibration for the second
socket which takes 100ms and was eliminated first. Also the other initial
cleanups and improvements take some time away.

So the real baseline becomes:

		total      avg/CPU          min          max
total  :      785.960        7.081        3.752       36.098

The max here is on the first CPU of the second socket. 20ms of that is due
to TSC synchronization and an extra 2ms to react on the SIPI.

With parallel bootup enabled this becomes:

		total      avg/CPU          min          max
prepare:       39.108        0.352        0.238        0.883
online :       45.166        0.407        0.170       20.357
total  :       84.274        0.759        0.408       21.240

That's a factor ~9.3 reduction on average.

Looking at the 27 primary threads of socket 0 then this becomes even more
interesting:

		total      avg/CPU          min          max
total  :      325.764       12.065       11.981       14.125

versus:
		total      avg/CPU          min          max
prepare:        8.945        0.331        0.238        0.834
online :        4.830        0.179        0.170        0.212
total  :       13.775        0.510        0.408        1.046

So the reduction factor is ~23.5 here. That's mostly because the 20ms TSC
sync is not skewing the picture.

For all 55 primaries, i.e with the 20ms TSC sync extra for socket 1 this
becomes:

                total      avg/CPU          min          max
total  :      685.489       12.463       11.975       36.098

versus:

                total      avg/CPU          min          max
prepare:       19.080        0.353        0.238        0.883
online :       30.283        0.561        0.170       20.357
total  :       49.363        0.914        0.408       21.240

The TSC sync reduces the win to a factor of ~13.8

With 'tsc=reliable' on the command line the socket sync is disabled which
brings it back to the socket 0 numbers:

                total      avg/CPU          min          max
prepare:       18.970        0.351        0.231        0.874
online :       10.328        0.191        0.169        0.358
total  :       29.298        0.543        0.400        1.232

Now looking at the secondary threads only:

                total      avg/CPU          min          max
total  :      100.471        1.794        0.375        4.745

versus:
                total      avg/CPU          min          max
prepare:       19.753        0.353        0.257        0.512
online :       14.671        0.262        0.179        3.461
total  :       34.424        0.615        0.436        3.973

Still a factor of ~3.

The average on the secondaries for the serialized bringup is significantly
lower than for the primaries because the SIPI response time is shorter and
the microcode update takes no time.

This varies wildly with the system, whether microcode in BIOS is already up
to date, how big the microcode patch is and how long the INIT/SIPI response
time is. On an AMD Zen3 machine INIT/SIPI response time is amazingly fast
(350us), but then it lacks TSC_ADJUST and does a two millisecond TSC sync
test for _every_ AP. All of this sucks...


Possible further enhancements
-----------------------------

It's definitely worthwhile to look into reducing the cross socket TSC sync
test time. It's probably safe enough to use 5ms or even 2ms instead of 20ms
on systems with TSC_ADJUST and a few other 'TSC is sane' indicators. Moving
it out of the hotplug path is eventually possible, but that needs some deep
thoughts.

Let's take the TSC sync out of the picture by adding 'tsc=reliable" to the
kernel command line. So the bringup of 111 APs takes:

                total      avg/CPU          min          max
prepare:       38.936        0.351        0.231        0.874
online :       25.231        0.227        0.169        3.465
total  :       64.167        0.578        0.400        4.339

Some of the outliers are not necessarily in the state callbacks as the
system is already scheduling and handles interrupts and soft
interrupts. Haven't analyzed that yet in detail.

In the prepare stage which runs on the control CPU the larger steps are:

  smpcfd:prepare           16us  avg/CPU
  threads:prepare          98us  avg/CPU
  workqueue:prepare        43us  avg/CPU
  trace/RB:prepare	  135us	 avg/CPU

The trace ringbuffer initialization allocates 354 pages and 354 control
structures one by one. That probably should allocate a large page and an
array of control structures and work from there. I'm sure that would reduce
this significantly. Steven?

smpcfd does just a percpu allocation. No idea why that takes that long.

Vs. threads and workqueues. David thought about spreading out the
preparation work and do it really in parallel. That's a nice idea, but the
threads and workqueue prepare steps are self serializing. The workqueue one
has a global mutex and aside of that both steps create kernel threads which
implicitely serialize on kthreadd. alloc_percpu(), which is used by
smpcfd:prepare is also globally serialized.

The rest of the prepare steps is pretty much in the single digit
microseconds range.

On the AP side it should be possible to move some of the initialization
steps before the alive synchronization point, but that really needs a lot
of analysis whether the functions are safe to invoke that early and outside
of the cpu_hotplug_lock held region for the case of two stage parallel
bringup; see below.

The largest part is:

    identify_secondary_cpu()	99us avg/CPU
   
    Inside of identify_secondary_cpu() the largest offender:

      mcheck_init()		73us avg/CPU

    This part is definitly worth to be looked at whether it can be at least
    partially moved to the early startup code before the alive
    synchronization point. There's a lot of deep analysis required and
    ideally we just rewrite the whole CPUID evaluation trainwreck
    completely.

The rest of the AP side is low single digit microseconds except of:

    perf/x86:starting		14us avg/CPU

    smpboot/threads:online	13us avg/CPU
    workqueue:online		17us avg/CPU
    mm/vmstat:online		17us avg/CPU
    sched:active		30us avg/CPU

sched:active is special. Onlining the first secondary HT thread on the
second socket creates a 3.2ms outlier which skews the whole picture. That's
caused by enabling the static key sched_smt_present which patches the world
and some more. For all other APs this is really in the 1us range. This
definitely could be postponed during bootup like the scheduler domain
rebuild is done after the bringup. But that's still fully serialized and
single threaded and obviously could be done later in the context of async
parallel init. It's unclear why this is different with the fully serialized
bringup where it takes significantly less time, but that's something which
needs to be investigated.


Is truly parallel bringup feasible?
-----------------------------------

In theory yes, realistically no. Why?

   1) The preparation phase

      Allocating memory, creating threads for the to be brought up CPU must
      obviously happen on an already online CPU.

      While it would be possible to bring up a subset of CPUs first and let
      them do the preparation steps for groups of still offline CPUs
      concurrently, the actual benefit of doing so is dubious.

      The prime example is kernel thread creation, which is implicitely
      serialized on kthreadd.

      A simple experiment shows that 4 concurrent workers on 4 different
      CPUs where each is creating 14 * 5 = 70 kernel threads are 5% slower
      than a single worker creating 4 * 14 * 5 = 280 threads.

      So we'd need to have multiple kthreadd instances to handle that,
      which would then serialize on tasklist lock and other things.

      That aside the preparation phase is also affected by the problem
      below.

   2) Assumptions about hotplug serialization

      a) There are quite some assumptions about CPU bringup being fully
         serialized across state transitions.  A lot of state callbacks rely
         on that and would require local locking.

	 Adding that local locking is surely possible, but that has several
	 downsides:

          - It adds complexity and makes it harder for developers to get
	    this correct. The subtle bugs resulting out of that are going
	    to be interesting

          - Fine grained locking has a charm, but only if the time spent
	    for the actual work is larger than the time required for
	    serialization and synchronization.

	    Serializing a callback which takes less than a microsecond and
	    then having a large number of CPUs contending on the lock will
	    not make it any faster at all. That's a well known issue of
	    parallelizing and neither made up nor kernel specific.

      b) Some operations definitely require to be protected by the
         cpu_hotplug_lock, especially those which affect cpumasks as the
         masks are guaranteed to be stable in a cpus_read_lock()'ed region.

       	 As this lock cannot be taken in atomic contexts, it's required
       	 that the control CPU holds the lock write locked across these
       	 state transitions. And no, we are not making this a spinlock just
       	 for that and we even can't.

       	 Just slapping a lock into the x86 specific part of the cpumask
       	 update function does not solve anything. The relevant patch in V17
       	 is completely useless as it only serializes the actual cpumask/map
       	 modifications, but all read side users are hosed if the update
       	 would be moved before the alive synchronization point, i.e. into a
       	 non hotplug lock protected region.

       	 Even if the hotplug lock would be held accross the whole parallel
       	 bringup operation then this would still expose all usage of these
       	 masks and maps in the actual hotplug state callbacks to concurrent
       	 modifications.

       	 And no, we are not going to expose an architecture specific raw
       	 spinlock to the hotplug state callbacks, especially not to those
       	 in generic code.

      c) Some cpu_read_lock()'ed regions also expect that there is no CPU
      	 state transition happening which would modify their local
      	 state. This would again require local serialization.

    3) The amount of work and churn:

       - Analyze the per architecture low level startup functions plus their
         descendant functions and make them ready for concurrency if
       	 necessary.

       - Analyze ~300 hotplug state callbacks and their descendant functions
         and make them ready for concurrency if necessary.

       - Analyze all cpus_read_lock()'ed regions and address their
         requirements.
      
       - Rewrite the core code to handle the cpu_hotplug_lock requirements
         only in distinct phases of the state machine.

       - Rewrite the core code to handle state callback failure and the
         related rollback in the context of the new rules.

      - ...

   Even if some people are dedicated enough to do that, it's very
   questionable whether the resulting complexity is justified.

   We've spent a serious amount of time to sanitize hotplug and bring it
   into a state where it is correct. This also made it reasonably simple
   for developers to implement hotplug state callbacks without having to
   become hotplug experts.

   Breaking this completely up will result in a flood of hard to diagnose
   subtle issues for sure. Who is going to deal with them?

   The experience with this series so far does not make me comfortable
   about that thought in any way.


Summary
-------

The obvious and low hanging fruits have to be solved first:

  - The CPUID evaluation and related setup mechanisms

  - The trace/ringbuffer oddity

  - The sched:active oddity for the first sibling on the second socket
  
  - Some other expensive things which I'm not seeing in my test setup due
    to lack of hardware or configuration.

Anything else is pretty much wishful thinking in my opinion.

  To be clear. I'm not standing in the way if there is a proper solution,
  but that requires to respect the basic engineering rules:

    1) Correctness first
    2) Keep it maintainable
    3) Keep it simple

  So far this stuff failed already at #1.

I completely understand why this is important for cloud people, but
the real question to ask here is what are the actual requirements.

  As far as I understand the main goal is to make a (kexec) reboot
  almost invisible to VM tenants.

  Now lets look at how this works:

     A) Freeze VMs and persist state
     B) kexec into the new kernel
     C) Restore VMs from persistant memory
     D) Thaw VMs

  So the key problem is how long it takes to get from #B to #C and finally
  to #D.

  As far as I understand #C takes a serious amount of time and cannot be
  parallelized for whatever reasons.

  At the same time the number of online CPUs required to restore the VMs
  state is less than the number of online CPUs required to actually
  operate them in #D.

  That means it would be good enough to return to userspace with a
  limited number of online CPUs as fast as possible. A certain amount of
  CPUs are going to be busy with restoring the VMs state, i.e. one CPU
  per VM. Some remaining non-busy CPU can bringup the rest of the system
  and the APs in order to be functional for #D, i.e the restore of VM
  operation.

  Trying to optimize this purely in kernel space by adding complexity of
  dubious value is simply bogus in my opinion.

  It's already possible today to limit the number of CPUs which are
  initially onlined and online the rest later from user space.

  There are two issue there:

    a) The death by MCE broadcast problem

       Quite some (contemporary) x86 CPU generations are affected by
       this:

         - MCE can be broadcasted to all CPUs and not only issued locally
           to the CPU which triggered it.

         - Any CPU which has CR4.MCE == 0, even if it sits in a wait
           for INIT/SIPI state, will cause an immediate shutdown of the
           machine if a broadcasted MCE is delivered.

    b) Do the parallel bringup via sysfs control knob

       The per CPU target state interface allows to do that today one
       by one, but it's akward and has quite some overhead.

       A knob to online the rest of the not yet onlined present CPUs
       with the benefit of the parallel bringup mechanism is
       missing.

    #a) That's a risk to take by the operator.

        Even the regular serialized bringup does not protect against this
     	issue up to the point where all present CPUs have at least
     	initialized CR4.

	Limiting the number of APs to online early via the kernel command
	line widens that window and increases the risk further by
	executing user space before all APs have CR4 initialized.

	But the same applies to a deferred online mechanism implemented in
	the kernel where some worker brings up the not yet online APs while
	the early online CPUs are already executing user space code.

    #b) Is a no brainer to implement on top of this.


Conclusion
----------

Adding the basic parallel bringup mechanism as provided by this series
makes a lot of sense. Improving particular issues as pointed out in the
analysis makes sense too.

But trying to solve an application specific problem fully in the kernel
with tons of complexity, without exploring straight forward and simple
approaches first, does not make any sense at all.

Thanks,

	tglx

---
 Documentation/admin-guide/kernel-parameters.txt |   20 
 Documentation/core-api/cpu_hotplug.rst          |   13 
 arch/Kconfig                                    |   23 +
 arch/arm/Kconfig                                |    1 
 arch/arm/include/asm/smp.h                      |    2 
 arch/arm/kernel/smp.c                           |   18 
 arch/arm64/Kconfig                              |    1 
 arch/arm64/include/asm/smp.h                    |    2 
 arch/arm64/kernel/smp.c                         |   14 
 arch/csky/Kconfig                               |    1 
 arch/csky/include/asm/smp.h                     |    2 
 arch/csky/kernel/smp.c                          |    8 
 arch/mips/Kconfig                               |    1 
 arch/mips/cavium-octeon/smp.c                   |    1 
 arch/mips/include/asm/smp-ops.h                 |    1 
 arch/mips/kernel/smp-bmips.c                    |    1 
 arch/mips/kernel/smp-cps.c                      |   14 
 arch/mips/kernel/smp.c                          |    8 
 arch/mips/loongson64/smp.c                      |    1 
 arch/parisc/Kconfig                             |    1 
 arch/parisc/kernel/process.c                    |    4 
 arch/parisc/kernel/smp.c                        |    7 
 arch/riscv/Kconfig                              |    1 
 arch/riscv/include/asm/smp.h                    |    2 
 arch/riscv/kernel/cpu-hotplug.c                 |   14 
 arch/x86/Kconfig                                |   45 --
 arch/x86/include/asm/apic.h                     |    5 
 arch/x86/include/asm/cpu.h                      |    5 
 arch/x86/include/asm/cpumask.h                  |    5 
 arch/x86/include/asm/processor.h                |    1 
 arch/x86/include/asm/realmode.h                 |    3 
 arch/x86/include/asm/sev-common.h               |    3 
 arch/x86/include/asm/smp.h                      |   26 -
 arch/x86/include/asm/topology.h                 |   23 -
 arch/x86/include/asm/tsc.h                      |    2 
 arch/x86/kernel/acpi/sleep.c                    |    9 
 arch/x86/kernel/apic/apic.c                     |   22 -
 arch/x86/kernel/callthunks.c                    |    4 
 arch/x86/kernel/cpu/amd.c                       |    2 
 arch/x86/kernel/cpu/cacheinfo.c                 |   21 
 arch/x86/kernel/cpu/common.c                    |   50 --
 arch/x86/kernel/cpu/topology.c                  |    3 
 arch/x86/kernel/head_32.S                       |   14 
 arch/x86/kernel/head_64.S                       |  121 +++++
 arch/x86/kernel/sev.c                           |    2 
 arch/x86/kernel/smp.c                           |    3 
 arch/x86/kernel/smpboot.c                       |  508 ++++++++----------------
 arch/x86/kernel/topology.c                      |   98 ----
 arch/x86/kernel/tsc.c                           |   20 
 arch/x86/kernel/tsc_sync.c                      |   36 -
 arch/x86/power/cpu.c                            |   37 -
 arch/x86/realmode/init.c                        |    3 
 arch/x86/realmode/rm/trampoline_64.S            |   27 +
 arch/x86/xen/enlighten_hvm.c                    |   11 
 arch/x86/xen/smp_hvm.c                          |   16 
 arch/x86/xen/smp_pv.c                           |   56 +-
 drivers/acpi/processor_idle.c                   |    4 
 include/linux/cpu.h                             |    4 
 include/linux/cpuhotplug.h                      |   17 
 kernel/cpu.c                                    |  397 +++++++++++++++++-
 kernel/smp.c                                    |    2 
 kernel/smpboot.c                                |  163 -------
 62 files changed, 953 insertions(+), 976 deletions(-)

Comments

Jürgen Groß April 17, 2023, 8:35 a.m. UTC | #1
On 15.04.23 01:44, Thomas Gleixner wrote:
> Hi!
> 
> This is a complete rework of the parallel bringup patch series (V17)
> 
>      https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@bytedance.com
> 
> to address the issues which were discovered in review:
> 
>   1) The X86 microcode loader serialization requirement
> 
>      https://lore.kernel.org/lkml/87v8iirxun.ffs@tglx
> 
>      Microcode loading on HT enabled X86 CPUs requires that the microcode is
>      loaded on the primary thread. The sibling thread(s) must be in
>      quiescent state; either looping in a place which is aware of potential
>      changes by the microcode update (see late loading) or in fully quiescent
>      state, i.e. waiting for INIT/SIPI.
> 
>      This is required by hardware/firmware on Intel. Aside of that it's a
>      vendor independent software correctness issue. Assume the following
>      sequence:
> 
>      CPU1.0	  	      CPU1.1
>      			      CPUID($A)
>      Load microcode.
>      Changes CPUID($A, $B)
>      			      CPUID($B)
> 
>      CPU1.1 makes a decision on $A and $B which might be inconsistent due
>      to the microcode update.
> 
>      The solution for this is to bringup the primary threads first and after
>      that the siblings. Loading microcode on the siblings is a NOOP on Intel
>      and on AMD it is guaranteed to only modify thread local state.
> 
>      This ensures that the APs can load microcode before reaching the alive
>      synchronization point w/o doing any further x86 specific
>      synchronization between the core siblings.
> 
>   2) The general design issues discussed in V16
> 
>      https://lore.kernel.org/lkml/87pm8y6yme.ffs@tglx
> 
>      The previous parallel bringup patches just glued this mechanism into
>      the existing code without a deeper analysis of the synchronization
>      mechanisms and without generalizing it so that the control logic is
>      mostly in the core code and not made an architecture specific tinker
>      space.
> 
>      Much of that had been pointed out 2 years ago in the discussions about
>      the early versions of parallel bringup already.
> 
> 
> The series is based on:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip x86/apic
> 
> and also available from git:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug
> 
> 
> Background
> ----------
> 
> The reason why people are interested in parallel bringup is to shorten
> the (kexec) reboot time of cloud servers to reduce the downtime of the
> VM tenants. There are obviously other interesting use cases for this
> like VM startup time, embedded devices...
> 
> The current fully serialized bringup does the following per AP:
> 
>      1) Prepare callbacks (allocate, intialize, create threads)
>      2) Kick the AP alive (e.g. INIT/SIPI on x86)
>      3) Wait for the AP to report alive state
>      4) Let the AP continue through the atomic bringup
>      5) Let the AP run the threaded bringup to full online state
> 
> There are two significant delays:
> 
>      #3 The time for an AP to report alive state in start_secondary() on x86
>         has been measured in the range between 350us and 3.5ms depending on
>         vendor and CPU type, BIOS microcode size etc.
> 
>      #4 The atomic bringup does the microcode update. This has been measured
>         to take up to ~8ms on the primary threads depending on the microcode
>         patch size to apply.
> 
> On a two socket SKL server with 56 cores (112 threads) the boot CPU spends
> on current mainline about 800ms busy waiting for the APs to come up and
> apply microcode. That's more than 80% of the actual onlining procedure.
> 
> By splitting the actual bringup mechanism into two parts this can be
> reduced to waiting for the first AP to report alive or if the system is
> large enough the first AP is already waiting when the boot CPU finished the
> wake-up of the last AP.
> 
> 
> The actual solution comes in several parts
> ------------------------------------------
> 
>   1) [P 1-2] General cleanups (init annotations, kernel doc...)
> 
>   2) [P 3] The obvious
> 
>      Avoid pointless delay calibration when TSC is synchronized across
>      sockets. That removes a whopping 100ms delay for the first CPU of a
>      socket. This is an improvement independent of parallel bringup and had
>      been discussed two years ago already.
> 
>   2) [P 3-6] Removal of the CPU0 hotplug hack.
> 
>      This was added 11 years ago with the promise to make this a real
>      hardware mechanism, but that never materialized. As physical CPU
>      hotplug is not really supported and the physical unplugging of CPU0
>      never materialized there is no reason to keep this cruft around. It's
>      just maintenance ballast for no value and the removal makes
>      implementing the parallel bringup feature way simpler.
> 
>   3) [P 7-16] Cleanup of the existing bringup mechanism:
> 
>       a) Code reorganisation so that the general hotplug specific code is
>          in smpboot.c and not sprinkled all over the place
> 
>       b) Decouple MTRR/PAT initialization from smp_callout_mask to prepare
>          for replacing that mask with a hotplug core code synchronization
>          mechanism.
> 
>       c) Make TSC synchronization function call based so that the control CPU
>          does not have to busy wait for nothing if synchronization is not
>          required.
> 
>       d) Remove the smp_callin_mask synchronization point as its not longer
>          required due to #3c.
> 
>       e) Rework the sparse_irq_lock held region in the core code so that the
>          next polling synchronization point in the x86 code can be removed to.
> 
>       f) Due to #3e it's not longer required to spin wait for the AP to set
>          it's online bit.  Remove wait_cpu_online() and the XENPV
>          counterpart. So the control CPU can directly wait for the online
>          idle completion by the AP and free the control CPU up for other
>          work.
> 
>       This reduces the synchronization points in the x86 code to one, which
>       is the AP alive one. This synchronization will be moved to core
>       infrastructure in the next section.
> 
>   4) [P 17-27] Replace the disconnected CPU state tracking
> 
>      The extra CPU state tracking which is used by a few architectures is
>      completely separate from the CPU hotplug core code.
> 
>      Replacing it by a variant integrated in the core hotplug machinery
>      allows to reduce architecture specific code and provides a generic
>      synchronization mechanism for (parallel) CPU bringup/teardown.
> 
>      - Convert x86 over and replace the AP alive synchronization on x86 with
>        the core variant which removes the remaining x86 hotplug
>        synchronization masks.
> 
>      - Convert the other architectures usage and remove the old interface
>        and code.
> 
>   5) [P 28-30] Split the bringup into two steps
> 
>      First step invokes the wakeup function on the BP, e.g. SIPI/STARTUP on
>      x86. The second one waits on the BP for the AP to report alive and
>      releases it for the complete onlining.
> 
>      As the hotplug state machine allows partial bringup this allows later
>      to kick all APs alive in a first iteration and then bring them up
>      completely one by one afterwards.
> 
>   6) [P 31] Switch the primary thread detection to a cpumask
> 
>      This makes the parallel bringup a simple cpumask based mechanism
>      without tons of conditionals and checks for primary threads.
> 
>   7) [P 32] Implement the parallel bringup core code
> 
>      The parallel bringup looks like this:
>      
>        1) Bring up the primary SMT threads to the CPUHP_KICK_AP_ALIVE step
>        	 one by one
> 
>        2) Bring up the primary SMT threads to the CPUHP_ONLINE step one by
>        	 one
> 
>        3) Bring up the secondary SMT threads to the CPUHP_KICK_AP_ALIVE
>        	 step one by one
> 
>        4) Bring up the secondary SMT threads to the CPUHP_ONLINE
>        	 step one by one
> 
>      In case that SMT is not supported this is obviously reduced to step #1
>      and #2.
> 
>   8) [P 33-37] Prepare X86 for parallel bringup and enable it
> 
> 
> Caveats
> -------
> 
> The non X86 changes have been all compile tested. Boot and runtime
> testing has only be done on a few real hardware platforms and qemu as
> available. That definitely needs some help from the people who have
> these systems at their fingertips.
> 
> 
> Results and analysis
> --------------------
> 
> Here are numbers for a dual socket SKL 56 cores/ 112 threads machine.  All
> numbers in milliseconds. The time measured is the time which the cpu_up()
> call takes for each CPU and phase. It's not exact as the system is already
> scheduling, handling interrupts and soft interrupts, which is obviously
> skewing the picture slightly.
> 
> Baseline tip tree x86/apic branch.
> 
> 		total      avg/CPU          min          max
> total  :      912.081        8.217        3.720      113.271
> 
> The max of 100ms is due to the silly delay calibration for the second
> socket which takes 100ms and was eliminated first. Also the other initial
> cleanups and improvements take some time away.
> 
> So the real baseline becomes:
> 
> 		total      avg/CPU          min          max
> total  :      785.960        7.081        3.752       36.098
> 
> The max here is on the first CPU of the second socket. 20ms of that is due
> to TSC synchronization and an extra 2ms to react on the SIPI.
> 
> With parallel bootup enabled this becomes:
> 
> 		total      avg/CPU          min          max
> prepare:       39.108        0.352        0.238        0.883
> online :       45.166        0.407        0.170       20.357
> total  :       84.274        0.759        0.408       21.240
> 
> That's a factor ~9.3 reduction on average.
> 
> Looking at the 27 primary threads of socket 0 then this becomes even more
> interesting:
> 
> 		total      avg/CPU          min          max
> total  :      325.764       12.065       11.981       14.125
> 
> versus:
> 		total      avg/CPU          min          max
> prepare:        8.945        0.331        0.238        0.834
> online :        4.830        0.179        0.170        0.212
> total  :       13.775        0.510        0.408        1.046
> 
> So the reduction factor is ~23.5 here. That's mostly because the 20ms TSC
> sync is not skewing the picture.
> 
> For all 55 primaries, i.e with the 20ms TSC sync extra for socket 1 this
> becomes:
> 
>                  total      avg/CPU          min          max
> total  :      685.489       12.463       11.975       36.098
> 
> versus:
> 
>                  total      avg/CPU          min          max
> prepare:       19.080        0.353        0.238        0.883
> online :       30.283        0.561        0.170       20.357
> total  :       49.363        0.914        0.408       21.240
> 
> The TSC sync reduces the win to a factor of ~13.8
> 
> With 'tsc=reliable' on the command line the socket sync is disabled which
> brings it back to the socket 0 numbers:
> 
>                  total      avg/CPU          min          max
> prepare:       18.970        0.351        0.231        0.874
> online :       10.328        0.191        0.169        0.358
> total  :       29.298        0.543        0.400        1.232
> 
> Now looking at the secondary threads only:
> 
>                  total      avg/CPU          min          max
> total  :      100.471        1.794        0.375        4.745
> 
> versus:
>                  total      avg/CPU          min          max
> prepare:       19.753        0.353        0.257        0.512
> online :       14.671        0.262        0.179        3.461
> total  :       34.424        0.615        0.436        3.973
> 
> Still a factor of ~3.
> 
> The average on the secondaries for the serialized bringup is significantly
> lower than for the primaries because the SIPI response time is shorter and
> the microcode update takes no time.
> 
> This varies wildly with the system, whether microcode in BIOS is already up
> to date, how big the microcode patch is and how long the INIT/SIPI response
> time is. On an AMD Zen3 machine INIT/SIPI response time is amazingly fast
> (350us), but then it lacks TSC_ADJUST and does a two millisecond TSC sync
> test for _every_ AP. All of this sucks...
> 
> 
> Possible further enhancements
> -----------------------------
> 
> It's definitely worthwhile to look into reducing the cross socket TSC sync
> test time. It's probably safe enough to use 5ms or even 2ms instead of 20ms
> on systems with TSC_ADJUST and a few other 'TSC is sane' indicators. Moving
> it out of the hotplug path is eventually possible, but that needs some deep
> thoughts.
> 
> Let's take the TSC sync out of the picture by adding 'tsc=reliable" to the
> kernel command line. So the bringup of 111 APs takes:
> 
>                  total      avg/CPU          min          max
> prepare:       38.936        0.351        0.231        0.874
> online :       25.231        0.227        0.169        3.465
> total  :       64.167        0.578        0.400        4.339
> 
> Some of the outliers are not necessarily in the state callbacks as the
> system is already scheduling and handles interrupts and soft
> interrupts. Haven't analyzed that yet in detail.
> 
> In the prepare stage which runs on the control CPU the larger steps are:
> 
>    smpcfd:prepare           16us  avg/CPU
>    threads:prepare          98us  avg/CPU
>    workqueue:prepare        43us  avg/CPU
>    trace/RB:prepare	  135us	 avg/CPU
> 
> The trace ringbuffer initialization allocates 354 pages and 354 control
> structures one by one. That probably should allocate a large page and an
> array of control structures and work from there. I'm sure that would reduce
> this significantly. Steven?
> 
> smpcfd does just a percpu allocation. No idea why that takes that long.
> 
> Vs. threads and workqueues. David thought about spreading out the
> preparation work and do it really in parallel. That's a nice idea, but the
> threads and workqueue prepare steps are self serializing. The workqueue one
> has a global mutex and aside of that both steps create kernel threads which
> implicitely serialize on kthreadd. alloc_percpu(), which is used by
> smpcfd:prepare is also globally serialized.
> 
> The rest of the prepare steps is pretty much in the single digit
> microseconds range.
> 
> On the AP side it should be possible to move some of the initialization
> steps before the alive synchronization point, but that really needs a lot
> of analysis whether the functions are safe to invoke that early and outside
> of the cpu_hotplug_lock held region for the case of two stage parallel
> bringup; see below.
> 
> The largest part is:
> 
>      identify_secondary_cpu()	99us avg/CPU
>     
>      Inside of identify_secondary_cpu() the largest offender:
> 
>        mcheck_init()		73us avg/CPU
> 
>      This part is definitly worth to be looked at whether it can be at least
>      partially moved to the early startup code before the alive
>      synchronization point. There's a lot of deep analysis required and
>      ideally we just rewrite the whole CPUID evaluation trainwreck
>      completely.
> 
> The rest of the AP side is low single digit microseconds except of:
> 
>      perf/x86:starting		14us avg/CPU
> 
>      smpboot/threads:online	13us avg/CPU
>      workqueue:online		17us avg/CPU
>      mm/vmstat:online		17us avg/CPU
>      sched:active		30us avg/CPU
> 
> sched:active is special. Onlining the first secondary HT thread on the
> second socket creates a 3.2ms outlier which skews the whole picture. That's
> caused by enabling the static key sched_smt_present which patches the world
> and some more. For all other APs this is really in the 1us range. This
> definitely could be postponed during bootup like the scheduler domain
> rebuild is done after the bringup. But that's still fully serialized and
> single threaded and obviously could be done later in the context of async
> parallel init. It's unclear why this is different with the fully serialized
> bringup where it takes significantly less time, but that's something which
> needs to be investigated.
> 
> 
> Is truly parallel bringup feasible?
> -----------------------------------
> 
> In theory yes, realistically no. Why?
> 
>     1) The preparation phase
> 
>        Allocating memory, creating threads for the to be brought up CPU must
>        obviously happen on an already online CPU.
> 
>        While it would be possible to bring up a subset of CPUs first and let
>        them do the preparation steps for groups of still offline CPUs
>        concurrently, the actual benefit of doing so is dubious.
> 
>        The prime example is kernel thread creation, which is implicitely
>        serialized on kthreadd.
> 
>        A simple experiment shows that 4 concurrent workers on 4 different
>        CPUs where each is creating 14 * 5 = 70 kernel threads are 5% slower
>        than a single worker creating 4 * 14 * 5 = 280 threads.
> 
>        So we'd need to have multiple kthreadd instances to handle that,
>        which would then serialize on tasklist lock and other things.
> 
>        That aside the preparation phase is also affected by the problem
>        below.
> 
>     2) Assumptions about hotplug serialization
> 
>        a) There are quite some assumptions about CPU bringup being fully
>           serialized across state transitions.  A lot of state callbacks rely
>           on that and would require local locking.
> 
> 	 Adding that local locking is surely possible, but that has several
> 	 downsides:
> 
>            - It adds complexity and makes it harder for developers to get
> 	    this correct. The subtle bugs resulting out of that are going
> 	    to be interesting
> 
>            - Fine grained locking has a charm, but only if the time spent
> 	    for the actual work is larger than the time required for
> 	    serialization and synchronization.
> 
> 	    Serializing a callback which takes less than a microsecond and
> 	    then having a large number of CPUs contending on the lock will
> 	    not make it any faster at all. That's a well known issue of
> 	    parallelizing and neither made up nor kernel specific.
> 
>        b) Some operations definitely require to be protected by the
>           cpu_hotplug_lock, especially those which affect cpumasks as the
>           masks are guaranteed to be stable in a cpus_read_lock()'ed region.
> 
>         	 As this lock cannot be taken in atomic contexts, it's required
>         	 that the control CPU holds the lock write locked across these
>         	 state transitions. And no, we are not making this a spinlock just
>         	 for that and we even can't.
> 
>         	 Just slapping a lock into the x86 specific part of the cpumask
>         	 update function does not solve anything. The relevant patch in V17
>         	 is completely useless as it only serializes the actual cpumask/map
>         	 modifications, but all read side users are hosed if the update
>         	 would be moved before the alive synchronization point, i.e. into a
>         	 non hotplug lock protected region.
> 
>         	 Even if the hotplug lock would be held accross the whole parallel
>         	 bringup operation then this would still expose all usage of these
>         	 masks and maps in the actual hotplug state callbacks to concurrent
>         	 modifications.
> 
>         	 And no, we are not going to expose an architecture specific raw
>         	 spinlock to the hotplug state callbacks, especially not to those
>         	 in generic code.
> 
>        c) Some cpu_read_lock()'ed regions also expect that there is no CPU
>        	 state transition happening which would modify their local
>        	 state. This would again require local serialization.
> 
>      3) The amount of work and churn:
> 
>         - Analyze the per architecture low level startup functions plus their
>           descendant functions and make them ready for concurrency if
>         	 necessary.
> 
>         - Analyze ~300 hotplug state callbacks and their descendant functions
>           and make them ready for concurrency if necessary.
> 
>         - Analyze all cpus_read_lock()'ed regions and address their
>           requirements.
>        
>         - Rewrite the core code to handle the cpu_hotplug_lock requirements
>           only in distinct phases of the state machine.
> 
>         - Rewrite the core code to handle state callback failure and the
>           related rollback in the context of the new rules.
> 
>        - ...
> 
>     Even if some people are dedicated enough to do that, it's very
>     questionable whether the resulting complexity is justified.
> 
>     We've spent a serious amount of time to sanitize hotplug and bring it
>     into a state where it is correct. This also made it reasonably simple
>     for developers to implement hotplug state callbacks without having to
>     become hotplug experts.
> 
>     Breaking this completely up will result in a flood of hard to diagnose
>     subtle issues for sure. Who is going to deal with them?
> 
>     The experience with this series so far does not make me comfortable
>     about that thought in any way.
> 
> 
> Summary
> -------
> 
> The obvious and low hanging fruits have to be solved first:
> 
>    - The CPUID evaluation and related setup mechanisms
> 
>    - The trace/ringbuffer oddity
> 
>    - The sched:active oddity for the first sibling on the second socket
>    
>    - Some other expensive things which I'm not seeing in my test setup due
>      to lack of hardware or configuration.
> 
> Anything else is pretty much wishful thinking in my opinion.
> 
>    To be clear. I'm not standing in the way if there is a proper solution,
>    but that requires to respect the basic engineering rules:
> 
>      1) Correctness first
>      2) Keep it maintainable
>      3) Keep it simple
> 
>    So far this stuff failed already at #1.
> 
> I completely understand why this is important for cloud people, but
> the real question to ask here is what are the actual requirements.
> 
>    As far as I understand the main goal is to make a (kexec) reboot
>    almost invisible to VM tenants.
> 
>    Now lets look at how this works:
> 
>       A) Freeze VMs and persist state
>       B) kexec into the new kernel
>       C) Restore VMs from persistant memory
>       D) Thaw VMs
> 
>    So the key problem is how long it takes to get from #B to #C and finally
>    to #D.
> 
>    As far as I understand #C takes a serious amount of time and cannot be
>    parallelized for whatever reasons.
> 
>    At the same time the number of online CPUs required to restore the VMs
>    state is less than the number of online CPUs required to actually
>    operate them in #D.
> 
>    That means it would be good enough to return to userspace with a
>    limited number of online CPUs as fast as possible. A certain amount of
>    CPUs are going to be busy with restoring the VMs state, i.e. one CPU
>    per VM. Some remaining non-busy CPU can bringup the rest of the system
>    and the APs in order to be functional for #D, i.e the restore of VM
>    operation.
> 
>    Trying to optimize this purely in kernel space by adding complexity of
>    dubious value is simply bogus in my opinion.
> 
>    It's already possible today to limit the number of CPUs which are
>    initially onlined and online the rest later from user space.
> 
>    There are two issue there:
> 
>      a) The death by MCE broadcast problem
> 
>         Quite some (contemporary) x86 CPU generations are affected by
>         this:
> 
>           - MCE can be broadcasted to all CPUs and not only issued locally
>             to the CPU which triggered it.
> 
>           - Any CPU which has CR4.MCE == 0, even if it sits in a wait
>             for INIT/SIPI state, will cause an immediate shutdown of the
>             machine if a broadcasted MCE is delivered.
> 
>      b) Do the parallel bringup via sysfs control knob
> 
>         The per CPU target state interface allows to do that today one
>         by one, but it's akward and has quite some overhead.
> 
>         A knob to online the rest of the not yet onlined present CPUs
>         with the benefit of the parallel bringup mechanism is
>         missing.
> 
>      #a) That's a risk to take by the operator.
> 
>          Even the regular serialized bringup does not protect against this
>       	issue up to the point where all present CPUs have at least
>       	initialized CR4.
> 
> 	Limiting the number of APs to online early via the kernel command
> 	line widens that window and increases the risk further by
> 	executing user space before all APs have CR4 initialized.
> 
> 	But the same applies to a deferred online mechanism implemented in
> 	the kernel where some worker brings up the not yet online APs while
> 	the early online CPUs are already executing user space code.
> 
>      #b) Is a no brainer to implement on top of this.
> 
> 
> Conclusion
> ----------
> 
> Adding the basic parallel bringup mechanism as provided by this series
> makes a lot of sense. Improving particular issues as pointed out in the
> analysis makes sense too.
> 
> But trying to solve an application specific problem fully in the kernel
> with tons of complexity, without exploring straight forward and simple
> approaches first, does not make any sense at all.
> 
> Thanks,
> 
> 	tglx
> 
> ---
>   Documentation/admin-guide/kernel-parameters.txt |   20
>   Documentation/core-api/cpu_hotplug.rst          |   13
>   arch/Kconfig                                    |   23 +
>   arch/arm/Kconfig                                |    1
>   arch/arm/include/asm/smp.h                      |    2
>   arch/arm/kernel/smp.c                           |   18
>   arch/arm64/Kconfig                              |    1
>   arch/arm64/include/asm/smp.h                    |    2
>   arch/arm64/kernel/smp.c                         |   14
>   arch/csky/Kconfig                               |    1
>   arch/csky/include/asm/smp.h                     |    2
>   arch/csky/kernel/smp.c                          |    8
>   arch/mips/Kconfig                               |    1
>   arch/mips/cavium-octeon/smp.c                   |    1
>   arch/mips/include/asm/smp-ops.h                 |    1
>   arch/mips/kernel/smp-bmips.c                    |    1
>   arch/mips/kernel/smp-cps.c                      |   14
>   arch/mips/kernel/smp.c                          |    8
>   arch/mips/loongson64/smp.c                      |    1
>   arch/parisc/Kconfig                             |    1
>   arch/parisc/kernel/process.c                    |    4
>   arch/parisc/kernel/smp.c                        |    7
>   arch/riscv/Kconfig                              |    1
>   arch/riscv/include/asm/smp.h                    |    2
>   arch/riscv/kernel/cpu-hotplug.c                 |   14
>   arch/x86/Kconfig                                |   45 --
>   arch/x86/include/asm/apic.h                     |    5
>   arch/x86/include/asm/cpu.h                      |    5
>   arch/x86/include/asm/cpumask.h                  |    5
>   arch/x86/include/asm/processor.h                |    1
>   arch/x86/include/asm/realmode.h                 |    3
>   arch/x86/include/asm/sev-common.h               |    3
>   arch/x86/include/asm/smp.h                      |   26 -
>   arch/x86/include/asm/topology.h                 |   23 -
>   arch/x86/include/asm/tsc.h                      |    2
>   arch/x86/kernel/acpi/sleep.c                    |    9
>   arch/x86/kernel/apic/apic.c                     |   22 -
>   arch/x86/kernel/callthunks.c                    |    4
>   arch/x86/kernel/cpu/amd.c                       |    2
>   arch/x86/kernel/cpu/cacheinfo.c                 |   21
>   arch/x86/kernel/cpu/common.c                    |   50 --
>   arch/x86/kernel/cpu/topology.c                  |    3
>   arch/x86/kernel/head_32.S                       |   14
>   arch/x86/kernel/head_64.S                       |  121 +++++
>   arch/x86/kernel/sev.c                           |    2
>   arch/x86/kernel/smp.c                           |    3
>   arch/x86/kernel/smpboot.c                       |  508 ++++++++----------------
>   arch/x86/kernel/topology.c                      |   98 ----
>   arch/x86/kernel/tsc.c                           |   20
>   arch/x86/kernel/tsc_sync.c                      |   36 -
>   arch/x86/power/cpu.c                            |   37 -
>   arch/x86/realmode/init.c                        |    3
>   arch/x86/realmode/rm/trampoline_64.S            |   27 +
>   arch/x86/xen/enlighten_hvm.c                    |   11
>   arch/x86/xen/smp_hvm.c                          |   16
>   arch/x86/xen/smp_pv.c                           |   56 +-
>   drivers/acpi/processor_idle.c                   |    4
>   include/linux/cpu.h                             |    4
>   include/linux/cpuhotplug.h                      |   17
>   kernel/cpu.c                                    |  397 +++++++++++++++++-
>   kernel/smp.c                                    |    2
>   kernel/smpboot.c                                |  163 -------
>   62 files changed, 953 insertions(+), 976 deletions(-)
> 
> 

Tested with a Xen PV dom0 on an 8 cpu system, no issues found.

Tested-by: Juergen Gross <jgross@suse.com>


Juergen
Peter Zijlstra April 17, 2023, 10:30 a.m. UTC | #2
On Sat, Apr 15, 2023 at 01:44:13AM +0200, Thomas Gleixner wrote:

> Background
> ----------
> 
> The reason why people are interested in parallel bringup is to shorten
> the (kexec) reboot time of cloud servers to reduce the downtime of the
> VM tenants. There are obviously other interesting use cases for this
> like VM startup time, embedded devices...

...

>   There are two issue there:
> 
>     a) The death by MCE broadcast problem
> 
>        Quite some (contemporary) x86 CPU generations are affected by
>        this:
> 
>          - MCE can be broadcasted to all CPUs and not only issued locally
>            to the CPU which triggered it.
> 
>          - Any CPU which has CR4.MCE == 0, even if it sits in a wait
>            for INIT/SIPI state, will cause an immediate shutdown of the
>            machine if a broadcasted MCE is delivered.

When doing kexec, CR4.MCE should already have been set to 1 by the prior
kernel, no?
Andrew Cooper April 17, 2023, 10:44 a.m. UTC | #3
On 17/04/2023 11:30 am, Peter Zijlstra wrote:
> On Sat, Apr 15, 2023 at 01:44:13AM +0200, Thomas Gleixner wrote:
>
>> Background
>> ----------
>>
>> The reason why people are interested in parallel bringup is to shorten
>> the (kexec) reboot time of cloud servers to reduce the downtime of the
>> VM tenants. There are obviously other interesting use cases for this
>> like VM startup time, embedded devices...
> ...
>
>>   There are two issue there:
>>
>>     a) The death by MCE broadcast problem
>>
>>        Quite some (contemporary) x86 CPU generations are affected by
>>        this:
>>
>>          - MCE can be broadcasted to all CPUs and not only issued locally
>>            to the CPU which triggered it.
>>
>>          - Any CPU which has CR4.MCE == 0, even if it sits in a wait
>>            for INIT/SIPI state, will cause an immediate shutdown of the
>>            machine if a broadcasted MCE is delivered.
> When doing kexec, CR4.MCE should already have been set to 1 by the prior
> kernel, no?

No(ish).  Purgatory can't take #MC, or NMIs for that matter.

It's cleaner to explicitly disable CR4.MCE and let the system reset
(with all the MC banks properly preserved), than it is to take #MC while
the IDT isn't in sync with the handlers, and wander off into the weeds.

~Andrew
Paul Menzel April 17, 2023, 11:19 a.m. UTC | #4
Dear Thomas,


Am 15.04.23 um 01:44 schrieb Thomas Gleixner:

> This is a complete rework of the parallel bringup patch series (V17)
> 
>      https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@bytedance.com
> 
> to address the issues which were discovered in review:

[…]

Thank you very much for your rework.

I tested this on the ASUS F2A85-M PRO, and get a delay of ten seconds.

```
[…]
[    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD 
Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[…]
[    0.259329] smp: Bringing up secondary CPUs ...
[    0.259527] x86: Booting SMP configuration:
[    0.259528] .... node  #0, CPUs:      #1
[    0.261007] After schedule_preempt_disabled
[   10.260990] CPU1 failed to report alive state
[   10.261070] smp: Brought up 1 node, 1 CPU
[   10.261073] smpboot: Max logical packages: 2
[   10.261074] smpboot: Total of 1 processors activated (7800.54 BogoMIPS)
[   10.261601] devtmpfs: initialized
[   10.261697] x86/mm: Memory block size: 128MB
```

This delay has been there with v6.3-rc6-46-gde4664485abbc and some 
custom (printk) patches on top and merging dwmw2/parallel-6.2-rc3-v16 
into it. I only tested this. I think dwmw2/parallel-6.2-v17 failed to 
build for me, when trying to merge it into Linus’ master version at that 
time. I didn’t come around to report it, and you posted your rework, so 
I am replying here.

I am going to try your branch directly in the next days, but just wanted 
to report back already.


Kind regards,

Paul
[    0.000000] Linux version 6.3.0-rc6-00311-gde8224969f66 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #446 SMP PREEMPT_DYNAMIC Sat Apr 15 14:12:29 UTC 2023
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] signal: max sigframe size: 1776
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005fe4cfff] usable
[    0.000000] BIOS-e820: [mem 0x000000005fe4d000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000017effffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0.0 present.
[    0.000000] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-g9917d2d915 04/17/2023
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Initial usec timer 6035615
[    0.000000] tsc: Detected 3900.273 MHz processor
[    0.000756] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000759] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x400000000
[    0.000768] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000942] last_pfn = 0x5fe4d max_arch_pfn = 0x400000000
[    0.004000] Using GB pages for direct mapping
[    0.004000] ACPI: Early table checksum verification disabled
[    0.004000] ACPI: RSDP 0x00000000000F6830 000024 (v02 COREv4)
[    0.004000] ACPI: XSDT 0x000000005FE5A0E0 000074 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: FACP 0x000000005FE5BBC0 000114 (v06 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: DSDT 0x000000005FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925)
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: SSDT 0x000000005FE5BCE0 00008A (v02 COREv4 COREBOOT 0000002A CORE 20200925)
[    0.004000] ACPI: MCFG 0x000000005FE5BD70 00003C (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: APIC 0x000000005FE5BDB0 000062 (v03 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HPET 0x000000005FE5BE20 000038 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HEST 0x000000005FE5BE60 0001D0 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: IVRS 0x000000005FE5C030 000070 (v02 AMD    AMDIOMMU 00000001 AMD  00000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C0A0 00051F (v02 AMD    ALIB     00000001 MSFT 04000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C5C0 0006B2 (v01 AMD    POWERNOW 00000001 AMD  00000001)
[    0.004000] ACPI: VFCT 0x000000005FE5CC80 00F269 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: Reserving FACP table memory at [mem 0x5fe5bbc0-0x5fe5bcd3]
[    0.004000] ACPI: Reserving DSDT table memory at [mem 0x5fe5a280-0x5fe5bbb9]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5bce0-0x5fe5bd69]
[    0.004000] ACPI: Reserving MCFG table memory at [mem 0x5fe5bd70-0x5fe5bdab]
[    0.004000] ACPI: Reserving APIC table memory at [mem 0x5fe5bdb0-0x5fe5be11]
[    0.004000] ACPI: Reserving HPET table memory at [mem 0x5fe5be20-0x5fe5be57]
[    0.004000] ACPI: Reserving HEST table memory at [mem 0x5fe5be60-0x5fe5c02f]
[    0.004000] ACPI: Reserving IVRS table memory at [mem 0x5fe5c030-0x5fe5c09f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c0a0-0x5fe5c5be]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c5c0-0x5fe5cc71]
[    0.004000] ACPI: Reserving VFCT table memory at [mem 0x5fe5cc80-0x5fe6bee8]
[    0.004000] No NUMA configuration found
[    0.004000] Faking a node at [mem 0x0000000000000000-0x000000017effffff]
[    0.004000] NODE_DATA(0) allocated [mem 0x17efe7000-0x17effdfff]
[    0.004000] Zone ranges:
[    0.004000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.004000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.004000]   Normal   [mem 0x0000000100000000-0x000000017effffff]
[    0.004000]   Device   empty
[    0.004000] Movable zone start for each node
[    0.004000] Early memory node ranges
[    0.004000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.004000]   node   0: [mem 0x0000000000100000-0x000000005fe4cfff]
[    0.004000]   node   0: [mem 0x0000000100000000-0x000000017effffff]
[    0.004000] Initmem setup node 0 [mem 0x0000000000001000-0x000000017effffff]
[    0.004000] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.004000] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 435 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 4096 pages in unavailable ranges
[    0.004000] ACPI: PM-Timer IO Port: 0x818
[    0.004000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.004000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.004000] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.004000] ACPI: HPET id: 0x10228210 base: 0xfed00000
[    0.004000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.004000] smpboot: smpboot: XXX end of prefill_possible_map
[    0.004000] After prefill_possible_map
[    0.004000] After init_cpu_to_node
[    0.004000] After init_gi_nodes
[    0.004000] After io_apic_init_mappings
[    0.004000] After x86_init.hyper.guest_late_init
[    0.004000] [mem 0x80000000-0xf7ffffff] available for PCI devices
[    0.004000] After e820
[    0.004000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.004000] After unwind_init
[    0.004000] After setup_arch
[    0.004000] After setup_command_line
[    0.004000] After setup_nr_cpu_ids
[    0.004000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.004000] percpu: Embedded 55 pages/cpu s188328 r8192 d28760 u1048576
[    0.004000] pcpu-alloc: s188328 r8192 d28760 u1048576 alloc=1*2097152
[    0.004000] pcpu-alloc: [0] 0 1 
[    0.004000] After setup_per_cpu_areas
[    0.004000] After smp_perpare_boot_cpu
[    0.004000] After boot_cpu_hotplug_init
[    0.004000] Fallback order for Node 0: 0 
[    0.004000] Built 1 zonelists, mobility grouping on.  Total pages: 898451
[    0.004000] Policy zone: Normal
[    0.004000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0
[    0.004000] Unknown kernel command line parameters "noisapnp BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66", will be passed to user space.
[    0.004000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.004000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.004000] stackdepot: allocating hash table via alloc_large_system_hash
[    0.004000] stackdepot hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] software IO TLB: area num 2.
[    0.004000] Memory: 3477160K/3651500K available (14336K kernel code, 2340K rwdata, 5308K rodata, 2908K init, 11060K bss, 174080K reserved, 0K cma-reserved)
[    0.004000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.004000] After mm_init
[    0.004000] After poking_init
[    0.004000] ftrace: allocating 38664 entries in 152 pages
[    0.004000] ftrace: allocated 152 pages with 3 groups
[    0.004000] Dynamic Preempt: full
[    0.004000] After sched_init
[    0.004000] rcu: Preemptible hierarchical RCU implementation.
[    0.004000] rcu: 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[    0.004000] 	Trampoline variant of Tasks RCU enabled.
[    0.004000] 	Rude variant of Tasks RCU enabled.
[    0.004000] 	Tracing variant of Tasks RCU enabled.
[    0.004000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.004000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.004000] After rcu_init
[    0.004000] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
[    0.004000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.004000] After random_init()
[    0.004000] After boot_init_stack_canary
[    0.004000] spurious 8259A interrupt: IRQ7.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] printk: console [tty0] enabled
[    0.004000] ACPI: Core revision 20221020
[    0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[    0.004000] APIC: Switch to symmetric I/O mode setup
[    0.004000] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[    0.004000] APIC: Done
[    0.004000] Before apic_bsb_setup
[    0.004000] check_timer begin
[    0.004000] check_timer after local_irq_disable
[    0.004000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.004000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x7070b6fd6fc, max_idle_ns: 881590611710 ns
[    0.144982] Calibrating delay loop (skipped), value calculated using timer frequency.. 7800.54 BogoMIPS (lpj=15601092)
[    0.144986] pid_max: default: 32768 minimum: 301
[    0.145079] LSM: initializing lsm=capability
[    0.145173] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145189] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145564] Bit 30 in CPUID ECX not set.
[    0.145587] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512
[    0.145589] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512, 1GB 0
[    0.145594] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.145597] Spectre V2 : Mitigation: Retpolines
[    0.145598] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.145599] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.145600] Spectre V2 : Enabling Speculation Barrier for firmware calls
[    0.145600] RETBleed: Mitigation: untrained return thunk
[    0.145602] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.145604] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[    0.150046] Freeing SMP alternatives memory: 32K
[    0.150051] After check_bugs
[    0.150052] After acpi_subsystem_init
[    0.150053] After arch_post_acpi_subsys_init
[    0.150054] After rcu_scheduler_starting
[    0.150126] After find_task_by_pid_ns and PF_NO_SETAFFINITY
[    0.150131] After numa_default_policy
[    0.150151] After rcu_read_lock
[    0.150152] After rcu_read_unlock
[    0.150153] After kthreadd_done
[    0.150165] smpboot: Start of smp_prepare_cpus_common
[    0.150167] smpboot: smpboot: zalloc 0
[    0.150168] smpboot: smpboot: zalloc 1
[    0.150169] smpboot: smpboot: After set_sched_topology()
[    0.150171] smpboot: smpboot: After smp_sanity_check()
[    0.150172] smpboot: smpboot: Before x86_init.timers.setup_percpu_clockev()
[    0.258192] smpboot: smpboot: After x86_init.timers.setup_percpu_clockev()
[    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[    0.258425] cblist_init_generic: Setting adjustable number of callback queues.
[    0.258427] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258458] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258484] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258513] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    0.258536] ... version:                0
[    0.258537] ... bit width:              48
[    0.258538] ... generic registers:      6
[    0.258539] ... value mask:             0000ffffffffffff
[    0.258540] ... max period:             00007fffffffffff
[    0.258541] ... fixed-purpose events:   0
[    0.258542] ... event mask:             000000000000003f
[    0.258662] rcu: Hierarchical SRCU implementation.
[    0.258663] rcu: 	Max phase no-delay instances is 1000.
[    0.259257] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    0.259329] smp: Bringing up secondary CPUs ...
[    0.259527] x86: Booting SMP configuration:
[    0.259528] .... node  #0, CPUs:      #1
[    0.261007] After schedule_preempt_disabled
[   10.260990] CPU1 failed to report alive state
[   10.261070] smp: Brought up 1 node, 1 CPU
[   10.261073] smpboot: Max logical packages: 2
[   10.261074] smpboot: Total of 1 processors activated (7800.54 BogoMIPS)
[   10.261601] devtmpfs: initialized
[   10.261697] x86/mm: Memory block size: 128MB
[   10.262788] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[   10.262795] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[   10.262883] pinctrl core: initialized pinctrl subsystem
[   10.262953] PM: RTC time: 07:39:50, date: 2023-04-17
[   10.263709] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[   10.263937] audit: initializing netlink subsys (disabled)
[   10.264192] thermal_sys: Registered thermal governor 'fair_share'
[   10.264193] thermal_sys: Registered thermal governor 'bang_bang'
[   10.264194] thermal_sys: Registered thermal governor 'step_wise'
[   10.264195] thermal_sys: Registered thermal governor 'user_space'
[   10.264215] cpuidle: using governor ladder
[   10.264220] cpuidle: using governor menu
[   10.264427] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
[   10.264432] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved as E820 entry
[   10.264444] PCI: Using configuration type 1 for base access
[   10.264646] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[   10.269064] audit: type=2000 audit(1681717190.140:1): state=initialized audit_enabled=0 res=1
[   10.281080] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[   10.281084] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[   10.281085] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[   10.281086] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[   10.286166] cryptd: max_cpu_qlen set to 1000
[   10.289615] ACPI: Added _OSI(Module Device)
[   10.289617] ACPI: Added _OSI(Processor Device)
[   10.289618] ACPI: Added _OSI(3.0 _SCP Extensions)
[   10.289620] ACPI: Added _OSI(Processor Aggregator Device)
[   10.293942] ACPI: DSDT successfully acquired and loaded

[   10.295626] ACPI: 4 ACPI AML tables successfully acquired and loaded
[   10.297161] ACPI: Interpreter enabled
[   10.297184] ACPI: PM: (supports S0 S1 S3 S5)
[   10.297186] ACPI: Using IOAPIC for interrupt routing
[   10.297236] HEST: Table parsing has been initialized.
[   10.297258] GHES: Failed to enable APEI firmware first mode.
[   10.297261] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[   10.297262] PCI: Ignoring E820 reservations for host bridge windows
[   10.297510] ACPI: Enabled 8 GPEs in block 00 to 1F
[   10.303230] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[   10.303241] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[   10.303320] acpi PNP0A03:00: _OSC: OS now controls [PME AER PCIeCapability LTR]
[   10.303335] acpi PNP0A03:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge
[   10.303414] acpi PNP0A03:00: host bridge window expanded to [io  0x0000-0x0cf7 window]; [io  0x03b0-0x03df window] ignored
[   10.303651] PCI host bridge to bus 0000:00
[   10.303653] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[   10.303655] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[   10.303658] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000dffff]
[   10.303660] pci_bus 0000:00: root bus resource [mem 0x80000000-0xffffffff]
[   10.303662] pci_bus 0000:00: root bus resource [bus 00-ff]
[   10.303686] pci 0000:00:00.0: [1022:1410] type 00 class 0x060000
[   10.303830] pci 0000:00:00.2: [1022:1419] type 00 class 0x080600
[   10.303921] pci 0000:00:01.0: [1002:9996] type 00 class 0x030000
[   10.303929] pci 0000:00:01.0: reg 0x10: [mem 0xe0000000-0xefffffff pref]
[   10.303934] pci 0000:00:01.0: reg 0x14: [io  0x1000-0x10ff]
[   10.303939] pci 0000:00:01.0: reg 0x18: [mem 0xf0180000-0xf01bffff]
[   10.303955] pci 0000:00:01.0: enabling Extended Tags
[   10.303965] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[   10.303983] pci 0000:00:01.0: supports D1 D2
[   10.304048] pci 0000:00:01.1: [1002:9902] type 00 class 0x040300
[   10.304056] pci 0000:00:01.1: reg 0x10: [mem 0xf01c0000-0xf01c3fff]
[   10.304077] pci 0000:00:01.1: enabling Extended Tags
[   10.304101] pci 0000:00:01.1: supports D1 D2
[   10.304185] pci 0000:00:11.0: [1022:7801] type 00 class 0x010601
[   10.304198] pci 0000:00:11.0: reg 0x10: [io  0x1410-0x1417]
[   10.304206] pci 0000:00:11.0: reg 0x14: [io  0x1420-0x1423]
[   10.304213] pci 0000:00:11.0: reg 0x18: [io  0x1418-0x141f]
[   10.304221] pci 0000:00:11.0: reg 0x1c: [io  0x1424-0x1427]
[   10.304228] pci 0000:00:11.0: reg 0x20: [io  0x1400-0x140f]
[   10.304235] pci 0000:00:11.0: reg 0x24: [mem 0xf01cc000-0xf01cc7ff]
[   10.304392] pci 0000:00:12.0: [1022:7807] type 00 class 0x0c0310
[   10.304406] pci 0000:00:12.0: reg 0x10: [mem 0xf01c8000-0xf01c8fff]
[   10.304587] pci 0000:00:12.2: [1022:7808] type 00 class 0x0c0320
[   10.304601] pci 0000:00:12.2: reg 0x10: [mem 0xf01cd000-0xf01cd0ff]
[   10.304666] pci 0000:00:12.2: supports D1 D2
[   10.304667] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[   10.304668] pci 0000:00:12.2: pme_poll = true
[   10.304669] pci 0000:00:12.2: after device_set_wakeup_capable()
[   10.304673] pci 0000:00:12.2: after pci_pme_active()
[   10.304814] pci 0000:00:13.0: [1022:7807] type 00 class 0x0c0310
[   10.304827] pci 0000:00:13.0: reg 0x10: [mem 0xf01c9000-0xf01c9fff]
[   10.305566] pci 0000:00:13.2: [1022:7808] type 00 class 0x0c0320
[   10.305583] pci 0000:00:13.2: reg 0x10: [mem 0xf01ce000-0xf01ce0ff]
[   10.305654] pci 0000:00:13.2: supports D1 D2
[   10.305656] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[   10.305657] pci 0000:00:13.2: pme_poll = true
[   10.305659] pci 0000:00:13.2: after device_set_wakeup_capable()
[   10.305662] pci 0000:00:13.2: after pci_pme_active()
[   10.305803] pci 0000:00:14.0: [1022:780b] type 00 class 0x0c0500
[   10.305975] pci 0000:00:14.2: [1022:780d] type 00 class 0x040300
[   10.305992] pci 0000:00:14.2: reg 0x10: [mem 0xf01c4000-0xf01c7fff 64bit]
[   10.306047] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[   10.306049] pci 0000:00:14.2: pme_poll = true
[   10.306050] pci 0000:00:14.2: after device_set_wakeup_capable()
[   10.306052] pci 0000:00:14.2: after pci_pme_active()
[   10.306185] pci 0000:00:14.3: [1022:780e] type 00 class 0x060100
[   10.306360] pci 0000:00:14.4: [1022:780f] type 01 class 0x060401
[   10.306502] pci 0000:00:14.5: [1022:7809] type 00 class 0x0c0310
[   10.306515] pci 0000:00:14.5: reg 0x10: [mem 0xf01ca000-0xf01cafff]
[   10.306682] pci 0000:00:15.0: [1022:43a0] type 01 class 0x060400
[   10.306711] pci 0000:00:15.0: enabling Extended Tags
[   10.306751] pci 0000:00:15.0: supports D1 D2
[   10.306911] pci 0000:00:15.1: [1022:43a1] type 01 class 0x060400
[   10.306942] pci 0000:00:15.1: enabling Extended Tags
[   10.306981] pci 0000:00:15.1: supports D1 D2
[   10.307137] pci 0000:00:15.2: [1022:43a2] type 01 class 0x060400
[   10.307166] pci 0000:00:15.2: enabling Extended Tags
[   10.307205] pci 0000:00:15.2: supports D1 D2
[   10.307280] pci 0000:00:16.0: [1022:7807] type 00 class 0x0c0310
[   10.307293] pci 0000:00:16.0: reg 0x10: [mem 0xf01cb000-0xf01cbfff]
[   10.307467] pci 0000:00:16.2: [1022:7808] type 00 class 0x0c0320
[   10.307481] pci 0000:00:16.2: reg 0x10: [mem 0xf01cf000-0xf01cf0ff]
[   10.307546] pci 0000:00:16.2: supports D1 D2
[   10.307547] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[   10.307548] pci 0000:00:16.2: pme_poll = true
[   10.307549] pci 0000:00:16.2: after device_set_wakeup_capable()
[   10.307552] pci 0000:00:16.2: after pci_pme_active()
[   10.307691] pci 0000:00:18.0: [1022:1400] type 00 class 0x060000
[   10.307758] pci 0000:00:18.1: [1022:1401] type 00 class 0x060000
[   10.307819] pci 0000:00:18.2: [1022:1402] type 00 class 0x060000
[   10.307880] pci 0000:00:18.3: [1022:1403] type 00 class 0x060000
[   10.308016] pci 0000:00:18.4: [1022:1404] type 00 class 0x060000
[   10.308079] pci 0000:00:18.5: [1022:1405] type 00 class 0x060000
[   10.308153] pci_bus 0000:01: extended config space not accessible
[   10.308218] pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
[   10.308227] pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
[   10.308230] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff window] (subtractive decode)
[   10.308232] pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000dffff] (subtractive decode)
[   10.308235] pci 0000:00:14.4:   bridge window [mem 0x80000000-0xffffffff] (subtractive decode)
[   10.308283] pci 0000:00:15.0: PCI bridge to [bus 02]
[   10.308368] pci 0000:03:00.0: [1b21:1042] type 00 class 0x0c0330
[   10.308404] pci 0000:03:00.0: reg 0x10: [mem 0xf0000000-0xf0007fff 64bit]
[   10.308579] pci 0000:03:00.0: PME# supported from D3hot D3cold
[   10.308581] pci 0000:03:00.0: pme_poll = true
[   10.308582] pci 0000:03:00.0: after device_set_wakeup_capable()
[   10.308587] pci 0000:03:00.0: after pci_pme_active()
[   10.308625] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:15.1 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[   10.317133] pci 0000:00:15.1: PCI bridge to [bus 03]
[   10.317145] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[   10.317154] pci 0000:00:15.2: bridge configuration invalid ([bus 00-00]), reconfiguring
[   10.317278] pci 0000:04:00.0: [10ec:8168] type 00 class 0x020000
[   10.317296] pci 0000:04:00.0: reg 0x10: [io  0x0000-0x00ff]
[   10.317318] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit pref]
[   10.317332] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[   10.317440] pci 0000:04:00.0: supports D1 D2
[   10.317442] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   10.317444] pci 0000:04:00.0: pme_poll = true
[   10.317445] pci 0000:04:00.0: after device_set_wakeup_capable()
[   10.317449] pci 0000:04:00.0: after pci_pme_active()
[   10.329042] pci 0000:00:15.2: PCI bridge to [bus 04-ff]
[   10.329053] pci 0000:00:15.2:   bridge window [io  0x0000-0x0fff]
[   10.329057] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff]
[   10.329062] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[   10.329065] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[   10.329575] ACPI: PCI: Interrupt link INTA configured for IRQ 0
[   10.329669] ACPI: PCI: Interrupt link INTB configured for IRQ 0
[   10.329759] ACPI: PCI: Interrupt link INTC configured for IRQ 0
[   10.329849] ACPI: PCI: Interrupt link INTD configured for IRQ 0
[   10.329940] ACPI: PCI: Interrupt link INTE configured for IRQ 0
[   10.330029] ACPI: PCI: Interrupt link INTF configured for IRQ 0
[   10.330119] ACPI: PCI: Interrupt link INTG configured for IRQ 0
[   10.330210] ACPI: PCI: Interrupt link INTH configured for IRQ 0
[   10.330444] iommu: Default domain type: Translated 
[   10.330446] iommu: DMA domain TLB invalidation policy: lazy mode 
[   10.330624] SCSI subsystem initialized
[   10.330718] libata version 3.00 loaded.
[   10.330752] ACPI: bus type USB registered
[   10.330775] usbcore: registered new interface driver usbfs
[   10.330786] usbcore: registered new interface driver hub
[   10.330798] usbcore: registered new device driver usb
[   10.331125] PCI: Using ACPI for IRQ routing
[   10.332694] PCI: pci_cache_line_size set to 64 bytes
[   10.332745] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[   10.332748] e820: reserve RAM buffer [mem 0x5fe4d000-0x5fffffff]
[   10.332750] e820: reserve RAM buffer [mem 0x17f000000-0x17fffffff]
[   10.332794] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[   10.332799] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[   10.334067] clocksource: Switched to clocksource tsc-early
[   10.350705] VFS: Disk quotas dquot_6.6.0
[   10.350734] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[   10.350847] pnp: PnP ACPI init
[   10.351137] system 00:00: [mem 0xfec10002-0xfec11001] could not be reserved
[   10.351446] pnp: PnP ACPI: found 2 devices
[   10.358408] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[   10.358544] NET: Registered PF_INET protocol family
[   10.358686] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[   10.360256] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[   10.360271] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[   10.360277] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[   10.360344] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[   10.360670] TCP: Hash tables configured (established 32768 bind 32768)
[   10.360739] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[   10.360763] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[   10.360870] NET: Registered PF_UNIX/PF_LOCAL protocol family
[   10.360906] pci 0000:00:15.2: BAR 15: assigned [mem 0x80000000-0x800fffff 64bit pref]
[   10.360912] pci 0000:00:15.2: BAR 13: assigned [io  0x2000-0x2fff]
[   10.360917] pci 0000:00:14.4: PCI bridge to [bus 01]
[   10.360928] pci 0000:00:15.0: PCI bridge to [bus 02]
[   10.360936] pci 0000:00:15.1: PCI bridge to [bus 03]
[   10.360939] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[   10.360947] pci 0000:04:00.0: BAR 4: assigned [mem 0x80000000-0x80003fff 64bit pref]
[   10.360959] pci 0000:04:00.0: BAR 2: assigned [mem 0x80004000-0x80004fff 64bit pref]
[   10.360971] pci 0000:04:00.0: BAR 0: assigned [io  0x2000-0x20ff]
[   10.360976] pci 0000:00:15.2: PCI bridge to [bus 04]
[   10.361116] pci 0000:00:15.2:   bridge window [io  0x2000-0x2fff]
[   10.361122] pci 0000:00:15.2:   bridge window [mem 0x80000000-0x800fffff 64bit pref]
[   10.361128] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[   10.361130] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[   10.361132] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000dffff]
[   10.361133] pci_bus 0000:00: resource 7 [mem 0x80000000-0xffffffff]
[   10.361135] pci_bus 0000:01: resource 4 [io  0x0000-0x0cf7 window]
[   10.361137] pci_bus 0000:01: resource 5 [io  0x0d00-0xffff window]
[   10.361138] pci_bus 0000:01: resource 6 [mem 0x000a0000-0x000dffff]
[   10.361140] pci_bus 0000:01: resource 7 [mem 0x80000000-0xffffffff]
[   10.361141] pci_bus 0000:03: resource 1 [mem 0xf0000000-0xf00fffff]
[   10.361143] pci_bus 0000:04: resource 0 [io  0x2000-0x2fff]
[   10.361144] pci_bus 0000:04: resource 2 [mem 0x80000000-0x800fffff 64bit pref]
[   10.361239] pci 0000:00:01.1: D0 power state depends on 0000:00:01.0
[   10.361456] pci 0000:00:12.0: AMD USB device
[   10.361478] pci 0000:00:12.0: AMD USB ohci handoff
[   10.361761] pci 0000:00:12.2: AMD USB device
[   10.361775] pci 0000:00:12.2: AMD USB ehci handoff
[   10.361903] pci 0000:00:12.2: PME# does not work under D3, disabling it
[   10.362049] pci 0000:00:13.0: AMD USB device
[   10.362062] pci 0000:00:13.0: AMD USB ohci handoff
[   10.362328] pci 0000:00:13.2: AMD USB device
[   10.362339] pci 0000:00:13.2: AMD USB ehci handoff
[   10.362469] pci 0000:00:13.2: PME# does not work under D3, disabling it
[   10.362622] pci 0000:00:14.5: AMD USB device
[   10.362634] pci 0000:00:14.5: AMD USB ohci handoff
[   10.362915] pci 0000:00:16.0: AMD USB device
[   10.362928] pci 0000:00:16.0: AMD USB ohci handoff
[   10.363195] pci 0000:00:16.2: AMD USB device
[   10.363206] pci 0000:00:16.2: AMD USB ehci handoff
[   10.363334] pci 0000:00:16.2: PME# does not work under D3, disabling it
[   10.363561] pci 0000:03:00.0: AMD USB xhci handoff
[   10.363610] PCI: CLS 64 bytes, default 64
[   10.363723] pci 0000:00:00.2: AMD-Vi: Applying erratum 746 workaround
[   10.363810] pci 0000:00:01.0: Adding to iommu group 0
[   10.363826] pci 0000:00:01.1: Adding to iommu group 0
[   10.363851] pci 0000:00:11.0: Adding to iommu group 1
[   10.363886] pci 0000:00:12.0: Adding to iommu group 2
[   10.363903] pci 0000:00:12.2: Adding to iommu group 2
[   10.363937] pci 0000:00:13.0: Adding to iommu group 3
[   10.363953] pci 0000:00:13.2: Adding to iommu group 3
[   10.363991] pci 0000:00:14.0: Adding to iommu group 4
[   10.364009] pci 0000:00:14.2: Adding to iommu group 4
[   10.364025] pci 0000:00:14.3: Adding to iommu group 4
[   10.364048] pci 0000:00:14.4: Adding to iommu group 5
[   10.364070] pci 0000:00:14.5: Adding to iommu group 6
[   10.364104] pci 0000:00:15.0: Adding to iommu group 7
[   10.364123] pci 0000:00:15.1: Adding to iommu group 7
[   10.364139] pci 0000:00:15.2: Adding to iommu group 7
[   10.364178] pci 0000:00:16.0: Adding to iommu group 8
[   10.364194] pci 0000:00:16.2: Adding to iommu group 8
[   10.364246] pci 0000:00:18.0: Adding to iommu group 9
[   10.364266] pci 0000:00:18.1: Adding to iommu group 9
[   10.364283] pci 0000:00:18.2: Adding to iommu group 9
[   10.364303] pci 0000:00:18.3: Adding to iommu group 9
[   10.364321] pci 0000:00:18.4: Adding to iommu group 9
[   10.364340] pci 0000:00:18.5: Adding to iommu group 9
[   10.364352] pci 0000:03:00.0: Adding to iommu group 7
[   10.364360] pci 0000:04:00.0: Adding to iommu group 7
[   10.366500] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[   10.366505] AMD-Vi: Extended features (0x800000853, 0x0): PreF PPR GT IA
[   10.366510] AMD-Vi: Interrupt remapping enabled
[   10.366691] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[   10.366693] software IO TLB: mapped [mem 0x000000005be4d000-0x000000005fe4d000] (64MB)
[   10.366742] LVT offset 0 assigned for vector 0x400
[   10.366763] perf: AMD IBS detected (0x000000ff)
[   10.366771] amd_uncore: 4  amd_nb counters detected
[   10.367574] workingset: timestamp_bits=37 max_order=20 bucket_order=0
[   10.367603] zbud: loaded
[   10.368066] NET: Registered PF_ALG protocol family
[   10.368071] Key type asymmetric registered
[   10.368073] Asymmetric key parser 'x509' registered
[   10.368346] alg: self-tests disabled
[   10.368439] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[   10.368477] io scheduler mq-deadline registered
[   10.368479] io scheduler kyber registered
[   10.370011] pcieport 0000:00:15.0: PME: Signaling with IRQ 25
[   10.370174] pcieport 0000:00:15.1: PME: Signaling with IRQ 26
[   10.370247] pcieport 0000:00:15.2: enabling device (0000 -> 0003)
[   10.370449] pcieport 0000:00:15.2: PME: Signaling with IRQ 27
[   10.370706] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[   10.370767] ACPI: button: Power Button [PWRF]
[   10.370823] ACPI: \_SB_.P000: Found 2 idle states
[   10.370937] ACPI: \_SB_.P001: Found 2 idle states
[   10.371827] thermal LNXTHERM:00: registered as thermal_zone0
[   10.371830] ACPI: thermal: Thermal Zone [TZ00] (0 C)
[   10.372146] Non-volatile memory driver v1.3
[   10.372218] AMD-Vi: AMD IOMMUv2 loaded and initialized
[   10.372338] ahci 0000:00:11.0: version 3.0
[   10.372612] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 8 ports 6 Gbps 0x40 impl SATA mode
[   10.372616] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck led clo pio 
[   10.374133] scsi host0: ahci
[   10.374333] scsi host1: ahci
[   10.374510] scsi host2: ahci
[   10.374710] scsi host3: ahci
[   10.374879] scsi host4: ahci
[   10.375057] scsi host5: ahci
[   10.375241] scsi host6: ahci
[   10.375421] scsi host7: ahci
[   10.375508] ata port1: DUMMY
[   10.375510] ata port2: DUMMY
[   10.375511] ata port3: DUMMY
[   10.375512] ata port4: DUMMY
[   10.375514] ata port5: DUMMY
[   10.375515] ata port6: DUMMY
[   10.375517] ata port7: SATA max UDMA/133 abar m2048@0xf01cc000 port 0xf01cc400 irq 19
[   10.375519] ata port8: DUMMY
[   10.375597] ACPI: bus type drm_connector registered
[   10.375823] i8042: PNP: No PS/2 controller found.
[   10.375824] i8042: Probing ports directly.
[   10.378675] serio: i8042 KBD port at 0x60,0x64 irq 1
[   10.378751] serio: i8042 AUX port at 0x60,0x64 irq 12
[   10.378874] mousedev: PS/2 mouse device common for all mice
[   10.378927] rtc_cmos 00:01: RTC can wake from S4
[   10.379173] rtc_cmos 00:01: registered as rtc0
[   10.379197] rtc_cmos 00:01: setting system clock to 2023-04-17T07:39:50 UTC (1681717190)
[   10.379234] rtc_cmos 00:01: alarms up to one day, y3k, 114 bytes nvram, hpet irqs
[   10.379268] device-mapper: uevent: version 1.0.3
[   10.379337] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com
[   10.379496] hid: raw HID events driver (C) Jiri Kosina
[   10.379531] usbcore: registered new interface driver usbhid
[   10.379532] usbhid: USB HID core driver
[   10.379638] Initializing XFRM netlink socket
[   10.379648] NET: Registered PF_PACKET protocol family
[   10.379650] x86/pm: family 0x15 cpu detected, MSR saving is needed during suspending.
[   10.379817] microcode: CPU0: patch_level=0x0600111f
[   10.379827] microcode: Microcode Update Driver: v2.2.
[   10.379831] IPI shorthand broadcast: enabled
[   10.379838] AVX version of gcm_enc/dec engaged.
[   10.379854] AES CTR mode by8 optimization enabled
[   10.380695] ata link7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   10.380969] ata dev7.0: ATA-9: SanDisk SDSSDP064G, 2.0.0, max UDMA/133
[   10.380972] ata dev7.0: 125045424 sectors, multi 1: LBA48 NCQ (depth 32)
[   10.381178] ata dev7.0: configured for UDMA/133
[   10.381289] scsi 6:0:0:0: Direct-Access     ATA      SanDisk SDSSDP06 0    PQ: 0 ANSI: 5
[   10.381718] sd 6:0:0:0: [sda] 125045424 512-byte logical blocks: (64.0 GB/59.6 GiB)
[   10.381730] sd 6:0:0:0: [sda] Write Protect is off
[   10.381733] sd 6:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   10.381749] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.381775] sd 6:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[   10.382547]  sda: sda1 sda2 sda3
[   10.382736] sd 6:0:0:0: [sda] Attached SCSI disk
[   10.385344] sched_clock: Marking stable (10268007974, 116977866)->(10387423156, -2437316)
[   10.385519] registered taskstats version 1
[   10.385734] zswap: loaded using pool lzo/zbud
[   10.390028] kmemleak: Kernel memory leak detector initialized (mem pool available: 15677)
[   10.390033] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[   10.393894] kmemleak: Automatic memory scanning thread started
[   10.394488] Key type encrypted registered
[   10.397453] PM:   Magic number: 3:139:673
[   10.397466] workqueue scsi_tmf_3: hash matches
[   10.410334] EXT4-fs (sda3): mounted filesystem fe29e0dc-6303-4401-987c-8472bc1b9516 with ordered data mode. Quota mode: none.
[   10.410378] VFS: Mounted root (ext4 filesystem) on device 8:3.
[   10.412268] devtmpfs: mounted
[   10.412286] After kernel_init_freeable
[   10.416793] Freeing unused kernel image (initmem) memory: 2908K
[   10.421291] Write protecting the kernel read-only data: 20480k
[   10.421559] Freeing unused kernel image (rodata/data gap) memory: 836K
[   10.458614] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[   10.458620] rodata_test: all tests were successful
[   10.458621] After mark_readonly
[   10.458621] After pti_finalize
[   10.458637] rcu_end_inkernel_boot
[   10.458644] Run /sbin/init as init process
[   10.458646]   with arguments:
[   10.458648]     /sbin/init
[   10.458649]     noisapnp
[   10.458650]   with environment:
[   10.458650]     HOME=/
[   10.458651]     TERM=linux
[   10.458652]     BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66
[   10.637190] systemd[1]: Inserted module 'autofs4'
[   10.663625] NET: Registered PF_INET6 protocol family
[   10.664473] Segment Routing with IPv6
[   10.664500] In-situ OAM (IOAM) with IPv6
[   10.691376] systemd[1]: systemd 252.6-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   10.691387] systemd[1]: Detected architecture x86-64.
[   10.696468] systemd[1]: Hostname set to <kodi>.
[   11.000700] systemd[1]: Queued start job for default target graphical.target.
[   11.011651] systemd[1]: Created slice system-getty.slice - Slice /system/getty.
[   11.012747] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[   11.013600] systemd[1]: Created slice user.slice - User and Session Slice.
[   11.013784] systemd[1]: Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[   11.013902] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[   11.014324] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[   11.014366] systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[   11.014416] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[   11.014455] systemd[1]: Reached target paths.target - Path Units.
[   11.014485] systemd[1]: Reached target remote-fs.target - Remote File Systems.
[   11.014515] systemd[1]: Reached target slices.target - Slice Units.
[   11.014554] systemd[1]: Reached target swap.target - Swaps.
[   11.014592] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[   11.017065] systemd[1]: Listening on systemd-coredump.socket - Process Core Dump Socket.
[   11.017317] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[   11.017479] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[   11.017786] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[   11.018059] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[   11.018337] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[   11.018594] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[   11.019420] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[   11.019693] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[   11.022504] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[   11.025189] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[   11.029590] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[   11.046849] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[   11.053730] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[   11.066015] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[   11.069414] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[   11.076547] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[   11.090852] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[   11.097815] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[   11.109388] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
[   11.109456] systemd[1]: systemd-firstboot.service - First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[   11.109525] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathIsReadWrite=!/).
[   11.109562] systemd[1]: Reached target local-fs.target - Local File Systems.
[   11.109629] systemd[1]: apparmor.service - Load AppArmor profiles was skipped because of an unmet condition check (ConditionSecurity=apparmor).
[   11.120810] loop: module loaded
[   11.122003] systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[   11.129598] fuse: init (API version 7.38)
[   11.137426] systemd[1]: Starting systemd-journald.service - Journal Service...
[   11.145752] systemd[1]: Starting systemd-random-seed.service - Load/Save Random Seed...
[   11.157494] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[   11.172634] systemd[1]: Starting systemd-sysusers.service - Create System Users...
[   11.197939] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[   11.216247] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[   11.216451] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[   11.216611] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[   11.216772] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[   11.233481] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[   11.234279] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[   11.238519] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[   11.239177] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[   11.247227] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[   11.247980] systemd[1]: modprobe@drm.service: Deactivated successfully.
[   11.253523] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[   11.254233] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[   11.255540] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[   11.256109] systemd[1]: modprobe@fuse.service: Deactivated successfully.
[   11.261747] systemd[1]: Finished modprobe@fuse.service - Load Kernel Module fuse.
[   11.262375] systemd[1]: modprobe@loop.service: Deactivated successfully.
[   11.265991] systemd[1]: Finished modprobe@loop.service - Load Kernel Module loop.
[   11.267213] systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.
[   11.268186] systemd[1]: Finished systemd-sysusers.service - Create System Users.
[   11.268690] systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 133 (systemd-binfmt)
[   11.282799] systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[   11.309441] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[   11.358583] systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
[   11.358687] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[   11.358816] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[   11.377438] systemd[1]: Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[   11.378820] systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[   11.385019] tsc: Refined TSC clocksource calibration: 3900.223 MHz
[   11.385026] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x70705a6472c, max_idle_ns: 881590586812 ns
[   11.385038] clocksource: Switched to clocksource tsc
[   11.397465] systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[   11.397822] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[   11.397985] systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
[   11.448710] systemd[1]: Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
[   11.457254] systemd[1]: Starting systemd-udevd.service - Rule-based Manager for Device Events and Files...
[   11.509082] systemd[1]: Started systemd-journald.service - Journal Service.
[   11.569863] systemd-journald[134]: Received client request to flush runtime journal.
[   12.053221] sd 6:0:0:0: Attached scsi generic sg0 type 0
[   12.140997] random: crng init done
[   12.367954] acpi_cpufreq: overriding BIOS provided _PSD data
[   12.510747] QUIRK: Enable AMD PLL fix
[   12.510804] ehci-pci 0000:00:12.2: EHCI Host Controller
[   12.510833] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 1
[   12.510844] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   12.510853] ehci-pci 0000:00:12.2: debug port 1
[   12.511023] ehci-pci 0000:00:12.2: irq 17, io mem 0xf01cd000
[   12.515767] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[   12.515773] piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection
[   12.516241] piix4_smbus 0000:00:14.0: Auxiliary SMBus Host Controller at 0xb20
[   12.525014] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00
[   12.525397] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[   12.525400] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.525402] usb usb1: Product: EHCI Host Controller
[   12.525404] usb usb1: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[   12.525405] usb usb1: SerialNumber: 0000:00:12.2
[   12.525862] hub 1-0:1.0: USB hub found
[   12.525889] hub 1-0:1.0: 5 ports detected
[   12.526592] ehci-pci 0000:00:13.2: EHCI Host Controller
[   12.526612] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 2
[   12.526623] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   12.526632] ehci-pci 0000:00:13.2: debug port 1
[   12.526767] ehci-pci 0000:00:13.2: irq 17, io mem 0xf01ce000
[   12.541014] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00
[   12.541263] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[   12.541266] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.541268] usb usb2: Product: EHCI Host Controller
[   12.541270] usb usb2: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[   12.541271] usb usb2: SerialNumber: 0000:00:13.2
[   12.541718] hub 2-0:1.0: USB hub found
[   12.541746] hub 2-0:1.0: 5 ports detected
[   12.542418] ehci-pci 0000:00:16.2: EHCI Host Controller
[   12.542436] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 3
[   12.542447] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   12.542456] ehci-pci 0000:00:16.2: debug port 1
[   12.542584] ehci-pci 0000:00:16.2: irq 17, io mem 0xf01cf000
[   12.557023] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00
[   12.557405] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[   12.557408] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.557410] usb usb3: Product: EHCI Host Controller
[   12.557411] usb usb3: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[   12.557413] usb usb3: SerialNumber: 0000:00:16.2
[   12.557858] hub 3-0:1.0: USB hub found
[   12.557884] hub 3-0:1.0: 4 ports detected
[   12.558447] ohci-pci 0000:00:12.0: OHCI PCI host controller
[   12.558466] ohci-pci 0000:00:12.0: new USB bus registered, assigned bus number 4
[   12.558608] ohci-pci 0000:00:12.0: irq 18, io mem 0xf01c8000
[   12.558618] ohci-pci 0000:00:13.0: OHCI PCI host controller
[   12.558635] ohci-pci 0000:00:13.0: new USB bus registered, assigned bus number 5
[   12.558703] ohci-pci 0000:00:13.0: irq 18, io mem 0xf01c9000
[   12.558710] ohci-pci 0000:00:14.5: OHCI PCI host controller
[   12.558725] ohci-pci 0000:00:14.5: new USB bus registered, assigned bus number 6
[   12.558793] ohci-pci 0000:00:14.5: irq 18, io mem 0xf01ca000
[   12.558800] ohci-pci 0000:00:16.0: OHCI PCI host controller
[   12.558814] ohci-pci 0000:00:16.0: new USB bus registered, assigned bus number 7
[   12.558888] ohci-pci 0000:00:16.0: irq 18, io mem 0xf01cb000
[   12.635709] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[   12.635716] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.635718] usb usb7: Product: OHCI PCI host controller
[   12.635720] usb usb7: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[   12.635721] usb usb7: SerialNumber: 0000:00:16.0
[   12.636145] hub 7-0:1.0: USB hub found
[   12.636172] hub 7-0:1.0: 4 ports detected
[   12.636849] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[   12.636851] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.636853] usb usb5: Product: OHCI PCI host controller
[   12.636854] usb usb5: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[   12.636856] usb usb5: SerialNumber: 0000:00:13.0
[   12.639549] hub 5-0:1.0: USB hub found
[   12.639579] hub 5-0:1.0: 5 ports detected
[   12.640465] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[   12.640468] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.640470] usb usb4: Product: OHCI PCI host controller
[   12.640471] usb usb4: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[   12.640473] usb usb4: SerialNumber: 0000:00:12.0
[   12.640848] hub 4-0:1.0: USB hub found
[   12.640874] hub 4-0:1.0: 5 ports detected
[   12.644079] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[   12.644084] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.644086] usb usb6: Product: OHCI PCI host controller
[   12.644088] usb usb6: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[   12.644089] usb usb6: SerialNumber: 0000:00:14.5
[   12.644484] hub 6-0:1.0: USB hub found
[   12.644509] hub 6-0:1.0: 2 ports detected
[   12.658738] r8169 0000:04:00.0: enabling device (0000 -> 0003)
[   12.697540] xhci_hcd 0000:03:00.0: xHCI Host Controller
[   12.697566] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 8
[   12.709878] r8169 0000:04:00.0 eth0: RTL8168f/8111f, 08:60:6e:74:7a:51, XID 480, IRQ 28
[   12.709886] r8169 0000:04:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[   12.735756] 1
[   12.735806] 2
[   12.736276] snd_hda_intel 0000:00:01.1: Force to non-snoop mode
[   12.736282] 3
[   12.736283] 4
[   12.736284] 5
[   12.736284] 7
[   12.736288] 8
[   12.736288] 9
[   12.740797] 1
[   12.740846] 2
[   12.741376] 3
[   12.741377] 4
[   12.741378] 5
[   12.741379] 7
[   12.741382] 8
[   12.741383] 9
[   12.769260] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/sound/card0/input1
[   12.769555] input: HDA ATI HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.1/sound/card0/input2
[   12.784439] snd_hda_codec_realtek hdaudioC1D0: ALC892: SKU not ready 0x00000100
[   12.785682] snd_hda_codec_realtek hdaudioC1D0: autoconfig for ALC892: line_outs=4 (0x14/0x16/0x15/0x17/0x0) type:line
[   12.785688] snd_hda_codec_realtek hdaudioC1D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   12.785691] snd_hda_codec_realtek hdaudioC1D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[   12.785693] snd_hda_codec_realtek hdaudioC1D0:    mono: mono_out=0x0
[   12.785694] snd_hda_codec_realtek hdaudioC1D0:    dig-out=0x1e/0x0
[   12.785696] snd_hda_codec_realtek hdaudioC1D0:    inputs:
[   12.785697] snd_hda_codec_realtek hdaudioC1D0:      Rear Mic=0x18
[   12.785699] snd_hda_codec_realtek hdaudioC1D0:      Front Mic=0x19
[   12.785700] snd_hda_codec_realtek hdaudioC1D0:      Line=0x1a
[   12.785701] snd_hda_codec_realtek hdaudioC1D0:      CD=0x1c
[   12.807528] xhci_hcd 0000:03:00.0: hcc params 0x0200f180 hci version 0x96 quirks 0x0000000000080010
[   12.813178] input: HD-Audio Generic Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input3
[   12.813514] input: HD-Audio Generic Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input4
[   12.813782] input: HD-Audio Generic Line as /devices/pci0000:00/0000:00:14.2/sound/card1/input5
[   12.814048] input: HD-Audio Generic Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card1/input6
[   12.814317] input: HD-Audio Generic Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card1/input7
[   12.814575] input: HD-Audio Generic Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card1/input8
[   12.814833] input: HD-Audio Generic Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card1/input9
[   12.815087] input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card1/input10
[   12.816233] xhci_hcd 0000:03:00.0: xHCI Host Controller
[   12.816249] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 9
[   12.816260] xhci_hcd 0000:03:00.0: Host supports USB 3.0 SuperSpeed
[   12.830985] usb usb8: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[   12.830993] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.830995] usb usb8: Product: xHCI Host Controller
[   12.830996] usb usb8: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 xhci-hcd
[   12.830998] usb usb8: SerialNumber: 0000:03:00.0
[   12.835543] hub 8-0:1.0: USB hub found
[   12.842296] hub 8-0:1.0: 2 ports detected
[   12.850842] usb usb9: We don't know the algorithms for LPM for this host, disabling LPM.
[   12.850983] usb usb9: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.03
[   12.850986] usb usb9: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.850988] usb usb9: Product: xHCI Host Controller
[   12.850989] usb usb9: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 xhci-hcd
[   12.850991] usb usb9: SerialNumber: 0000:03:00.0
[   12.856278] r8169 0000:04:00.0 enp4s0: renamed from eth0
[   12.862771] hub 9-0:1.0: USB hub found
[   12.873601] hub 9-0:1.0: 2 ports detected
[   13.025026] usb 4-1: new low-speed USB device number 2 using ohci-pci
[   13.103377] r8169 0000:04:00.0: Direct firmware load for rtl_nic/rtl8168f-1.fw failed with error -2
[   13.103389] r8169 0000:04:00.0: Unable to load firmware rtl_nic/rtl8168f-1.fw (-2)
[   13.103907] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
[   13.175505] r8169 0000:04:00.0 enp4s0: Link is Down
[   13.179870] [drm] radeon kernel modesetting enabled.
[   13.181586] [drm] initializing kernel modesetting (ARUBA 0x1002:0x9996 0x1002:0x9996 0x00).
[   13.181652] ATOM BIOS: 113
[   13.181757] radeon 0000:00:01.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
[   13.181761] radeon 0000:00:01.0: GTT: 1024M 0x0000000020000000 - 0x000000005FFFFFFF
[   13.181769] [drm] Detected VRAM RAM=512M, BAR=256M
[   13.181770] [drm] RAM width 64bits DDR
[   13.181947] [drm] radeon: 512M of VRAM memory ready
[   13.181952] [drm] radeon: 1024M of GTT memory ready.
[   13.181994] [drm] Loading ARUBA Microcode
[   13.190350] [drm] Internal thermal controller without fan control
[   13.191100] [drm] radeon: dpm initialized
[   13.196015] [drm] Found VCE firmware/feedback version 50.0.1 / 17!
[   13.196073] [drm] GART: num cpu pages 262144, num gpu pages 262144
[   13.234337] [drm] GART: Restore entries: num cpu pages 262144, num gpu pages 262144
[   13.237934] [drm] GART: Done restoring entries
[   13.237938] [drm] PCIE GART of 1024M enabled (table at 0x00000000001D6000).
[   13.238177] radeon 0000:00:01.0: WB enabled
[   13.238180] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00
[   13.238558] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 0x0000000000075a18
[   13.246045] usb 4-1: New USB device found, idVendor=413c, idProduct=2106, bcdDevice= 1.01
[   13.246050] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   13.246052] usb 4-1: Product: Dell QuietKey Keyboard
[   13.246054] usb 4-1: Manufacturer: DELL
[   13.253819] input: DELL Dell QuietKey Keyboard as /devices/pci0000:00/0000:00:12.0/usb4/4-1/4-1:1.0/0003:413C:2106.0001/input/input11
[   13.260936] radeon 0000:00:01.0: fence driver on ring 6 use gpu addr 0x0000000020000c18
[   13.260940] radeon 0000:00:01.0: fence driver on ring 7 use gpu addr 0x0000000020000c1c
[   13.260942] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 0x0000000020000c04
[   13.260944] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 0x0000000020000c08
[   13.260945] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 0x0000000020000c0c
[   13.260947] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 0x0000000020000c10
[   13.262964] radeon 0000:00:01.0: radeon: MSI limited to 32-bit
[   13.263151] radeon 0000:00:01.0: radeon: using MSI.
[   13.263222] [drm] radeon: irq initialized.
[   13.281648] [drm] ring test on 0 succeeded in 3 usecs
[   13.281658] [drm] ring test on 3 succeeded in 4 usecs
[   13.281665] [drm] ring test on 4 succeeded in 4 usecs
[   13.295659] [drm] ring test on 5 succeeded in 2 usecs
[   13.297656] [drm] UVD initialized successfully.
[   13.313663] hid-generic 0003:413C:2106.0001: input,hidraw0: USB HID v1.10 Keyboard [DELL Dell QuietKey Keyboard] on usb-0000:00:12.0-1/input0
[   13.446953] [drm] ring test on 6 succeeded in 18 usecs
[   13.446967] [drm] ring test on 7 succeeded in 3 usecs
[   13.446968] [drm] VCE initialized successfully.
[   13.447122] snd_hda_intel 0000:00:01.1: bound 0000:00:01.0 (ops radeon_audio_component_bind_ops [radeon])
[   13.447293] [drm] ib test on ring 0 succeeded in 0 usecs
[   13.447346] [drm] ib test on ring 3 succeeded in 0 usecs
[   13.447396] [drm] ib test on ring 4 succeeded in 0 usecs
[   13.465099] [drm] ib test on ring 5 succeeded
[   13.481132] [drm] ib test on ring 6 succeeded in 1 usecs
[   13.497085] [drm] ib test on ring 7 succeeded in 1 usecs
[   13.500056] [drm] Radeon Display Connectors
[   13.500060] [drm] Connector 0:
[   13.500061] [drm]   DP-1
[   13.500062] [drm]   HPD1
[   13.500062] [drm]   DDC: 0x6530 0x6530 0x6534 0x6534 0x6538 0x6538 0x653c 0x653c
[   13.500065] [drm]   Encoders:
[   13.500065] [drm]     DFP1: INTERNAL_UNIPHY2
[   13.500066] [drm] Connector 1:
[   13.500067] [drm]   VGA-1
[   13.500068] [drm]   HPD2
[   13.500069] [drm]   DDC: 0x6540 0x6540 0x6544 0x6544 0x6548 0x6548 0x654c 0x654c
[   13.500070] [drm]   Encoders:
[   13.500071] [drm]     CRT1: INTERNAL_UNIPHY2
[   13.500072] [drm]     CRT1: NUTMEG
[   13.500072] [drm] Connector 2:
[   13.500073] [drm]   HDMI-A-1
[   13.500074] [drm]   HPD3
[   13.500075] [drm]   DDC: 0x6550 0x6550 0x6554 0x6554 0x6558 0x6558 0x655c 0x655c
[   13.500076] [drm]   Encoders:
[   13.500077] [drm]     DFP2: INTERNAL_UNIPHY
[   13.741081] usb 4-2: new low-speed USB device number 3 using ohci-pci
[   13.779818] [drm] fb mappable at 0xE03E9000
[   13.779826] [drm] vram apper at 0xE0000000
[   13.779828] [drm] size 5242880
[   13.779830] [drm] fb depth is 24
[   13.779832] [drm]    pitch is 5120
[   13.780398] fbcon: radeondrmfb (fb0) is primary device
[   13.936158] usb 4-2: New USB device found, idVendor=046d, idProduct=c016, bcdDevice= 3.40
[   13.936167] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   13.936170] usb 4-2: Product: Optical USB Mouse
[   13.936173] usb 4-2: Manufacturer: Logitech
[   13.945345] input: Logitech Optical USB Mouse as /devices/pci0000:00/0000:00:12.0/usb4/4-2/4-2:1.0/0003:046D:C016.0002/input/input12
[   13.946178] hid-generic 0003:046D:C016.0002: input,hidraw1: USB HID v1.10 Mouse [Logitech Optical USB Mouse] on usb-0000:00:12.0-2/input0
[   13.968191] Console: switching to colour frame buffer device 160x64
[   13.973460] radeon 0000:00:01.0: [drm] fb0: radeondrmfb frame buffer device
[   13.981574] [drm] Initialized radeon 2.50.0 20080528 for 0000:00:01.0 on minor 0
[   15.876613] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
[   15.876628] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0: link becomes ready
[   16.713266] [drm] amdgpu kernel modesetting enabled.
[   17.110008] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=249 'systemd'
Paul Menzel April 17, 2023, 11:24 a.m. UTC | #5
[Correct David’s address]

Am 17.04.23 um 13:19 schrieb Paul Menzel:
> Dear Thomas,
> 
> 
> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
> 
>> This is a complete rework of the parallel bringup patch series (V17)
>>
>>      
>> https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@bytedance.com
>>
>> to address the issues which were discovered in review:
> 
> […]
> 
> Thank you very much for your rework.
> 
> I tested this on the ASUS F2A85-M PRO, and get a delay of ten seconds.
> 
> ```
> […]
> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD 
> Graphics (family: 0x15, model: 0x13, stepping: 0x1)
> […]
> [    0.259329] smp: Bringing up secondary CPUs ...
> [    0.259527] x86: Booting SMP configuration:
> [    0.259528] .... node  #0, CPUs:      #1
> [    0.261007] After schedule_preempt_disabled
> [   10.260990] CPU1 failed to report alive state
> [   10.261070] smp: Brought up 1 node, 1 CPU
> [   10.261073] smpboot: Max logical packages: 2
> [   10.261074] smpboot: Total of 1 processors activated (7800.54 BogoMIPS)
> [   10.261601] devtmpfs: initialized
> [   10.261697] x86/mm: Memory block size: 128MB
> ```
> 
> This delay has been there with v6.3-rc6-46-gde4664485abbc and some 
> custom (printk) patches on top and merging dwmw2/parallel-6.2-rc3-v16 
> into it. I only tested this. I think dwmw2/parallel-6.2-v17 failed to 
> build for me, when trying to merge it into Linus’ master version at that 
> time. I didn’t come around to report it, and you posted your rework, so 
> I am replying here.
> 
> I am going to try your branch directly in the next days, but just wanted 
> to report back already.
> 
> 
> Kind regards,
> 
> Paul
Thomas Gleixner April 17, 2023, 2:48 p.m. UTC | #6
Paul!

On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote:
> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD 
> Graphics (family: 0x15, model: 0x13, stepping: 0x1)
> […]
> [    0.259329] smp: Bringing up secondary CPUs ...
> [    0.259527] x86: Booting SMP configuration:
> [    0.259528] .... node  #0, CPUs:      #1
> [    0.261007] After schedule_preempt_disabled
> [   10.260990] CPU1 failed to report alive state

Weird. CPU1 fails to come up and report that it has reached the
synchronization point.

Does it work when you add cpuhp.parallel=off on the kernel command line?

Thanks,

        tglx
Paul Menzel April 17, 2023, 5:40 p.m. UTC | #7
Dear Thomas,


Am 17.04.23 um 16:48 schrieb Thomas Gleixner:

> On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote:
>> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
>> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD
>> Graphics (family: 0x15, model: 0x13, stepping: 0x1)
>> […]
>> [    0.259329] smp: Bringing up secondary CPUs ...
>> [    0.259527] x86: Booting SMP configuration:
>> [    0.259528] .... node  #0, CPUs:      #1
>> [    0.261007] After schedule_preempt_disabled
>> [   10.260990] CPU1 failed to report alive state
> 
> Weird. CPU1 fails to come up and report that it has reached the
> synchronization point.
> 
> Does it work when you add cpuhp.parallel=off on the kernel command line?

Yes, the ten seconds delay is gone with `cpuhp.parallel=off`.

There was a patch set in the past, that worked on that device. I think 
up to v4 it did *not* work at all and hung [1]. I need some days to 
collect the results again.


Kind regards,

Paul


[1]: 
https://lore.kernel.org/lkml/ab28d2ce-4a9c-387d-9eda-558045a0c35b@molgen.mpg.de/
[    0.000000] Linux version 6.3.0-rc6-00311-gde8224969f66 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #446 SMP PREEMPT_DYNAMIC Sat Apr 15 14:12:29 UTC 2023
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 cpuhp.parallel=off
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] signal: max sigframe size: 1776
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005fe4cfff] usable
[    0.000000] BIOS-e820: [mem 0x000000005fe4d000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000017effffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0.0 present.
[    0.000000] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-gb640ed51b2 04/17/2023
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Initial usec timer 9249065
[    0.000000] tsc: Detected 3899.954 MHz processor
[    0.000755] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000759] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x400000000
[    0.000768] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000938] last_pfn = 0x5fe4d max_arch_pfn = 0x400000000
[    0.004000] Using GB pages for direct mapping
[    0.004000] ACPI: Early table checksum verification disabled
[    0.004000] ACPI: RSDP 0x00000000000F6830 000024 (v02 COREv4)
[    0.004000] ACPI: XSDT 0x000000005FE5A0E0 000074 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: FACP 0x000000005FE5BBC0 000114 (v06 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: DSDT 0x000000005FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925)
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: SSDT 0x000000005FE5BCE0 00008A (v02 COREv4 COREBOOT 0000002A CORE 20200925)
[    0.004000] ACPI: MCFG 0x000000005FE5BD70 00003C (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: APIC 0x000000005FE5BDB0 000062 (v03 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HPET 0x000000005FE5BE20 000038 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HEST 0x000000005FE5BE60 0001D0 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: IVRS 0x000000005FE5C030 000070 (v02 AMD    AMDIOMMU 00000001 AMD  00000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C0A0 00051F (v02 AMD    ALIB     00000001 MSFT 04000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C5C0 0006B2 (v01 AMD    POWERNOW 00000001 AMD  00000001)
[    0.004000] ACPI: VFCT 0x000000005FE5CC80 00F269 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: Reserving FACP table memory at [mem 0x5fe5bbc0-0x5fe5bcd3]
[    0.004000] ACPI: Reserving DSDT table memory at [mem 0x5fe5a280-0x5fe5bbb9]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5bce0-0x5fe5bd69]
[    0.004000] ACPI: Reserving MCFG table memory at [mem 0x5fe5bd70-0x5fe5bdab]
[    0.004000] ACPI: Reserving APIC table memory at [mem 0x5fe5bdb0-0x5fe5be11]
[    0.004000] ACPI: Reserving HPET table memory at [mem 0x5fe5be20-0x5fe5be57]
[    0.004000] ACPI: Reserving HEST table memory at [mem 0x5fe5be60-0x5fe5c02f]
[    0.004000] ACPI: Reserving IVRS table memory at [mem 0x5fe5c030-0x5fe5c09f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c0a0-0x5fe5c5be]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c5c0-0x5fe5cc71]
[    0.004000] ACPI: Reserving VFCT table memory at [mem 0x5fe5cc80-0x5fe6bee8]
[    0.004000] No NUMA configuration found
[    0.004000] Faking a node at [mem 0x0000000000000000-0x000000017effffff]
[    0.004000] NODE_DATA(0) allocated [mem 0x17efe9000-0x17effffff]
[    0.004000] Zone ranges:
[    0.004000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.004000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.004000]   Normal   [mem 0x0000000100000000-0x000000017effffff]
[    0.004000]   Device   empty
[    0.004000] Movable zone start for each node
[    0.004000] Early memory node ranges
[    0.004000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.004000]   node   0: [mem 0x0000000000100000-0x000000005fe4cfff]
[    0.004000]   node   0: [mem 0x0000000100000000-0x000000017effffff]
[    0.004000] Initmem setup node 0 [mem 0x0000000000001000-0x000000017effffff]
[    0.004000] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.004000] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 435 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 4096 pages in unavailable ranges
[    0.004000] ACPI: PM-Timer IO Port: 0x818
[    0.004000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.004000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.004000] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.004000] ACPI: HPET id: 0x10228210 base: 0xfed00000
[    0.004000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.004000] smpboot: smpboot: XXX end of prefill_possible_map
[    0.004000] After prefill_possible_map
[    0.004000] After init_cpu_to_node
[    0.004000] After init_gi_nodes
[    0.004000] After io_apic_init_mappings
[    0.004000] After x86_init.hyper.guest_late_init
[    0.004000] [mem 0x80000000-0xf7ffffff] available for PCI devices
[    0.004000] After e820
[    0.004000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.004000] After unwind_init
[    0.004000] After setup_arch
[    0.004000] After setup_command_line
[    0.004000] After setup_nr_cpu_ids
[    0.004000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.004000] percpu: Embedded 55 pages/cpu s188328 r8192 d28760 u1048576
[    0.004000] pcpu-alloc: s188328 r8192 d28760 u1048576 alloc=1*2097152
[    0.004000] pcpu-alloc: [0] 0 1 
[    0.004000] After setup_per_cpu_areas
[    0.004000] After smp_perpare_boot_cpu
[    0.004000] After boot_cpu_hotplug_init
[    0.004000] Fallback order for Node 0: 0 
[    0.004000] Built 1 zonelists, mobility grouping on.  Total pages: 898451
[    0.004000] Policy zone: Normal
[    0.004000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 cpuhp.parallel=off
[    0.004000] Unknown kernel command line parameters "noisapnp BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66", will be passed to user space.
[    0.004000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.004000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.004000] stackdepot: allocating hash table via alloc_large_system_hash
[    0.004000] stackdepot hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] software IO TLB: area num 2.
[    0.004000] Memory: 3477168K/3651500K available (14336K kernel code, 2340K rwdata, 5308K rodata, 2908K init, 11060K bss, 174072K reserved, 0K cma-reserved)
[    0.004000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.004000] After mm_init
[    0.004000] After poking_init
[    0.004000] ftrace: allocating 38664 entries in 152 pages
[    0.004000] ftrace: allocated 152 pages with 3 groups
[    0.004000] Dynamic Preempt: full
[    0.004000] After sched_init
[    0.004000] rcu: Preemptible hierarchical RCU implementation.
[    0.004000] rcu: 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[    0.004000] 	Trampoline variant of Tasks RCU enabled.
[    0.004000] 	Rude variant of Tasks RCU enabled.
[    0.004000] 	Tracing variant of Tasks RCU enabled.
[    0.004000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.004000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.004000] After rcu_init
[    0.004000] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
[    0.004000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.004000] After random_init()
[    0.004000] After boot_init_stack_canary
[    0.004000] spurious 8259A interrupt: IRQ7.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] printk: console [tty0] enabled
[    0.004000] ACPI: Core revision 20221020
[    0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[    0.004000] APIC: Switch to symmetric I/O mode setup
[    0.004000] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[    0.004000] APIC: Done
[    0.004000] Before apic_bsb_setup
[    0.004000] check_timer begin
[    0.004000] check_timer after local_irq_disable
[    0.004000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.004000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x706e603bb55, max_idle_ns: 881590819133 ns
[    0.145156] Calibrating delay loop (skipped), value calculated using timer frequency.. 7799.90 BogoMIPS (lpj=15599816)
[    0.145160] pid_max: default: 32768 minimum: 301
[    0.145254] LSM: initializing lsm=capability
[    0.145348] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145365] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145738] Bit 30 in CPUID ECX not set.
[    0.145762] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512
[    0.145764] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512, 1GB 0
[    0.145769] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.145772] Spectre V2 : Mitigation: Retpolines
[    0.145773] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.145774] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.145774] Spectre V2 : Enabling Speculation Barrier for firmware calls
[    0.145775] RETBleed: Mitigation: untrained return thunk
[    0.145777] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.145779] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[    0.150244] Freeing SMP alternatives memory: 32K
[    0.150249] After check_bugs
[    0.150250] After acpi_subsystem_init
[    0.150251] After arch_post_acpi_subsys_init
[    0.150252] After rcu_scheduler_starting
[    0.150324] After find_task_by_pid_ns and PF_NO_SETAFFINITY
[    0.150329] After numa_default_policy
[    0.150349] After rcu_read_lock
[    0.150350] After rcu_read_unlock
[    0.150351] After kthreadd_done
[    0.150363] smpboot: Start of smp_prepare_cpus_common
[    0.150365] smpboot: smpboot: zalloc 0
[    0.150366] smpboot: smpboot: zalloc 1
[    0.150367] smpboot: smpboot: After set_sched_topology()
[    0.150369] smpboot: smpboot: After smp_sanity_check()
[    0.150369] smpboot: smpboot: Before x86_init.timers.setup_percpu_clockev()
[    0.258381] smpboot: smpboot: After x86_init.timers.setup_percpu_clockev()
[    0.258382] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[    0.258615] cblist_init_generic: Setting adjustable number of callback queues.
[    0.258618] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258648] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258675] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258703] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    0.258726] ... version:                0
[    0.258727] ... bit width:              48
[    0.258728] ... generic registers:      6
[    0.258729] ... value mask:             0000ffffffffffff
[    0.258730] ... max period:             00007fffffffffff
[    0.258731] ... fixed-purpose events:   0
[    0.258732] ... event mask:             000000000000003f
[    0.258852] rcu: Hierarchical SRCU implementation.
[    0.258853] rcu: 	Max phase no-delay instances is 1000.
[    0.259441] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    0.259515] smp: Bringing up secondary CPUs ...
[    0.259714] x86: Booting SMP configuration:
[    0.259715] .... node  #0, CPUs:      #1
[    0.121151] Bit 30 in CPUID ECX not set.
[    0.259844] After schedule_preempt_disabled
[    0.259849] smp: Brought up 1 node, 2 CPUs
[    0.259849] smpboot: Max logical packages: 1
[    0.259849] smpboot: Total of 2 processors activated (15599.81 BogoMIPS)
[    0.261311] devtmpfs: initialized
[    0.261311] x86/mm: Memory block size: 128MB
[    0.262220] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.262220] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.262220] pinctrl core: initialized pinctrl subsystem
[    0.262220] PM: RTC time: 17:24:09, date: 2023-04-17
[    0.262220] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.262434] audit: initializing netlink subsys (disabled)
[    0.262453] audit: type=2000 audit(1681752249.140:1): state=initialized audit_enabled=0 res=1
[    0.262453] thermal_sys: Registered thermal governor 'fair_share'
[    0.262453] thermal_sys: Registered thermal governor 'bang_bang'
[    0.262453] thermal_sys: Registered thermal governor 'step_wise'
[    0.262453] thermal_sys: Registered thermal governor 'user_space'
[    0.262453] cpuidle: using governor ladder
[    0.262453] cpuidle: using governor menu
[    0.262453] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
[    0.262453] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved as E820 entry
[    0.262453] PCI: Using configuration type 1 for base access
[    0.262453] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.273235] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.273235] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.273235] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.273235] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.275428] cryptd: max_cpu_qlen set to 1000
[    0.275428] ACPI: Added _OSI(Module Device)
[    0.275428] ACPI: Added _OSI(Processor Device)
[    0.275428] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.275428] ACPI: Added _OSI(Processor Aggregator Device)
[    0.280879] ACPI: DSDT successfully acquired and loaded

[    0.281154] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    0.281649] ACPI: Interpreter enabled
[    0.281669] ACPI: PM: (supports S0 S1 S3 S5)
[    0.281671] ACPI: Using IOAPIC for interrupt routing
[    0.281724] HEST: Table parsing has been initialized.
[    0.281743] GHES: Failed to enable APEI firmware first mode.
[    0.281745] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.281746] PCI: Ignoring E820 reservations for host bridge windows
[    0.282003] ACPI: Enabled 8 GPEs in block 00 to 1F
[    0.286735] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.286746] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.286838] acpi PNP0A03:00: _OSC: OS now controls [PME AER PCIeCapability LTR]
[    0.286852] acpi PNP0A03:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge
[    0.286942] acpi PNP0A03:00: host bridge window expanded to [io  0x0000-0x0cf7 window]; [io  0x03b0-0x03df window] ignored
[    0.287197] PCI host bridge to bus 0000:00
[    0.287199] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.287201] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.287203] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000dffff]
[    0.287205] pci_bus 0000:00: root bus resource [mem 0x80000000-0xffffffff]
[    0.287210] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.287234] pci 0000:00:00.0: [1022:1410] type 00 class 0x060000
[    0.287388] pci 0000:00:00.2: [1022:1419] type 00 class 0x080600
[    0.287479] pci 0000:00:01.0: [1002:9996] type 00 class 0x030000
[    0.287488] pci 0000:00:01.0: reg 0x10: [mem 0xe0000000-0xefffffff pref]
[    0.287493] pci 0000:00:01.0: reg 0x14: [io  0x1000-0x10ff]
[    0.287498] pci 0000:00:01.0: reg 0x18: [mem 0xf0180000-0xf01bffff]
[    0.287514] pci 0000:00:01.0: enabling Extended Tags
[    0.287524] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.287541] pci 0000:00:01.0: supports D1 D2
[    0.287611] pci 0000:00:01.1: [1002:9902] type 00 class 0x040300
[    0.287618] pci 0000:00:01.1: reg 0x10: [mem 0xf01c0000-0xf01c3fff]
[    0.287640] pci 0000:00:01.1: enabling Extended Tags
[    0.287663] pci 0000:00:01.1: supports D1 D2
[    0.287750] pci 0000:00:11.0: [1022:7801] type 00 class 0x010601
[    0.287763] pci 0000:00:11.0: reg 0x10: [io  0x1410-0x1417]
[    0.287771] pci 0000:00:11.0: reg 0x14: [io  0x1420-0x1423]
[    0.287778] pci 0000:00:11.0: reg 0x18: [io  0x1418-0x141f]
[    0.287785] pci 0000:00:11.0: reg 0x1c: [io  0x1424-0x1427]
[    0.287793] pci 0000:00:11.0: reg 0x20: [io  0x1400-0x140f]
[    0.287800] pci 0000:00:11.0: reg 0x24: [mem 0xf01cc000-0xf01cc7ff]
[    0.287954] pci 0000:00:12.0: [1022:7807] type 00 class 0x0c0310
[    0.287967] pci 0000:00:12.0: reg 0x10: [mem 0xf01c8000-0xf01c8fff]
[    0.288152] pci 0000:00:12.2: [1022:7808] type 00 class 0x0c0320
[    0.288165] pci 0000:00:12.2: reg 0x10: [mem 0xf01cd000-0xf01cd0ff]
[    0.288230] pci 0000:00:12.2: supports D1 D2
[    0.288231] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    0.288233] pci 0000:00:12.2: pme_poll = true
[    0.288234] pci 0000:00:12.2: after device_set_wakeup_capable()
[    0.288237] pci 0000:00:12.2: after pci_pme_active()
[    0.288373] pci 0000:00:13.0: [1022:7807] type 00 class 0x0c0310
[    0.288387] pci 0000:00:13.0: reg 0x10: [mem 0xf01c9000-0xf01c9fff]
[    0.288570] pci 0000:00:13.2: [1022:7808] type 00 class 0x0c0320
[    0.288583] pci 0000:00:13.2: reg 0x10: [mem 0xf01ce000-0xf01ce0ff]
[    0.288650] pci 0000:00:13.2: supports D1 D2
[    0.288652] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    0.288653] pci 0000:00:13.2: pme_poll = true
[    0.288654] pci 0000:00:13.2: after device_set_wakeup_capable()
[    0.288657] pci 0000:00:13.2: after pci_pme_active()
[    0.288798] pci 0000:00:14.0: [1022:780b] type 00 class 0x0c0500
[    0.288980] pci 0000:00:14.2: [1022:780d] type 00 class 0x040300
[    0.288997] pci 0000:00:14.2: reg 0x10: [mem 0xf01c4000-0xf01c7fff 64bit]
[    0.289052] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[    0.289054] pci 0000:00:14.2: pme_poll = true
[    0.289055] pci 0000:00:14.2: after device_set_wakeup_capable()
[    0.289058] pci 0000:00:14.2: after pci_pme_active()
[    0.290131] pci 0000:00:14.3: [1022:780e] type 00 class 0x060100
[    0.290335] pci 0000:00:14.4: [1022:780f] type 01 class 0x060401
[    0.290490] pci 0000:00:14.5: [1022:7809] type 00 class 0x0c0310
[    0.290504] pci 0000:00:14.5: reg 0x10: [mem 0xf01ca000-0xf01cafff]
[    0.290677] pci 0000:00:15.0: [1022:43a0] type 01 class 0x060400
[    0.290705] pci 0000:00:15.0: enabling Extended Tags
[    0.290746] pci 0000:00:15.0: supports D1 D2
[    0.290907] pci 0000:00:15.1: [1022:43a1] type 01 class 0x060400
[    0.290939] pci 0000:00:15.1: enabling Extended Tags
[    0.290978] pci 0000:00:15.1: supports D1 D2
[    0.291146] pci 0000:00:15.2: [1022:43a2] type 01 class 0x060400
[    0.291174] pci 0000:00:15.2: enabling Extended Tags
[    0.291214] pci 0000:00:15.2: supports D1 D2
[    0.291289] pci 0000:00:16.0: [1022:7807] type 00 class 0x0c0310
[    0.291303] pci 0000:00:16.0: reg 0x10: [mem 0xf01cb000-0xf01cbfff]
[    0.291473] pci 0000:00:16.2: [1022:7808] type 00 class 0x0c0320
[    0.291486] pci 0000:00:16.2: reg 0x10: [mem 0xf01cf000-0xf01cf0ff]
[    0.291551] pci 0000:00:16.2: supports D1 D2
[    0.291552] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[    0.291554] pci 0000:00:16.2: pme_poll = true
[    0.291555] pci 0000:00:16.2: after device_set_wakeup_capable()
[    0.291558] pci 0000:00:16.2: after pci_pme_active()
[    0.291684] pci 0000:00:18.0: [1022:1400] type 00 class 0x060000
[    0.291748] pci 0000:00:18.1: [1022:1401] type 00 class 0x060000
[    0.291806] pci 0000:00:18.2: [1022:1402] type 00 class 0x060000
[    0.291866] pci 0000:00:18.3: [1022:1403] type 00 class 0x060000
[    0.291998] pci 0000:00:18.4: [1022:1404] type 00 class 0x060000
[    0.292064] pci 0000:00:18.5: [1022:1405] type 00 class 0x060000
[    0.292136] pci_bus 0000:01: extended config space not accessible
[    0.292200] pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
[    0.292208] pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
[    0.292211] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff window] (subtractive decode)
[    0.292214] pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000dffff] (subtractive decode)
[    0.292216] pci 0000:00:14.4:   bridge window [mem 0x80000000-0xffffffff] (subtractive decode)
[    0.292261] pci 0000:00:15.0: PCI bridge to [bus 02]
[    0.292345] pci 0000:03:00.0: [1b21:1042] type 00 class 0x0c0330
[    0.292382] pci 0000:03:00.0: reg 0x10: [mem 0xf0000000-0xf0007fff 64bit]
[    0.292559] pci 0000:03:00.0: PME# supported from D3hot D3cold
[    0.292560] pci 0000:03:00.0: pme_poll = true
[    0.292562] pci 0000:03:00.0: after device_set_wakeup_capable()
[    0.292567] pci 0000:03:00.0: after pci_pme_active()
[    0.292605] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:15.1 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    0.305215] pci 0000:00:15.1: PCI bridge to [bus 03]
[    0.305228] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[    0.305237] pci 0000:00:15.2: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.305359] pci 0000:04:00.0: [10ec:8168] type 00 class 0x020000
[    0.305378] pci 0000:04:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.305399] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit pref]
[    0.305413] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[    0.305520] pci 0000:04:00.0: supports D1 D2
[    0.305522] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.305524] pci 0000:04:00.0: pme_poll = true
[    0.305525] pci 0000:04:00.0: after device_set_wakeup_capable()
[    0.305529] pci 0000:04:00.0: after pci_pme_active()
[    0.321217] pci 0000:00:15.2: PCI bridge to [bus 04-ff]
[    0.321228] pci 0000:00:15.2:   bridge window [io  0x0000-0x0fff]
[    0.321232] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff]
[    0.321237] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.321240] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    0.321759] ACPI: PCI: Interrupt link INTA configured for IRQ 0
[    0.321855] ACPI: PCI: Interrupt link INTB configured for IRQ 0
[    0.321949] ACPI: PCI: Interrupt link INTC configured for IRQ 0
[    0.322041] ACPI: PCI: Interrupt link INTD configured for IRQ 0
[    0.322134] ACPI: PCI: Interrupt link INTE configured for IRQ 0
[    0.322232] ACPI: PCI: Interrupt link INTF configured for IRQ 0
[    0.322325] ACPI: PCI: Interrupt link INTG configured for IRQ 0
[    0.322418] ACPI: PCI: Interrupt link INTH configured for IRQ 0
[    0.322640] iommu: Default domain type: Translated 
[    0.322642] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.322839] SCSI subsystem initialized
[    0.325159] libata version 3.00 loaded.
[    0.325159] ACPI: bus type USB registered
[    0.325159] usbcore: registered new interface driver usbfs
[    0.325159] usbcore: registered new interface driver hub
[    0.325159] usbcore: registered new device driver usb
[    0.325159] PCI: Using ACPI for IRQ routing
[    0.325159] PCI: pci_cache_line_size set to 64 bytes
[    0.325159] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    0.325159] e820: reserve RAM buffer [mem 0x5fe4d000-0x5fffffff]
[    0.325159] e820: reserve RAM buffer [mem 0x17f000000-0x17fffffff]
[    0.325159] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.325159] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    0.330232] clocksource: Switched to clocksource tsc-early
[    0.330474] VFS: Disk quotas dquot_6.6.0
[    0.330500] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.330620] pnp: PnP ACPI init
[    0.330969] system 00:00: [mem 0xfec10002-0xfec11001] could not be reserved
[    0.331362] pnp: PnP ACPI: found 2 devices
[    0.337986] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.338186] NET: Registered PF_INET protocol family
[    0.338359] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.340153] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.340169] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.340177] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.340251] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    0.340614] TCP: Hash tables configured (established 32768 bind 32768)
[    0.340685] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.340711] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.340827] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.340862] pci 0000:00:15.2: BAR 15: assigned [mem 0x80000000-0x800fffff 64bit pref]
[    0.340869] pci 0000:00:15.2: BAR 13: assigned [io  0x2000-0x2fff]
[    0.340874] pci 0000:00:14.4: PCI bridge to [bus 01]
[    0.340886] pci 0000:00:15.0: PCI bridge to [bus 02]
[    0.340894] pci 0000:00:15.1: PCI bridge to [bus 03]
[    0.340898] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[    0.340908] pci 0000:04:00.0: BAR 4: assigned [mem 0x80000000-0x80003fff 64bit pref]
[    0.340922] pci 0000:04:00.0: BAR 2: assigned [mem 0x80004000-0x80004fff 64bit pref]
[    0.340934] pci 0000:04:00.0: BAR 0: assigned [io  0x2000-0x20ff]
[    0.340941] pci 0000:00:15.2: PCI bridge to [bus 04]
[    0.340943] pci 0000:00:15.2:   bridge window [io  0x2000-0x2fff]
[    0.340949] pci 0000:00:15.2:   bridge window [mem 0x80000000-0x800fffff 64bit pref]
[    0.340956] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.340958] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.340960] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000dffff]
[    0.340964] pci_bus 0000:00: resource 7 [mem 0x80000000-0xffffffff]
[    0.340966] pci_bus 0000:01: resource 4 [io  0x0000-0x0cf7 window]
[    0.340968] pci_bus 0000:01: resource 5 [io  0x0d00-0xffff window]
[    0.340970] pci_bus 0000:01: resource 6 [mem 0x000a0000-0x000dffff]
[    0.340973] pci_bus 0000:01: resource 7 [mem 0x80000000-0xffffffff]
[    0.340975] pci_bus 0000:03: resource 1 [mem 0xf0000000-0xf00fffff]
[    0.340977] pci_bus 0000:04: resource 0 [io  0x2000-0x2fff]
[    0.340979] pci_bus 0000:04: resource 2 [mem 0x80000000-0x800fffff 64bit pref]
[    0.341104] pci 0000:00:01.1: D0 power state depends on 0000:00:01.0
[    0.341412] pci 0000:00:12.0: AMD USB device
[    0.341436] pci 0000:00:12.0: AMD USB ohci handoff
[    0.341833] pci 0000:00:12.2: AMD USB device
[    0.341847] pci 0000:00:12.2: AMD USB ehci handoff
[    0.342032] pci 0000:00:12.2: PME# does not work under D3, disabling it
[    0.342235] pci 0000:00:13.0: AMD USB device
[    0.342250] pci 0000:00:13.0: AMD USB ohci handoff
[    0.342633] pci 0000:00:13.2: AMD USB device
[    0.342650] pci 0000:00:13.2: AMD USB ehci handoff
[    0.342830] pci 0000:00:13.2: PME# does not work under D3, disabling it
[    0.343041] pci 0000:00:14.5: AMD USB device
[    0.343057] pci 0000:00:14.5: AMD USB ohci handoff
[    0.343577] pci 0000:00:16.0: AMD USB device
[    0.343598] pci 0000:00:16.0: AMD USB ohci handoff
[    0.343993] pci 0000:00:16.2: AMD USB device
[    0.344008] pci 0000:00:16.2: AMD USB ehci handoff
[    0.344202] pci 0000:00:16.2: PME# does not work under D3, disabling it
[    0.344510] pci 0000:03:00.0: AMD USB xhci handoff
[    0.344577] PCI: CLS 64 bytes, default 64
[    0.344678] pci 0000:00:00.2: AMD-Vi: Applying erratum 746 workaround
[    0.344790] pci 0000:00:01.0: Adding to iommu group 0
[    0.344813] pci 0000:00:01.1: Adding to iommu group 0
[    0.344844] pci 0000:00:11.0: Adding to iommu group 1
[    0.344885] pci 0000:00:12.0: Adding to iommu group 2
[    0.344905] pci 0000:00:12.2: Adding to iommu group 2
[    0.344949] pci 0000:00:13.0: Adding to iommu group 3
[    0.344970] pci 0000:00:13.2: Adding to iommu group 3
[    0.345020] pci 0000:00:14.0: Adding to iommu group 4
[    0.345043] pci 0000:00:14.2: Adding to iommu group 4
[    0.345066] pci 0000:00:14.3: Adding to iommu group 4
[    0.345102] pci 0000:00:14.4: Adding to iommu group 5
[    0.345128] pci 0000:00:14.5: Adding to iommu group 6
[    0.345172] pci 0000:00:15.0: Adding to iommu group 7
[    0.345198] pci 0000:00:15.1: Adding to iommu group 7
[    0.345219] pci 0000:00:15.2: Adding to iommu group 7
[    0.345261] pci 0000:00:16.0: Adding to iommu group 8
[    0.345285] pci 0000:00:16.2: Adding to iommu group 8
[    0.345349] pci 0000:00:18.0: Adding to iommu group 9
[    0.345374] pci 0000:00:18.1: Adding to iommu group 9
[    0.345400] pci 0000:00:18.2: Adding to iommu group 9
[    0.345428] pci 0000:00:18.3: Adding to iommu group 9
[    0.345451] pci 0000:00:18.4: Adding to iommu group 9
[    0.345479] pci 0000:00:18.5: Adding to iommu group 9
[    0.345489] pci 0000:03:00.0: Adding to iommu group 7
[    0.345499] pci 0000:04:00.0: Adding to iommu group 7
[    0.348265] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.348272] AMD-Vi: Extended features (0x800000853, 0x0): PreF PPR GT IA
[    0.348279] AMD-Vi: Interrupt remapping enabled
[    0.348484] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.348486] software IO TLB: mapped [mem 0x000000005be4d000-0x000000005fe4d000] (64MB)
[    0.348547] LVT offset 0 assigned for vector 0x400
[    0.348595] perf: AMD IBS detected (0x000000ff)
[    0.348602] amd_uncore: 4  amd_nb counters detected
[    0.352225] workingset: timestamp_bits=37 max_order=20 bucket_order=0
[    0.352259] zbud: loaded
[    0.352752] NET: Registered PF_ALG protocol family
[    0.352758] Key type asymmetric registered
[    0.352759] Asymmetric key parser 'x509' registered
[    0.353113] alg: self-tests disabled
[    0.353209] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.353263] io scheduler mq-deadline registered
[    0.353265] io scheduler kyber registered
[    0.354479] pcieport 0000:00:15.0: PME: Signaling with IRQ 25
[    0.354644] pcieport 0000:00:15.1: PME: Signaling with IRQ 26
[    0.354712] pcieport 0000:00:15.2: enabling device (0000 -> 0003)
[    0.354941] pcieport 0000:00:15.2: PME: Signaling with IRQ 27
[    0.355211] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.355298] ACPI: button: Power Button [PWRF]
[    0.355363] ACPI: \_SB_.P000: Found 2 idle states
[    0.355495] ACPI: \_SB_.P001: Found 2 idle states
[    0.356442] thermal LNXTHERM:00: registered as thermal_zone0
[    0.356445] ACPI: thermal: Thermal Zone [TZ00] (21 C)
[    0.356797] Non-volatile memory driver v1.3
[    0.356864] AMD-Vi: AMD IOMMUv2 loaded and initialized
[    0.357065] ahci 0000:00:11.0: version 3.0
[    0.357356] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 8 ports 6 Gbps 0x40 impl SATA mode
[    0.357360] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck led clo pio 
[    0.358785] scsi host0: ahci
[    0.358999] scsi host1: ahci
[    0.359215] scsi host2: ahci
[    0.359424] scsi host3: ahci
[    0.359624] scsi host4: ahci
[    0.359836] scsi host5: ahci
[    0.360048] scsi host6: ahci
[    0.360243] scsi host7: ahci
[    0.360336] ata port1: DUMMY
[    0.360338] ata port2: DUMMY
[    0.360339] ata port3: DUMMY
[    0.360341] ata port4: DUMMY
[    0.360342] ata port5: DUMMY
[    0.360343] ata port6: DUMMY
[    0.360345] ata port7: SATA max UDMA/133 abar m2048@0xf01cc000 port 0xf01cc400 irq 19
[    0.360347] ata port8: DUMMY
[    0.360431] ACPI: bus type drm_connector registered
[    0.360734] i8042: PNP: No PS/2 controller found.
[    0.360736] i8042: Probing ports directly.
[    0.363078] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.363185] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.363347] mousedev: PS/2 mouse device common for all mice
[    0.363420] rtc_cmos 00:01: RTC can wake from S4
[    0.363924] rtc_cmos 00:01: registered as rtc0
[    0.363950] rtc_cmos 00:01: setting system clock to 2023-04-17T17:24:09 UTC (1681752249)
[    0.364011] rtc_cmos 00:01: alarms up to one day, y3k, 114 bytes nvram, hpet irqs
[    0.364048] device-mapper: uevent: version 1.0.3
[    0.364136] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com
[    0.364301] hid: raw HID events driver (C) Jiri Kosina
[    0.364335] usbcore: registered new interface driver usbhid
[    0.364336] usbhid: USB HID core driver
[    0.364427] Initializing XFRM netlink socket
[    0.364436] NET: Registered PF_PACKET protocol family
[    0.364438] x86/pm: family 0x15 cpu detected, MSR saving is needed during suspending.
[    0.364652] microcode: CPU1: patch_level=0x0600111f
[    0.364652] microcode: CPU0: patch_level=0x0600111f
[    0.364664] microcode: Microcode Update Driver: v2.2.
[    0.364668] IPI shorthand broadcast: enabled
[    0.364678] AVX version of gcm_enc/dec engaged.
[    0.364710] AES CTR mode by8 optimization enabled
[    0.368578] sched_clock: Marking stable (249972593, 117151517)->(369867515, -2743405)
[    0.368823] registered taskstats version 1
[    0.369080] zswap: loaded using pool lzo/zbud
[    0.371663] ata link7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    0.371939] ata dev7.0: ATA-9: SanDisk SDSSDP064G, 2.0.0, max UDMA/133
[    0.371942] ata dev7.0: 125045424 sectors, multi 1: LBA48 NCQ (depth 32)
[    0.372148] ata dev7.0: configured for UDMA/133
[    0.372283] scsi 6:0:0:0: Direct-Access     ATA      SanDisk SDSSDP06 0    PQ: 0 ANSI: 5
[    0.373496] kmemleak: Kernel memory leak detector initialized (mem pool available: 15679)
[    0.373501] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.373959] sd 6:0:0:0: [sda] 125045424 512-byte logical blocks: (64.0 GB/59.6 GiB)
[    0.373982] sd 6:0:0:0: [sda] Write Protect is off
[    0.373987] sd 6:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    0.374015] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.374059] sd 6:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[    0.380407] kmemleak: Automatic memory scanning thread started
[    0.380633]  sda: sda1 sda2 sda3
[    0.381120] Key type encrypted registered
[    0.381248] sd 6:0:0:0: [sda] Attached SCSI disk
[    0.384294] PM:   Magic number: 3:443:441
[    0.396721] EXT4-fs (sda3): mounted filesystem fe29e0dc-6303-4401-987c-8472bc1b9516 with ordered data mode. Quota mode: none.
[    0.396766] VFS: Mounted root (ext4 filesystem) on device 8:3.
[    0.398650] devtmpfs: mounted
[    0.398668] After kernel_init_freeable
[    0.403170] Freeing unused kernel image (initmem) memory: 2908K
[    0.415639] Write protecting the kernel read-only data: 20480k
[    0.415922] Freeing unused kernel image (rodata/data gap) memory: 836K
[    0.453024] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    0.453030] rodata_test: all tests were successful
[    0.453031] After mark_readonly
[    0.453031] After pti_finalize
[    0.453045] rcu_end_inkernel_boot
[    0.453054] Run /sbin/init as init process
[    0.453055]   with arguments:
[    0.453057]     /sbin/init
[    0.453058]     noisapnp
[    0.453059]   with environment:
[    0.453060]     HOME=/
[    0.453060]     TERM=linux
[    0.453061]     BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66
[    0.629518] systemd[1]: Inserted module 'autofs4'
[    0.656601] NET: Registered PF_INET6 protocol family
[    0.657495] Segment Routing with IPv6
[    0.657523] In-situ OAM (IOAM) with IPv6
[    0.683745] systemd[1]: systemd 252.6-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.683756] systemd[1]: Detected architecture x86-64.
[    0.688485] systemd[1]: Hostname set to <kodi>.
[    0.959071] systemd[1]: Queued start job for default target graphical.target.
[    0.982198] systemd[1]: Created slice system-getty.slice - Slice /system/getty.
[    0.983286] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[    0.984123] systemd[1]: Created slice user.slice - User and Session Slice.
[    0.984302] systemd[1]: Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[    0.984417] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[    0.984855] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    0.984894] systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[    0.984934] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[    0.984972] systemd[1]: Reached target paths.target - Path Units.
[    0.985006] systemd[1]: Reached target remote-fs.target - Remote File Systems.
[    0.985038] systemd[1]: Reached target slices.target - Slice Units.
[    0.985078] systemd[1]: Reached target swap.target - Swaps.
[    0.985116] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[    0.987550] systemd[1]: Listening on systemd-coredump.socket - Process Core Dump Socket.
[    0.987802] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[    0.987975] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[    0.988274] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[    0.988543] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    0.988822] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[    0.989072] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[    0.989946] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[    0.990210] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[    0.992904] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    0.995944] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    0.999142] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    1.002682] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    1.007256] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[    1.014276] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[    1.017399] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[    1.020657] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[    1.024006] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[    1.027278] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[    1.036066] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
[    1.036151] systemd[1]: systemd-firstboot.service - First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    1.036237] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathIsReadWrite=!/).
[    1.036279] systemd[1]: Reached target local-fs.target - Local File Systems.
[    1.036362] systemd[1]: apparmor.service - Load AppArmor profiles was skipped because of an unmet condition check (ConditionSecurity=apparmor).
[    1.041356] loop: module loaded
[    1.043749] systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[    1.048983] systemd[1]: Starting systemd-journald.service - Journal Service...
[    1.050740] fuse: init (API version 7.38)
[    1.052824] systemd[1]: Starting systemd-random-seed.service - Load/Save Random Seed...
[    1.063924] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[    1.068061] systemd[1]: Starting systemd-sysusers.service - Create System Users...
[    1.071328] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[    1.093517] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    1.094134] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    1.094447] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    1.094985] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    1.095974] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[    1.096888] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    1.097257] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[    1.098001] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    1.098339] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    1.099077] systemd[1]: modprobe@drm.service: Deactivated successfully.
[    1.099446] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[    1.100518] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[    1.100846] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[    1.101749] systemd[1]: modprobe@fuse.service: Deactivated successfully.
[    1.102064] systemd[1]: Finished modprobe@fuse.service - Load Kernel Module fuse.
[    1.102810] systemd[1]: modprobe@loop.service: Deactivated successfully.
[    1.103142] systemd[1]: Finished modprobe@loop.service - Load Kernel Module loop.
[    1.103830] systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 138 (systemd-binfmt)
[    1.123242] systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[    1.140028] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[    1.148223] systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
[    1.148346] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[    1.148515] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[    1.159909] systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.
[    1.184529] systemd[1]: Finished systemd-sysusers.service - Create System Users.
[    1.186000] systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[    1.186357] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[    1.186600] systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
[    1.209823] systemd[1]: Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[    1.210512] systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[    1.254273] systemd[1]: Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
[    1.273469] systemd[1]: Starting systemd-udevd.service - Rule-based Manager for Device Events and Files...
[    1.313286] systemd[1]: Started systemd-journald.service - Journal Service.
[    1.359636] systemd-journald[139]: Received client request to flush runtime journal.
[    1.369173] tsc: Refined TSC clocksource calibration: 3900.223 MHz
[    1.369184] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x70705a6472c, max_idle_ns: 881590586812 ns
[    1.369200] clocksource: Switched to clocksource tsc
[    1.594559] sd 6:0:0:0: Attached scsi generic sg0 type 0
[    1.731704] acpi_cpufreq: overriding BIOS provided _PSD data
[    1.953174] random: crng init done
[    1.992654] QUIRK: Enable AMD PLL fix
[    1.992706] ehci-pci 0000:00:12.2: EHCI Host Controller
[    1.992738] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 1
[    1.992751] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    1.992760] ehci-pci 0000:00:12.2: debug port 1
[    1.992971] ehci-pci 0000:00:12.2: irq 17, io mem 0xf01cd000
[    2.005197] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00
[    2.005468] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.005471] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.005474] usb usb1: Product: EHCI Host Controller
[    2.005476] usb usb1: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[    2.005478] usb usb1: SerialNumber: 0000:00:12.2
[    2.006279] hub 1-0:1.0: USB hub found
[    2.006321] hub 1-0:1.0: 5 ports detected
[    2.007319] ehci-pci 0000:00:13.2: EHCI Host Controller
[    2.007345] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 2
[    2.007360] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.007369] ehci-pci 0000:00:13.2: debug port 1
[    2.007523] ehci-pci 0000:00:13.2: irq 17, io mem 0xf01ce000
[    2.016141] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[    2.016147] piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection
[    2.016788] piix4_smbus 0000:00:14.0: Auxiliary SMBus Host Controller at 0xb20
[    2.021184] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00
[    2.021523] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.021527] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.021529] usb usb2: Product: EHCI Host Controller
[    2.021531] usb usb2: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[    2.021533] usb usb2: SerialNumber: 0000:00:13.2
[    2.032086] hub 2-0:1.0: USB hub found
[    2.032171] hub 2-0:1.0: 5 ports detected
[    2.033420] ehci-pci 0000:00:16.2: EHCI Host Controller
[    2.033455] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 3
[    2.033470] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.033479] ehci-pci 0000:00:16.2: debug port 1
[    2.033641] ehci-pci 0000:00:16.2: irq 17, io mem 0xf01cf000
[    2.049177] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00
[    2.049454] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.049459] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.049461] usb usb3: Product: EHCI Host Controller
[    2.049463] usb usb3: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ehci_hcd
[    2.049465] usb usb3: SerialNumber: 0000:00:16.2
[    2.050025] hub 3-0:1.0: USB hub found
[    2.050070] hub 3-0:1.0: 4 ports detected
[    2.050881] ohci-pci 0000:00:16.0: OHCI PCI host controller
[    2.050910] ohci-pci 0000:00:16.0: new USB bus registered, assigned bus number 4
[    2.050999] ohci-pci 0000:00:12.0: OHCI PCI host controller
[    2.051021] ohci-pci 0000:00:12.0: new USB bus registered, assigned bus number 5
[    2.051110] ohci-pci 0000:00:16.0: irq 18, io mem 0xf01cb000
[    2.051111] ohci-pci 0000:00:12.0: irq 18, io mem 0xf01c8000
[    2.051120] ohci-pci 0000:00:13.0: OHCI PCI host controller
[    2.051141] ohci-pci 0000:00:13.0: new USB bus registered, assigned bus number 6
[    2.051240] ohci-pci 0000:00:13.0: irq 18, io mem 0xf01c9000
[    2.051250] ohci-pci 0000:00:14.5: OHCI PCI host controller
[    2.051268] ohci-pci 0000:00:14.5: new USB bus registered, assigned bus number 7
[    2.051356] ohci-pci 0000:00:14.5: irq 18, io mem 0xf01ca000
[    2.113579] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.113586] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.113589] usb usb4: Product: OHCI PCI host controller
[    2.113591] usb usb4: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[    2.113593] usb usb4: SerialNumber: 0000:00:16.0
[    2.114096] hub 4-0:1.0: USB hub found
[    2.114127] hub 4-0:1.0: 4 ports detected
[    2.126194] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.126202] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.126204] usb usb5: Product: OHCI PCI host controller
[    2.126207] usb usb5: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[    2.126209] usb usb5: SerialNumber: 0000:00:12.0
[    2.127341] hub 5-0:1.0: USB hub found
[    2.127380] hub 5-0:1.0: 5 ports detected
[    2.128522] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.128528] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.128531] usb usb6: Product: OHCI PCI host controller
[    2.128533] usb usb6: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[    2.128535] usb usb6: SerialNumber: 0000:00:13.0
[    2.129049] hub 6-0:1.0: USB hub found
[    2.129082] hub 6-0:1.0: 5 ports detected
[    2.130311] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.130316] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.130319] usb usb7: Product: OHCI PCI host controller
[    2.130321] usb usb7: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 ohci_hcd
[    2.130323] usb usb7: SerialNumber: 0000:00:14.5
[    2.130855] hub 7-0:1.0: USB hub found
[    2.130895] hub 7-0:1.0: 2 ports detected
[    2.148137] 1
[    2.148196] 2
[    2.148820] snd_hda_intel 0000:00:01.1: Force to non-snoop mode
[    2.148832] 3
[    2.148833] 4
[    2.148834] 5
[    2.148835] 7
[    2.148837] 8
[    2.148837] 9
[    2.148929] 1
[    2.148979] 2
[    2.149637] 3
[    2.149641] 4
[    2.149642] 5
[    2.149643] 7
[    2.149645] 8
[    2.149646] 9
[    2.166674] xhci_hcd 0000:03:00.0: xHCI Host Controller
[    2.166707] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 8
[    2.206853] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/sound/card0/input1
[    2.208133] input: HDA ATI HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.1/sound/card0/input2
[    2.221615] snd_hda_codec_realtek hdaudioC1D0: ALC892: SKU not ready 0x00000100
[    2.222377] snd_hda_codec_realtek hdaudioC1D0: autoconfig for ALC892: line_outs=4 (0x14/0x16/0x15/0x17/0x0) type:line
[    2.222384] snd_hda_codec_realtek hdaudioC1D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    2.222387] snd_hda_codec_realtek hdaudioC1D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    2.222390] snd_hda_codec_realtek hdaudioC1D0:    mono: mono_out=0x0
[    2.222392] snd_hda_codec_realtek hdaudioC1D0:    dig-out=0x1e/0x0
[    2.222393] snd_hda_codec_realtek hdaudioC1D0:    inputs:
[    2.222395] snd_hda_codec_realtek hdaudioC1D0:      Rear Mic=0x18
[    2.222398] snd_hda_codec_realtek hdaudioC1D0:      Front Mic=0x19
[    2.222399] snd_hda_codec_realtek hdaudioC1D0:      Line=0x1a
[    2.222401] snd_hda_codec_realtek hdaudioC1D0:      CD=0x1c
[    2.231811] xhci_hcd 0000:03:00.0: hcc params 0x0200f180 hci version 0x96 quirks 0x0000000000080010
[    2.235245] r8169 0000:04:00.0: enabling device (0000 -> 0003)
[    2.265051] xhci_hcd 0000:03:00.0: xHCI Host Controller
[    2.265076] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 9
[    2.265092] xhci_hcd 0000:03:00.0: Host supports USB 3.0 SuperSpeed
[    2.266951] usb usb8: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.266958] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.266961] usb usb8: Product: xHCI Host Controller
[    2.266963] usb usb8: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 xhci-hcd
[    2.266965] usb usb8: SerialNumber: 0000:03:00.0
[    2.267014] input: HD-Audio Generic Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input3
[    2.267354] input: HD-Audio Generic Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input4
[    2.267750] input: HD-Audio Generic Line as /devices/pci0000:00/0000:00:14.2/sound/card1/input5
[    2.268089] input: HD-Audio Generic Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card1/input6
[    2.268437] input: HD-Audio Generic Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card1/input7
[    2.268783] input: HD-Audio Generic Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card1/input8
[    2.269132] input: HD-Audio Generic Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card1/input9
[    2.269714] hub 8-0:1.0: USB hub found
[    2.269815] hub 8-0:1.0: 2 ports detected
[    2.269956] input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card1/input10
[    2.272826] usb usb9: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.273006] usb usb9: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.03
[    2.273010] usb usb9: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.273012] usb usb9: Product: xHCI Host Controller
[    2.273014] usb usb9: Manufacturer: Linux 6.3.0-rc6-00311-gde8224969f66 xhci-hcd
[    2.273016] usb usb9: SerialNumber: 0000:03:00.0
[    2.275521] hub 9-0:1.0: USB hub found
[    2.275649] hub 9-0:1.0: 2 ports detected
[    2.295318] r8169 0000:04:00.0 eth0: RTL8168f/8111f, 08:60:6e:74:7a:51, XID 480, IRQ 32
[    2.295327] r8169 0000:04:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    2.311845] r8169 0000:04:00.0 enp4s0: renamed from eth0
[    2.421403] r8169 0000:04:00.0: Direct firmware load for rtl_nic/rtl8168f-1.fw failed with error -2
[    2.421415] r8169 0000:04:00.0: Unable to load firmware rtl_nic/rtl8168f-1.fw (-2)
[    2.421903] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
[    2.465163] usb 5-1: new low-speed USB device number 2 using ohci-pci
[    2.472735] [drm] radeon kernel modesetting enabled.
[    2.474200] [drm] initializing kernel modesetting (ARUBA 0x1002:0x9996 0x1002:0x9996 0x00).
[    2.474271] ATOM BIOS: 113
[    2.474372] radeon 0000:00:01.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
[    2.474376] radeon 0000:00:01.0: GTT: 1024M 0x0000000020000000 - 0x000000005FFFFFFF
[    2.474384] [drm] Detected VRAM RAM=512M, BAR=256M
[    2.474385] [drm] RAM width 64bits DDR
[    2.474589] [drm] radeon: 512M of VRAM memory ready
[    2.474595] [drm] radeon: 1024M of GTT memory ready.
[    2.474642] [drm] Loading ARUBA Microcode
[    2.481608] [drm] Internal thermal controller without fan control
[    2.481974] [drm] radeon: dpm initialized
[    2.486664] [drm] Found VCE firmware/feedback version 50.0.1 / 17!
[    2.486711] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    2.492402] r8169 0000:04:00.0 enp4s0: Link is Down
[    2.526396] [drm] GART: Restore entries: num cpu pages 262144, num gpu pages 262144
[    2.529843] [drm] GART: Done restoring entries
[    2.529847] [drm] PCIE GART of 1024M enabled (table at 0x00000000001D6000).
[    2.530087] radeon 0000:00:01.0: WB enabled
[    2.530090] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00
[    2.530468] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 0x0000000000075a18
[    2.550526] radeon 0000:00:01.0: fence driver on ring 6 use gpu addr 0x0000000020000c18
[    2.550531] radeon 0000:00:01.0: fence driver on ring 7 use gpu addr 0x0000000020000c1c
[    2.550533] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 0x0000000020000c04
[    2.550535] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 0x0000000020000c08
[    2.550536] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 0x0000000020000c0c
[    2.550538] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 0x0000000020000c10
[    2.550809] radeon 0000:00:01.0: radeon: MSI limited to 32-bit
[    2.551003] radeon 0000:00:01.0: radeon: using MSI.
[    2.551071] [drm] radeon: irq initialized.
[    2.569579] [drm] ring test on 0 succeeded in 3 usecs
[    2.569589] [drm] ring test on 3 succeeded in 4 usecs
[    2.569596] [drm] ring test on 4 succeeded in 4 usecs
[    2.583595] [drm] ring test on 5 succeeded in 2 usecs
[    2.585593] [drm] UVD initialized successfully.
[    2.670213] usb 5-1: New USB device found, idVendor=413c, idProduct=2106, bcdDevice= 1.01
[    2.670218] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.670220] usb 5-1: Product: Dell QuietKey Keyboard
[    2.670222] usb 5-1: Manufacturer: DELL
[    2.678027] input: DELL Dell QuietKey Keyboard as /devices/pci0000:00/0000:00:12.0/usb5/5-1/5-1:1.0/0003:413C:2106.0001/input/input11
[    2.694994] [drm] ring test on 6 succeeded in 18 usecs
[    2.695006] [drm] ring test on 7 succeeded in 3 usecs
[    2.695007] [drm] VCE initialized successfully.
[    2.695174] snd_hda_intel 0000:00:01.1: bound 0000:00:01.0 (ops radeon_audio_component_bind_ops [radeon])
[    2.695346] [drm] ib test on ring 0 succeeded in 0 usecs
[    2.695400] [drm] ib test on ring 3 succeeded in 0 usecs
[    2.695449] [drm] ib test on ring 4 succeeded in 0 usecs
[    2.713280] [drm] ib test on ring 5 succeeded
[    2.729270] [drm] ib test on ring 6 succeeded in 1 usecs
[    2.737964] hid-generic 0003:413C:2106.0001: input,hidraw0: USB HID v1.10 Keyboard [DELL Dell QuietKey Keyboard] on usb-0000:00:12.0-1/input0
[    2.745270] [drm] ib test on ring 7 succeeded in 1 usecs
[    2.748783] [drm] Radeon Display Connectors
[    2.748789] [drm] Connector 0:
[    2.748790] [drm]   DP-1
[    2.748791] [drm]   HPD1
[    2.748792] [drm]   DDC: 0x6530 0x6530 0x6534 0x6534 0x6538 0x6538 0x653c 0x653c
[    2.748795] [drm]   Encoders:
[    2.748796] [drm]     DFP1: INTERNAL_UNIPHY2
[    2.748797] [drm] Connector 1:
[    2.748798] [drm]   VGA-1
[    2.748799] [drm]   HPD2
[    2.748800] [drm]   DDC: 0x6540 0x6540 0x6544 0x6544 0x6548 0x6548 0x654c 0x654c
[    2.748802] [drm]   Encoders:
[    2.748803] [drm]     CRT1: INTERNAL_UNIPHY2
[    2.748804] [drm]     CRT1: NUTMEG
[    2.748805] [drm] Connector 2:
[    2.748806] [drm]   HDMI-A-1
[    2.748807] [drm]   HPD3
[    2.748808] [drm]   DDC: 0x6550 0x6550 0x6554 0x6554 0x6558 0x6558 0x655c 0x655c
[    2.748811] [drm]   Encoders:
[    2.748811] [drm]     DFP2: INTERNAL_UNIPHY
[    3.019999] [drm] fb mappable at 0xE03E9000
[    3.020006] [drm] vram apper at 0xE0000000
[    3.020009] [drm] size 5242880
[    3.020011] [drm] fb depth is 24
[    3.020012] [drm]    pitch is 5120
[    3.020508] fbcon: radeondrmfb (fb0) is primary device
[    3.201239] usb 5-2: new low-speed USB device number 3 using ohci-pci
[    3.215988] Console: switching to colour frame buffer device 160x64
[    3.217767] radeon 0000:00:01.0: [drm] fb0: radeondrmfb frame buffer device
[    3.237462] [drm] Initialized radeon 2.50.0 20080528 for 0000:00:01.0 on minor 0
[    3.397485] usb 5-2: New USB device found, idVendor=046d, idProduct=c016, bcdDevice= 3.40
[    3.397493] usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.397495] usb 5-2: Product: Optical USB Mouse
[    3.397497] usb 5-2: Manufacturer: Logitech
[    3.406006] input: Logitech Optical USB Mouse as /devices/pci0000:00/0000:00:12.0/usb5/5-2/5-2:1.0/0003:046D:C016.0002/input/input12
[    3.406790] hid-generic 0003:046D:C016.0002: input,hidraw1: USB HID v1.10 Mouse [Logitech Optical USB Mouse] on usb-0000:00:12.0-2/input0
[    5.100538] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
[    5.100555] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0: link becomes ready
[    5.945432] [drm] amdgpu kernel modesetting enabled.
[   10.066361] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=258 'systemd'
Thomas Gleixner April 18, 2023, 6:58 a.m. UTC | #8
Paul!

On Mon, Apr 17 2023 at 19:40, Paul Menzel wrote:
> Am 17.04.23 um 16:48 schrieb Thomas Gleixner:
>
>> On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote:
>>> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
>>> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD
>>> Graphics (family: 0x15, model: 0x13, stepping: 0x1)
>>> […]
>>> [    0.259329] smp: Bringing up secondary CPUs ...
>>> [    0.259527] x86: Booting SMP configuration:
>>> [    0.259528] .... node  #0, CPUs:      #1
>>> [    0.261007] After schedule_preempt_disabled
>>> [   10.260990] CPU1 failed to report alive state
>> 
>> Weird. CPU1 fails to come up and report that it has reached the
>> synchronization point.
>> 
>> Does it work when you add cpuhp.parallel=off on the kernel command line?
>
> Yes, the ten seconds delay is gone with `cpuhp.parallel=off`.
>
> There was a patch set in the past, that worked on that device. I think 
> up to v4 it did *not* work at all and hung [1]. I need some days to 
> collect the results again.

Can you please apply the patch below on top of the pile remove the
command line option again?

Thanks,


        tglx
---
 kernel/cpu.c |    1 +
 1 file changed, 1 insertion(+)

--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1777,6 +1777,7 @@ static void __init cpuhp_bringup_mask(co
 			 */
 			WARN_ON(cpuhp_invoke_callback_range(false, cpu, st, CPUHP_OFFLINE));
 		}
+		msleep(20);
 	}
 }
Thomas Gleixner April 18, 2023, 8:40 a.m. UTC | #9
On Tue, Apr 18 2023 at 08:58, Thomas Gleixner wrote:
> On Mon, Apr 17 2023 at 19:40, Paul Menzel wrote:
>> Am 17.04.23 um 16:48 schrieb Thomas Gleixner:
>>
>>> On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote:
>>>> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
>>>> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD
>>>> Graphics (family: 0x15, model: 0x13, stepping: 0x1)
>>>> […]
>>>> [    0.259329] smp: Bringing up secondary CPUs ...
>>>> [    0.259527] x86: Booting SMP configuration:
>>>> [    0.259528] .... node  #0, CPUs:      #1
>>>> [    0.261007] After schedule_preempt_disabled
>>>> [   10.260990] CPU1 failed to report alive state
>>> 
>>> Weird. CPU1 fails to come up and report that it has reached the
>>> synchronization point.
>>> 
>>> Does it work when you add cpuhp.parallel=off on the kernel command line?
>>
>> Yes, the ten seconds delay is gone with `cpuhp.parallel=off`.
>>
>> There was a patch set in the past, that worked on that device. I think 
>> up to v4 it did *not* work at all and hung [1]. I need some days to 
>> collect the results again.
>
> Can you please apply the patch below on top of the pile remove the
> command line option again?

Bah. That patch does not make any sense at all. Not enough coffee.

Can you please provide the output of cpuid?

Thanks,

        tglx
Paul Menzel April 18, 2023, 8:10 p.m. UTC | #10
Dear Thomas,


Am 18.04.23 um 10:40 schrieb Thomas Gleixner:
> On Tue, Apr 18 2023 at 08:58, Thomas Gleixner wrote:
>> On Mon, Apr 17 2023 at 19:40, Paul Menzel wrote:
>>> Am 17.04.23 um 16:48 schrieb Thomas Gleixner:
>>>
>>>> On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote:
>>>>> Am 15.04.23 um 01:44 schrieb Thomas Gleixner:
>>>>> [    0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1)
>>>>> […]
>>>>> [    0.259329] smp: Bringing up secondary CPUs ...
>>>>> [    0.259527] x86: Booting SMP configuration:
>>>>> [    0.259528] .... node  #0, CPUs:      #1
>>>>> [    0.261007] After schedule_preempt_disabled
>>>>> [   10.260990] CPU1 failed to report alive state
>>>>
>>>> Weird. CPU1 fails to come up and report that it has reached the
>>>> synchronization point.
>>>>
>>>> Does it work when you add cpuhp.parallel=off on the kernel command line?
>>>
>>> Yes, the ten seconds delay is gone with `cpuhp.parallel=off`.
>>>
>>> There was a patch set in the past, that worked on that device. I think
>>> up to v4 it did *not* work at all and hung [1]. I need some days to
>>> collect the results again.
>>
>> Can you please apply the patch below on top of the pile remove the
>> command line option again?
> 
> Bah. That patch does not make any sense at all. Not enough coffee.
> 
> Can you please provide the output of cpuid?

Of course. Here the top, and the whole output is attached.

```
CPU 0:
    vendor_id = "AuthenticAMD"
    version information (1/eax):
       processor type  = primary processor (0)
       family          = 0xf (15)
       model           = 0x3 (3)
       stepping id     = 0x1 (1)
       extended family = 0x6 (6)
       extended model  = 0x1 (1)
       (family synth)  = 0x15 (21)
       (model synth)   = 0x13 (19)
       (simple synth)  = AMD (unknown type) (Richland RL-A1) 
[Piledriver], 32nm
[…]
```


Kind regards,

Paul
CPU 0:
   vendor_id = "AuthenticAMD"
   version information (1/eax):
      processor type  = primary processor (0)
      family          = 0xf (15)
      model           = 0x3 (3)
      stepping id     = 0x1 (1)
      extended family = 0x6 (6)
      extended model  = 0x1 (1)
      (family synth)  = 0x15 (21)
      (model synth)   = 0x13 (19)
      (simple synth)  = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm
   miscellaneous (1/ebx):
      process local APIC physical ID = 0x0 (0)
      maximum IDs for CPUs in pkg    = 0x2 (2)
      CLFLUSH line size              = 0x8 (8)
      brand index                    = 0x0 (0)
   brand id = 0x00 (0): unknown
   feature information (1/edx):
      x87 FPU on chip                        = true
      VME: virtual-8086 mode enhancement     = true
      DE: debugging extensions               = true
      PSE: page size extensions              = true
      TSC: time stamp counter                = true
      RDMSR and WRMSR support                = true
      PAE: physical address extensions       = true
      MCE: machine check exception           = true
      CMPXCHG8B inst.                        = true
      APIC on chip                           = true
      SYSENTER and SYSEXIT                   = true
      MTRR: memory type range registers      = true
      PTE global bit                         = true
      MCA: machine check architecture        = true
      CMOV: conditional move/compare instr   = true
      PAT: page attribute table              = true
      PSE-36: page size extension            = true
      PSN: processor serial number           = false
      CLFLUSH instruction                    = true
      DS: debug store                        = false
      ACPI: thermal monitor and clock ctrl   = false
      MMX Technology                         = true
      FXSAVE/FXRSTOR                         = true
      SSE extensions                         = true
      SSE2 extensions                        = true
      SS: self snoop                         = false
      hyper-threading / multi-core supported = true
      TM: therm. monitor                     = false
      IA64                                   = false
      PBE: pending break event               = false
   feature information (1/ecx):
      PNI/SSE3: Prescott New Instructions     = true
      PCLMULDQ instruction                    = true
      DTES64: 64-bit debug store              = false
      MONITOR/MWAIT                           = true
      CPL-qualified debug store               = false
      VMX: virtual machine extensions         = false
      SMX: safer mode extensions              = false
      Enhanced Intel SpeedStep Technology     = false
      TM2: thermal monitor 2                  = false
      SSSE3 extensions                        = true
      context ID: adaptive or shared L1 data  = false
      SDBG: IA32_DEBUG_INTERFACE              = false
      FMA instruction                         = true
      CMPXCHG16B instruction                  = true
      xTPR disable                            = false
      PDCM: perfmon and debug                 = false
      PCID: process context identifiers       = false
      DCA: direct cache access                = false
      SSE4.1 extensions                       = true
      SSE4.2 extensions                       = true
      x2APIC: extended xAPIC support          = false
      MOVBE instruction                       = false
      POPCNT instruction                      = true
      time stamp counter deadline             = false
      AES instruction                         = true
      XSAVE/XSTOR states                      = true
      OS-enabled XSAVE/XSTOR                  = true
      AVX: advanced vector extensions         = true
      F16C half-precision convert instruction = true
      RDRAND instruction                      = false
      hypervisor guest status                 = false
   cache and TLB information (2):
   processor serial number = 0061-0F31-0000-0000-0000-0000
   deterministic cache parameters (4):
      --- cache 0 ---
      cache type                         = no more caches (0)
   MONITOR/MWAIT (5):
      smallest monitor-line size (bytes)       = 0x40 (64)
      largest monitor-line size (bytes)        = 0x40 (64)
      enum of Monitor-MWAIT exts supported     = true
      supports intrs as break-event for MWAIT  = true
      number of C0 sub C-states using MWAIT    = 0x0 (0)
      number of C1 sub C-states using MWAIT    = 0x0 (0)
      number of C2 sub C-states using MWAIT    = 0x0 (0)
      number of C3 sub C-states using MWAIT    = 0x0 (0)
      number of C4 sub C-states using MWAIT    = 0x0 (0)
      number of C5 sub C-states using MWAIT    = 0x0 (0)
      number of C6 sub C-states using MWAIT    = 0x0 (0)
      number of C7 sub C-states using MWAIT    = 0x0 (0)
   Thermal and Power Management Features (6):
      digital thermometer                     = false
      Intel Turbo Boost Technology            = false
      ARAT always running APIC timer          = false
      PLN power limit notification            = false
      ECMD extended clock modulation duty     = false
      PTM package thermal management          = false
      HWP base registers                      = false
      HWP notification                        = false
      HWP activity window                     = false
      HWP energy performance preference       = false
      HWP package level request               = false
      HDC base registers                      = false
      Intel Turbo Boost Max Technology 3.0    = false
      HWP capabilities                        = false
      HWP PECI override                       = false
      flexible HWP                            = false
      IA32_HWP_REQUEST MSR fast access mode   = false
      HW_FEEDBACK MSRs supported              = false
      ignoring idle logical processor HWP req = false
      Thread Director                         = false
      IA32_HW_FEEDBACK_THREAD_CONFIG bit 25   = false
      digital thermometer thresholds          = 0x0 (0)
      hardware coordination feedback          = true
      ACNT2 available                         = false
      performance-energy bias capability      = false
      number of enh hardware feedback classes = 0x0 (0)
      performance capability reporting        = false
      energy efficiency capability reporting  = false
      size of feedback struct (4KB pages)     = 0x1 (1)
      index of CPU's row in feedback struct   = 0x0 (0)
   extended feature flags (7):
      FSGSBASE instructions                    = false
      IA32_TSC_ADJUST MSR supported            = false
      SGX: Software Guard Extensions supported = false
      BMI1 instructions                        = true
      HLE hardware lock elision                = false
      AVX2: advanced vector extensions 2       = false
      FDP_EXCPTN_ONLY                          = false
      SMEP supervisor mode exec protection     = false
      BMI2 instructions                        = false
      enhanced REP MOVSB/STOSB                 = false
      INVPCID instruction                      = false
      RTM: restricted transactional memory     = false
      RDT-CMT/PQoS cache monitoring            = false
      deprecated FPU CS/DS                     = false
      MPX: intel memory protection extensions  = false
      RDT-CAT/PQE cache allocation             = false
      AVX512F: AVX-512 foundation instructions = false
      AVX512DQ: double & quadword instructions = false
      RDSEED instruction                       = false
      ADX instructions                         = false
      SMAP: supervisor mode access prevention  = false
      AVX512IFMA: integer fused multiply add   = false
      PCOMMIT instruction                      = false
      CLFLUSHOPT instruction                   = false
      CLWB instruction                         = false
      Intel processor trace                    = false
      AVX512PF: prefetch instructions          = false
      AVX512ER: exponent & reciprocal instrs   = false
      AVX512CD: conflict detection instrs      = false
      SHA instructions                         = false
      AVX512BW: byte & word instructions       = false
      AVX512VL: vector length                  = false
      PREFETCHWT1                              = false
      AVX512VBMI: vector byte manipulation     = false
      UMIP: user-mode instruction prevention   = false
      PKU protection keys for user-mode        = false
      OSPKE CR4.PKE and RDPKRU/WRPKRU          = false
      WAITPKG instructions                     = false
      AVX512_VBMI2: byte VPCOMPRESS, VPEXPAND  = false
      CET_SS: CET shadow stack                 = false
      GFNI: Galois Field New Instructions      = false
      VAES instructions                        = false
      VPCLMULQDQ instruction                   = false
      AVX512_VNNI: neural network instructions = false
      AVX512_BITALG: bit count/shiffle         = false
      TME: Total Memory Encryption             = false
      AVX512: VPOPCNTDQ instruction            = false
      LA57: 57-bit addrs & 5-level paging      = false
      BNDLDX/BNDSTX MAWAU value in 64-bit mode = 0x0 (0)
      RDPID: read processor ID supported       = false
      KL: key locker                           = false
      bus lock detection                       = false
      CLDEMOTE supports cache line demote      = false
      MOVDIRI instruction                      = false
      MOVDIR64B instruction                    = false
      ENQCMD instruction                       = false
      SGX_LC: SGX launch config supported      = false
      PKS: supervisor protection keys          = false
      SGX-KEYS: SGX attestation services       = false
      AVX512_4VNNIW: neural network instrs     = false
      AVX512_4FMAPS: multiply acc single prec  = false
      fast short REP MOV                       = false
      UINTR: user interrupts                   = false
      AVX512_VP2INTERSECT: intersect mask regs = false
      IA32_MCU_OPT_CTRL SRBDS mitigation MSR   = false
      VERW MD_CLEAR microcode support          = false
      RTM transaction always aborts            = false
      IA32_TSX_FORCE_ABORT MSR                 = false
      SERIALIZE instruction                    = false
      hybrid part                              = false
      TSXLDTRK: TSX suspend load addr tracking = false
      PCONFIG instruction                      = false
      LBR: architectural last branch records   = false
      CET_IBT: CET indirect branch tracking    = false
      AMX-BF16: tile bfloat16 support          = false
      AVX512_FP16: fp16 support                = false
      AMX-TILE: tile architecture support      = false
      AMX-INT8: tile 8-bit integer support     = false
      IBRS/IBPB: indirect branch restrictions  = false
      STIBP: 1 thr indirect branch predictor   = false
      L1D_FLUSH: IA32_FLUSH_CMD MSR            = false
      IA32_ARCH_CAPABILITIES MSR               = false
      IA32_CORE_CAPABILITIES MSR               = false
      SSBD: speculative store bypass disable   = false
   Direct Cache Access Parameters (9):
      PLATFORM_DCA_CAP MSR bits = 0
   Architecture Performance Monitoring Features (0xa):
      version ID                               = 0x0 (0)
      number of counters per logical processor = 0x0 (0)
      bit width of counter                     = 0x0 (0)
      length of EBX bit vector                 = 0x0 (0)
      core cycle event                         = not available
      instruction retired event                = not available
      reference cycles event                   = not available
      last-level cache ref event               = not available
      last-level cache miss event              = not available
      branch inst retired event                = not available
      branch mispred retired event             = not available
      top-down slots event                     = not available
      fixed counter  0 supported               = false
      fixed counter  1 supported               = false
      fixed counter  2 supported               = false
      fixed counter  3 supported               = false
      fixed counter  4 supported               = false
      fixed counter  5 supported               = false
      fixed counter  6 supported               = false
      fixed counter  7 supported               = false
      fixed counter  8 supported               = false
      fixed counter  9 supported               = false
      fixed counter 10 supported               = false
      fixed counter 11 supported               = false
      fixed counter 12 supported               = false
      fixed counter 13 supported               = false
      fixed counter 14 supported               = false
      fixed counter 15 supported               = false
      fixed counter 16 supported               = false
      fixed counter 17 supported               = false
      fixed counter 18 supported               = false
      fixed counter 19 supported               = false
      fixed counter 20 supported               = false
      fixed counter 21 supported               = false
      fixed counter 22 supported               = false
      fixed counter 23 supported               = false
      fixed counter 24 supported               = false
      fixed counter 25 supported               = false
      fixed counter 26 supported               = false
      fixed counter 27 supported               = false
      fixed counter 28 supported               = false
      fixed counter 29 supported               = false
      fixed counter 30 supported               = false
      fixed counter 31 supported               = false
      number of contiguous fixed counters      = 0x0 (0)
      bit width of fixed counters              = 0x0 (0)
      anythread deprecation                    = false
   x2APIC features / processor topology (0xb):
      extended APIC ID                      = 0
      --- level 0 ---
      level number                          = 0x0 (0)
      level type                            = invalid (0)
      bit width of level                    = 0x0 (0)
      number of logical processors at level = 0x0 (0)
   XSAVE features (0xd/0):
      XCR0 valid bit field mask               = 0x4000000000000007
         x87 state                            = true
         SSE state                            = true
         AVX state                            = true
         MPX BNDREGS                          = false
         MPX BNDCSR                           = false
         AVX-512 opmask                       = false
         AVX-512 ZMM_Hi256                    = false
         AVX-512 Hi16_ZMM                     = false
         PKRU state                           = false
         XTILECFG state                       = false
         XTILEDATA state                      = false
      bytes required by fields in XCR0        = 0x00000340 (832)
      bytes required by XSAVE/XRSTOR area     = 0x000003c0 (960)
      XSAVEOPT instruction                    = false
      XSAVEC instruction                      = false
      XGETBV instruction                      = false
      XSAVES/XRSTORS instructions             = false
      XFD: extended feature disable supported = false
      SAVE area size in bytes                 = 0x00000000 (0)
      IA32_XSS valid bit field mask           = 0x0000000000000000
         PT state                             = false
         PASID state                          = false
         CET_U user state                     = false
         CET_S supervisor state               = false
         HDC state                            = false
         UINTR state                          = false
         LBR state                            = false
         HWP state                            = false
   AVX/YMM features (0xd/2):
      AVX/YMM save state byte size             = 0x00000100 (256)
      AVX/YMM save state byte offset           = 0x00000240 (576)
      supported in IA32_XSS or XCR0            = XCR0 (user state)
      64-byte alignment in compacted XSAVE     = false
      XFD faulting supported                   = false
   LWP features (0xd/0x3e):
      LWP save state byte size                 = 0x00000080 (128)
      LWP save state byte offset               = 0x00000340 (832)
      supported in IA32_XSS or XCR0            = XCR0 (user state)
      64-byte alignment in compacted XSAVE     = false
      XFD faulting supported                   = false
   extended processor signature (0x80000001/eax):
      family/generation = 0xf (15)
      model           = 0x3 (3)
      stepping id     = 0x1 (1)
      extended family = 0x6 (6)
      extended model  = 0x1 (1)
      (family synth)  = 0x15 (21)
      (model synth)   = 0x13 (19)
      (simple synth)  = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm
   extended feature flags (0x80000001/edx):
      x87 FPU on chip                       = true
      virtual-8086 mode enhancement         = true
      debugging extensions                  = true
      page size extensions                  = true
      time stamp counter                    = true
      RDMSR and WRMSR support               = true
      physical address extensions           = true
      machine check exception               = true
      CMPXCHG8B inst.                       = true
      APIC on chip                          = true
      SYSCALL and SYSRET instructions       = true
      memory type range registers           = true
      global paging extension               = true
      machine check architecture            = true
      conditional move/compare instruction  = true
      page attribute table                  = true
      page size extension                   = true
      multiprocessing capable               = false
      no-execute page protection            = true
      AMD multimedia instruction extensions = true
      MMX Technology                        = true
      FXSAVE/FXRSTOR                        = true
      SSE extensions                        = true
      1-GB large page support               = true
      RDTSCP                                = true
      long mode (AA-64)                     = true
      3DNow! instruction extensions         = false
      3DNow! instructions                   = false
   extended brand id (0x80000001/ebx):
      raw     = 0x20000000 (536870912)
      BrandId = 0x0 (0)
      PkgType = FM2 (PGA) (2)
   AMD feature flags (0x80000001/ecx):
      LAHF/SAHF supported in 64-bit mode     = true
      CMP Legacy                             = true
      SVM: secure virtual machine            = true
      extended APIC space                    = true
      AltMovCr8                              = true
      LZCNT advanced bit manipulation        = true
      SSE4A support                          = true
      misaligned SSE mode                    = true
      3DNow! PREFETCH/PREFETCHW instructions = true
      OS visible workaround                  = true
      instruction based sampling             = true
      XOP support                            = true
      SKINIT/STGI support                    = true
      watchdog timer support                 = true
      lightweight profiling support          = true
      4-operand FMA instruction              = true
      TCE: translation cache extension       = true
      NodeId MSR C001100C                    = true
      TBM support                            = true
      topology extensions                    = true
      core performance counter extensions    = true
      NB/DF performance counter extensions   = true
      data breakpoint extension              = false
      performance time-stamp counter support = false
      LLC performance counter extensions     = false
      MWAITX/MONITORX supported              = false
      Address mask extension support         = false
   brand = "AMD A6-6400K APU with Radeon(tm) HD Graphics   "
   L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
      instruction # entries     = 0x18 (24)
      instruction associativity = 0xff (255)
      data # entries            = 0x40 (64)
      data associativity        = 0xff (255)
   L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
      instruction # entries     = 0x30 (48)
      instruction associativity = 0xff (255)
      data # entries            = 0x40 (64)
      data associativity        = 0xff (255)
   L1 data cache information (0x80000005/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x4 (4)
      size (KB)         = 0x10 (16)
   L1 instruction cache information (0x80000005/edx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x2 (2)
      size (KB)         = 0x40 (64)
   L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
      instruction # entries     = 0x400 (1024)
      instruction associativity = 8 to 15-way (6)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
      instruction # entries     = 0x200 (512)
      instruction associativity = 4 to 5-way (4)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   L2 unified cache information (0x80000006/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 16 to 31-way (8)
      size (KB)         = 0x400 (1024)
   L3 cache information (0x80000006/edx):
      line size (bytes)     = 0x0 (0)
      lines per tag         = 0x0 (0)
      associativity         = L2 off (0)
      size (in 512KB units) = 0x0 (0)
   RAS Capability (0x80000007/ebx):
      MCA overflow recovery support = false
      SUCCOR support                = false
      HWA: hardware assert support  = false
      scalable MCA support          = false
   Advanced Power Management Features (0x80000007/ecx):
      CmpUnitPwrSampleTimeRatio = 0x0 (0)
   Advanced Power Management Features (0x80000007/edx):
      TS: temperature sensing diode           = true
      FID: frequency ID control               = false
      VID: voltage ID control                 = false
      TTP: thermal trip                       = true
      TM: thermal monitor                     = true
      STC: software thermal control           = false
      100 MHz multiplier control              = true
      hardware P-State control                = true
      TscInvariant                            = true
      CPB: core performance boost             = true
      read-only effective frequency interface = true
      processor feedback interface            = false
      APM power reporting                     = false
      connected standby                       = false
      RAPL: running average power limit       = false
   Physical Address and Linear Address Size (0x80000008/eax):
      maximum physical address bits         = 0x30 (48)
      maximum linear (virtual) address bits = 0x30 (48)
      maximum guest physical address bits   = 0x0 (0)
   Extended Feature Extensions ID (0x80000008/ebx):
      CLZERO instruction                       = false
      instructions retired count support       = false
      always save/restore error pointers       = false
      INVLPGB instruction                      = false
      RDPRU instruction                        = false
      memory bandwidth enforcement             = false
      MCOMMIT instruction                      = false
      WBNOINVD instruction                     = false
      IBPB: indirect branch prediction barrier = true
      interruptible WBINVD, WBNOINVD           = false
      IBRS: indirect branch restr speculation  = false
      STIBP: 1 thr indirect branch predictor   = false
      CPU prefers: IBRS always on              = false
      CPU prefers: STIBP always on             = false
      IBRS preferred over software solution    = false
      IBRS provides same mode protection       = false
      EFER[LMSLE] not supported                = false
      INVLPGB supports TLB flush guest nested  = false
      ppin processor id number supported       = false
      SSBD: speculative store bypass disable   = false
      virtualized SSBD                         = false
      SSBD fixed in hardware                   = false
      CPPC: collaborative processor perf ctrl  = false
      PSFD: predictive store forward disable   = false
      not vulnerable to branch type confusion  = false
      branch sampling feature support          = false
      (vuln to branch type confusion synth)    = true
   Size Identifiers (0x80000008/ecx):
      number of CPU cores                 = 0x2 (2)
      ApicIdCoreIdSize                    = 0x4 (4)
      performance time-stamp counter size = 40 bits (0)
   Feature Extended Size (0x80000008/edx):
      max page count for INVLPGB instruction = 0x0 (0)
      RDPRU instruction max input support    = 0x0 (0)
   SVM Secure Virtual Machine (0x8000000a/eax):
      SvmRev: SVM revision = 0x1 (1)
   SVM Secure Virtual Machine (0x8000000a/edx):
      nested paging                           = true
      LBR virtualization                      = true
      SVM lock                                = true
      NRIP save                               = true
      MSR based TSC rate control              = true
      VMCB clean bits support                 = true
      flush by ASID                           = true
      decode assists                          = true
      SSSE3/SSE5 opcode set disable           = false
      pause intercept filter                  = true
      pause filter threshold                  = true
      AVIC: AMD virtual interrupt controller  = false
      virtualized VMLOAD/VMSAVE               = false
      virtualized global interrupt flag (GIF) = false
      GMET: guest mode execute trap           = false
      X2AVIC: virtualized X2APIC              = false
      supervisor shadow stack                 = false
      guest Spec_ctl support                  = false
      ROGPT: read-only guest page table       = false
      host MCE override                       = false
      INVLPGB/TLBSYNC hyperv interc enable    = false
      VNMI: NMI virtualization                = false
      IBS virtualization                      = false
      guest SVME addr check                   = false
   NASID: number of address space identifiers = 0x10000 (65536):
   L1 TLB information: 1G pages (0x80000019/eax):
      instruction # entries     = 0x18 (24)
      instruction associativity = full (15)
      data # entries            = 0x40 (64)
      data associativity        = full (15)
   L2 TLB information: 1G pages (0x80000019/ebx):
      instruction # entries     = 0x400 (1024)
      instruction associativity = 8 to 15-way (6)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   Performance Optimization Identifiers (0x8000001a/eax):
      128-bit SSE executed full-width = true
      MOVU* better than MOVL*/MOVH*   = true
      256-bit SSE executed full-width = false
   Instruction Based Sampling Identifiers (0x8000001b/eax):
      IBS feature flags valid                  = true
      IBS fetch sampling                       = true
      IBS execution sampling                   = true
      read write of op counter                 = true
      op counting mode                         = true
      branch target address reporting          = true
      IbsOpCurCnt and IbsOpMaxCnt extend 7     = true
      invalid RIP indication support           = true
      fused branch micro-op indication support = false
      IBS fetch control extended MSR support   = false
      IBS op data 4 MSR support                = false
      IBS L3 miss filtering support            = false
   Lightweight Profiling Capabilities: Availability (0x8000001c/eax):
      lightweight profiling                  = false
      LWPVAL instruction                     = false
      instruction retired event              = false
      branch retired event                   = false
      DC miss event                          = false
      core clocks not halted event           = false
      core reference clocks not halted event = false
      continuous mode sampling               = false
      tsc in event record                    = false
      interrupt on threshold overflow        = false
   Lightweight Profiling Capabilities: Supported (0x8000001c/edx):
      lightweight profiling                  = true
      LWPVAL instruction                     = true
      instruction retired event              = true
      branch retired event                   = true
      DC miss event                          = false
      core clocks not halted event           = false
      core reference clocks not halted event = false
      continuous mode sampling               = false
      tsc in event record                    = false
      interrupt on threshold overflow        = true
   Lightweight Profiling Capabilities (0x8000001c/ebx):
      LWPCB byte size             = 0x13 (19)
      event record byte size      = 0x20 (32)
      maximum EventId             = 0x3 (3)
      EventInterval1 field offset = 0x80 (128)
   Lightweight Profiling Capabilities (0x8000001c/ecx):
      latency counter bit size          = 0x0 (0)
      data cache miss address valid     = false
      amount cache latency is rounded   = 0x0 (0)
      LWP implementation version        = 0x1 (1)
      event ring buffer size in records = 0x1 (1)
      branch prediction filtering       = false
      IP filtering                      = false
      cache level filtering             = false
      cache latency filteing            = false
   Cache Properties (0x8000001d):
      --- cache 0 ---
      type                            = data (1)
      level                           = 0x1 (1)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x0 (0)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x4 (4)
      number of sets                  = 64
      write-back invalidate           = false
      cache inclusive of lower levels = false
      (synth size)                    = 16384 (16 KB)
      --- cache 1 ---
      type                            = instruction (2)
      level                           = 0x1 (1)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x1 (1)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x2 (2)
      number of sets                  = 512
      write-back invalidate           = false
      cache inclusive of lower levels = false
      (synth size)                    = 65536 (64 KB)
      --- cache 2 ---
      type                            = unified (3)
      level                           = 0x2 (2)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x1 (1)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x10 (16)
      number of sets                  = 1024
      write-back invalidate           = true
      cache inclusive of lower levels = false
      (synth size)                    = 1048576 (1024 KB)
   extended APIC ID = 16
   Compute Unit Identifiers (0x8000001e/ebx):
      compute unit ID        = 0x0 (0)
      cores per compute unit = 0x2 (2)
   Node Identifiers (0x8000001e/ecx):
      node ID             = 0x0 (0)
      nodes per processor = 0x1 (1)
   (instruction supported synth):
      CMPXCHG8B                = true
      conditional move/compare = true
      PREFETCH/PREFETCHW       = true
   (multi-processing synth) = multi-core (c=2)
   (multi-processing method) = AMD
   (APIC widths synth): CORE_width=1 SMT_width=0
   (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=0
   (uarch synth) = AMD Piledriver, 32nm
   (synth) = AMD A-Series (Richland RL-A1) [Piledriver], 32nm
CPU 1:
   vendor_id = "AuthenticAMD"
   version information (1/eax):
      processor type  = primary processor (0)
      family          = 0xf (15)
      model           = 0x3 (3)
      stepping id     = 0x1 (1)
      extended family = 0x6 (6)
      extended model  = 0x1 (1)
      (family synth)  = 0x15 (21)
      (model synth)   = 0x13 (19)
      (simple synth)  = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm
   miscellaneous (1/ebx):
      process local APIC physical ID = 0x1 (1)
      maximum IDs for CPUs in pkg    = 0x2 (2)
      CLFLUSH line size              = 0x8 (8)
      brand index                    = 0x0 (0)
   brand id = 0x00 (0): unknown
   feature information (1/edx):
      x87 FPU on chip                        = true
      VME: virtual-8086 mode enhancement     = true
      DE: debugging extensions               = true
      PSE: page size extensions              = true
      TSC: time stamp counter                = true
      RDMSR and WRMSR support                = true
      PAE: physical address extensions       = true
      MCE: machine check exception           = true
      CMPXCHG8B inst.                        = true
      APIC on chip                           = true
      SYSENTER and SYSEXIT                   = true
      MTRR: memory type range registers      = true
      PTE global bit                         = true
      MCA: machine check architecture        = true
      CMOV: conditional move/compare instr   = true
      PAT: page attribute table              = true
      PSE-36: page size extension            = true
      PSN: processor serial number           = false
      CLFLUSH instruction                    = true
      DS: debug store                        = false
      ACPI: thermal monitor and clock ctrl   = false
      MMX Technology                         = true
      FXSAVE/FXRSTOR                         = true
      SSE extensions                         = true
      SSE2 extensions                        = true
      SS: self snoop                         = false
      hyper-threading / multi-core supported = true
      TM: therm. monitor                     = false
      IA64                                   = false
      PBE: pending break event               = false
   feature information (1/ecx):
      PNI/SSE3: Prescott New Instructions     = true
      PCLMULDQ instruction                    = true
      DTES64: 64-bit debug store              = false
      MONITOR/MWAIT                           = true
      CPL-qualified debug store               = false
      VMX: virtual machine extensions         = false
      SMX: safer mode extensions              = false
      Enhanced Intel SpeedStep Technology     = false
      TM2: thermal monitor 2                  = false
      SSSE3 extensions                        = true
      context ID: adaptive or shared L1 data  = false
      SDBG: IA32_DEBUG_INTERFACE              = false
      FMA instruction                         = true
      CMPXCHG16B instruction                  = true
      xTPR disable                            = false
      PDCM: perfmon and debug                 = false
      PCID: process context identifiers       = false
      DCA: direct cache access                = false
      SSE4.1 extensions                       = true
      SSE4.2 extensions                       = true
      x2APIC: extended xAPIC support          = false
      MOVBE instruction                       = false
      POPCNT instruction                      = true
      time stamp counter deadline             = false
      AES instruction                         = true
      XSAVE/XSTOR states                      = true
      OS-enabled XSAVE/XSTOR                  = true
      AVX: advanced vector extensions         = true
      F16C half-precision convert instruction = true
      RDRAND instruction                      = false
      hypervisor guest status                 = false
   cache and TLB information (2):
   processor serial number = 0061-0F31-0000-0000-0000-0000
   deterministic cache parameters (4):
      --- cache 0 ---
      cache type                         = no more caches (0)
   MONITOR/MWAIT (5):
      smallest monitor-line size (bytes)       = 0x40 (64)
      largest monitor-line size (bytes)        = 0x40 (64)
      enum of Monitor-MWAIT exts supported     = true
      supports intrs as break-event for MWAIT  = true
      number of C0 sub C-states using MWAIT    = 0x0 (0)
      number of C1 sub C-states using MWAIT    = 0x0 (0)
      number of C2 sub C-states using MWAIT    = 0x0 (0)
      number of C3 sub C-states using MWAIT    = 0x0 (0)
      number of C4 sub C-states using MWAIT    = 0x0 (0)
      number of C5 sub C-states using MWAIT    = 0x0 (0)
      number of C6 sub C-states using MWAIT    = 0x0 (0)
      number of C7 sub C-states using MWAIT    = 0x0 (0)
   Thermal and Power Management Features (6):
      digital thermometer                     = false
      Intel Turbo Boost Technology            = false
      ARAT always running APIC timer          = false
      PLN power limit notification            = false
      ECMD extended clock modulation duty     = false
      PTM package thermal management          = false
      HWP base registers                      = false
      HWP notification                        = false
      HWP activity window                     = false
      HWP energy performance preference       = false
      HWP package level request               = false
      HDC base registers                      = false
      Intel Turbo Boost Max Technology 3.0    = false
      HWP capabilities                        = false
      HWP PECI override                       = false
      flexible HWP                            = false
      IA32_HWP_REQUEST MSR fast access mode   = false
      HW_FEEDBACK MSRs supported              = false
      ignoring idle logical processor HWP req = false
      Thread Director                         = false
      IA32_HW_FEEDBACK_THREAD_CONFIG bit 25   = false
      digital thermometer thresholds          = 0x0 (0)
      hardware coordination feedback          = true
      ACNT2 available                         = false
      performance-energy bias capability      = false
      number of enh hardware feedback classes = 0x0 (0)
      performance capability reporting        = false
      energy efficiency capability reporting  = false
      size of feedback struct (4KB pages)     = 0x1 (1)
      index of CPU's row in feedback struct   = 0x0 (0)
   extended feature flags (7):
      FSGSBASE instructions                    = false
      IA32_TSC_ADJUST MSR supported            = false
      SGX: Software Guard Extensions supported = false
      BMI1 instructions                        = true
      HLE hardware lock elision                = false
      AVX2: advanced vector extensions 2       = false
      FDP_EXCPTN_ONLY                          = false
      SMEP supervisor mode exec protection     = false
      BMI2 instructions                        = false
      enhanced REP MOVSB/STOSB                 = false
      INVPCID instruction                      = false
      RTM: restricted transactional memory     = false
      RDT-CMT/PQoS cache monitoring            = false
      deprecated FPU CS/DS                     = false
      MPX: intel memory protection extensions  = false
      RDT-CAT/PQE cache allocation             = false
      AVX512F: AVX-512 foundation instructions = false
      AVX512DQ: double & quadword instructions = false
      RDSEED instruction                       = false
      ADX instructions                         = false
      SMAP: supervisor mode access prevention  = false
      AVX512IFMA: integer fused multiply add   = false
      PCOMMIT instruction                      = false
      CLFLUSHOPT instruction                   = false
      CLWB instruction                         = false
      Intel processor trace                    = false
      AVX512PF: prefetch instructions          = false
      AVX512ER: exponent & reciprocal instrs   = false
      AVX512CD: conflict detection instrs      = false
      SHA instructions                         = false
      AVX512BW: byte & word instructions       = false
      AVX512VL: vector length                  = false
      PREFETCHWT1                              = false
      AVX512VBMI: vector byte manipulation     = false
      UMIP: user-mode instruction prevention   = false
      PKU protection keys for user-mode        = false
      OSPKE CR4.PKE and RDPKRU/WRPKRU          = false
      WAITPKG instructions                     = false
      AVX512_VBMI2: byte VPCOMPRESS, VPEXPAND  = false
      CET_SS: CET shadow stack                 = false
      GFNI: Galois Field New Instructions      = false
      VAES instructions                        = false
      VPCLMULQDQ instruction                   = false
      AVX512_VNNI: neural network instructions = false
      AVX512_BITALG: bit count/shiffle         = false
      TME: Total Memory Encryption             = false
      AVX512: VPOPCNTDQ instruction            = false
      LA57: 57-bit addrs & 5-level paging      = false
      BNDLDX/BNDSTX MAWAU value in 64-bit mode = 0x0 (0)
      RDPID: read processor ID supported       = false
      KL: key locker                           = false
      bus lock detection                       = false
      CLDEMOTE supports cache line demote      = false
      MOVDIRI instruction                      = false
      MOVDIR64B instruction                    = false
      ENQCMD instruction                       = false
      SGX_LC: SGX launch config supported      = false
      PKS: supervisor protection keys          = false
      SGX-KEYS: SGX attestation services       = false
      AVX512_4VNNIW: neural network instrs     = false
      AVX512_4FMAPS: multiply acc single prec  = false
      fast short REP MOV                       = false
      UINTR: user interrupts                   = false
      AVX512_VP2INTERSECT: intersect mask regs = false
      IA32_MCU_OPT_CTRL SRBDS mitigation MSR   = false
      VERW MD_CLEAR microcode support          = false
      RTM transaction always aborts            = false
      IA32_TSX_FORCE_ABORT MSR                 = false
      SERIALIZE instruction                    = false
      hybrid part                              = false
      TSXLDTRK: TSX suspend load addr tracking = false
      PCONFIG instruction                      = false
      LBR: architectural last branch records   = false
      CET_IBT: CET indirect branch tracking    = false
      AMX-BF16: tile bfloat16 support          = false
      AVX512_FP16: fp16 support                = false
      AMX-TILE: tile architecture support      = false
      AMX-INT8: tile 8-bit integer support     = false
      IBRS/IBPB: indirect branch restrictions  = false
      STIBP: 1 thr indirect branch predictor   = false
      L1D_FLUSH: IA32_FLUSH_CMD MSR            = false
      IA32_ARCH_CAPABILITIES MSR               = false
      IA32_CORE_CAPABILITIES MSR               = false
      SSBD: speculative store bypass disable   = false
   Direct Cache Access Parameters (9):
      PLATFORM_DCA_CAP MSR bits = 0
   Architecture Performance Monitoring Features (0xa):
      version ID                               = 0x0 (0)
      number of counters per logical processor = 0x0 (0)
      bit width of counter                     = 0x0 (0)
      length of EBX bit vector                 = 0x0 (0)
      core cycle event                         = not available
      instruction retired event                = not available
      reference cycles event                   = not available
      last-level cache ref event               = not available
      last-level cache miss event              = not available
      branch inst retired event                = not available
      branch mispred retired event             = not available
      top-down slots event                     = not available
      fixed counter  0 supported               = false
      fixed counter  1 supported               = false
      fixed counter  2 supported               = false
      fixed counter  3 supported               = false
      fixed counter  4 supported               = false
      fixed counter  5 supported               = false
      fixed counter  6 supported               = false
      fixed counter  7 supported               = false
      fixed counter  8 supported               = false
      fixed counter  9 supported               = false
      fixed counter 10 supported               = false
      fixed counter 11 supported               = false
      fixed counter 12 supported               = false
      fixed counter 13 supported               = false
      fixed counter 14 supported               = false
      fixed counter 15 supported               = false
      fixed counter 16 supported               = false
      fixed counter 17 supported               = false
      fixed counter 18 supported               = false
      fixed counter 19 supported               = false
      fixed counter 20 supported               = false
      fixed counter 21 supported               = false
      fixed counter 22 supported               = false
      fixed counter 23 supported               = false
      fixed counter 24 supported               = false
      fixed counter 25 supported               = false
      fixed counter 26 supported               = false
      fixed counter 27 supported               = false
      fixed counter 28 supported               = false
      fixed counter 29 supported               = false
      fixed counter 30 supported               = false
      fixed counter 31 supported               = false
      number of contiguous fixed counters      = 0x0 (0)
      bit width of fixed counters              = 0x0 (0)
      anythread deprecation                    = false
   x2APIC features / processor topology (0xb):
      extended APIC ID                      = 0
      --- level 0 ---
      level number                          = 0x0 (0)
      level type                            = invalid (0)
      bit width of level                    = 0x0 (0)
      number of logical processors at level = 0x0 (0)
   XSAVE features (0xd/0):
      XCR0 valid bit field mask               = 0x4000000000000007
         x87 state                            = true
         SSE state                            = true
         AVX state                            = true
         MPX BNDREGS                          = false
         MPX BNDCSR                           = false
         AVX-512 opmask                       = false
         AVX-512 ZMM_Hi256                    = false
         AVX-512 Hi16_ZMM                     = false
         PKRU state                           = false
         XTILECFG state                       = false
         XTILEDATA state                      = false
      bytes required by fields in XCR0        = 0x00000340 (832)
      bytes required by XSAVE/XRSTOR area     = 0x000003c0 (960)
      XSAVEOPT instruction                    = false
      XSAVEC instruction                      = false
      XGETBV instruction                      = false
      XSAVES/XRSTORS instructions             = false
      XFD: extended feature disable supported = false
      SAVE area size in bytes                 = 0x00000000 (0)
      IA32_XSS valid bit field mask           = 0x0000000000000000
         PT state                             = false
         PASID state                          = false
         CET_U user state                     = false
         CET_S supervisor state               = false
         HDC state                            = false
         UINTR state                          = false
         LBR state                            = false
         HWP state                            = false
   AVX/YMM features (0xd/2):
      AVX/YMM save state byte size             = 0x00000100 (256)
      AVX/YMM save state byte offset           = 0x00000240 (576)
      supported in IA32_XSS or XCR0            = XCR0 (user state)
      64-byte alignment in compacted XSAVE     = false
      XFD faulting supported                   = false
   LWP features (0xd/0x3e):
      LWP save state byte size                 = 0x00000080 (128)
      LWP save state byte offset               = 0x00000340 (832)
      supported in IA32_XSS or XCR0            = XCR0 (user state)
      64-byte alignment in compacted XSAVE     = false
      XFD faulting supported                   = false
   extended processor signature (0x80000001/eax):
      family/generation = 0xf (15)
      model           = 0x3 (3)
      stepping id     = 0x1 (1)
      extended family = 0x6 (6)
      extended model  = 0x1 (1)
      (family synth)  = 0x15 (21)
      (model synth)   = 0x13 (19)
      (simple synth)  = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm
   extended feature flags (0x80000001/edx):
      x87 FPU on chip                       = true
      virtual-8086 mode enhancement         = true
      debugging extensions                  = true
      page size extensions                  = true
      time stamp counter                    = true
      RDMSR and WRMSR support               = true
      physical address extensions           = true
      machine check exception               = true
      CMPXCHG8B inst.                       = true
      APIC on chip                          = true
      SYSCALL and SYSRET instructions       = true
      memory type range registers           = true
      global paging extension               = true
      machine check architecture            = true
      conditional move/compare instruction  = true
      page attribute table                  = true
      page size extension                   = true
      multiprocessing capable               = false
      no-execute page protection            = true
      AMD multimedia instruction extensions = true
      MMX Technology                        = true
      FXSAVE/FXRSTOR                        = true
      SSE extensions                        = true
      1-GB large page support               = true
      RDTSCP                                = true
      long mode (AA-64)                     = true
      3DNow! instruction extensions         = false
      3DNow! instructions                   = false
   extended brand id (0x80000001/ebx):
      raw     = 0x20000000 (536870912)
      BrandId = 0x0 (0)
      PkgType = FM2 (PGA) (2)
   AMD feature flags (0x80000001/ecx):
      LAHF/SAHF supported in 64-bit mode     = true
      CMP Legacy                             = true
      SVM: secure virtual machine            = true
      extended APIC space                    = true
      AltMovCr8                              = true
      LZCNT advanced bit manipulation        = true
      SSE4A support                          = true
      misaligned SSE mode                    = true
      3DNow! PREFETCH/PREFETCHW instructions = true
      OS visible workaround                  = true
      instruction based sampling             = true
      XOP support                            = true
      SKINIT/STGI support                    = true
      watchdog timer support                 = true
      lightweight profiling support          = true
      4-operand FMA instruction              = true
      TCE: translation cache extension       = true
      NodeId MSR C001100C                    = true
      TBM support                            = true
      topology extensions                    = true
      core performance counter extensions    = true
      NB/DF performance counter extensions   = true
      data breakpoint extension              = false
      performance time-stamp counter support = false
      LLC performance counter extensions     = false
      MWAITX/MONITORX supported              = false
      Address mask extension support         = false
   brand = "AMD A6-6400K APU with Radeon(tm) HD Graphics   "
   L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
      instruction # entries     = 0x18 (24)
      instruction associativity = 0xff (255)
      data # entries            = 0x40 (64)
      data associativity        = 0xff (255)
   L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
      instruction # entries     = 0x30 (48)
      instruction associativity = 0xff (255)
      data # entries            = 0x40 (64)
      data associativity        = 0xff (255)
   L1 data cache information (0x80000005/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x4 (4)
      size (KB)         = 0x10 (16)
   L1 instruction cache information (0x80000005/edx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x2 (2)
      size (KB)         = 0x40 (64)
   L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
      instruction # entries     = 0x400 (1024)
      instruction associativity = 8 to 15-way (6)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
      instruction # entries     = 0x200 (512)
      instruction associativity = 4 to 5-way (4)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   L2 unified cache information (0x80000006/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 16 to 31-way (8)
      size (KB)         = 0x400 (1024)
   L3 cache information (0x80000006/edx):
      line size (bytes)     = 0x0 (0)
      lines per tag         = 0x0 (0)
      associativity         = L2 off (0)
      size (in 512KB units) = 0x0 (0)
   RAS Capability (0x80000007/ebx):
      MCA overflow recovery support = false
      SUCCOR support                = false
      HWA: hardware assert support  = false
      scalable MCA support          = false
   Advanced Power Management Features (0x80000007/ecx):
      CmpUnitPwrSampleTimeRatio = 0x0 (0)
   Advanced Power Management Features (0x80000007/edx):
      TS: temperature sensing diode           = true
      FID: frequency ID control               = false
      VID: voltage ID control                 = false
      TTP: thermal trip                       = true
      TM: thermal monitor                     = true
      STC: software thermal control           = false
      100 MHz multiplier control              = true
      hardware P-State control                = true
      TscInvariant                            = true
      CPB: core performance boost             = true
      read-only effective frequency interface = true
      processor feedback interface            = false
      APM power reporting                     = false
      connected standby                       = false
      RAPL: running average power limit       = false
   Physical Address and Linear Address Size (0x80000008/eax):
      maximum physical address bits         = 0x30 (48)
      maximum linear (virtual) address bits = 0x30 (48)
      maximum guest physical address bits   = 0x0 (0)
   Extended Feature Extensions ID (0x80000008/ebx):
      CLZERO instruction                       = false
      instructions retired count support       = false
      always save/restore error pointers       = false
      INVLPGB instruction                      = false
      RDPRU instruction                        = false
      memory bandwidth enforcement             = false
      MCOMMIT instruction                      = false
      WBNOINVD instruction                     = false
      IBPB: indirect branch prediction barrier = true
      interruptible WBINVD, WBNOINVD           = false
      IBRS: indirect branch restr speculation  = false
      STIBP: 1 thr indirect branch predictor   = false
      CPU prefers: IBRS always on              = false
      CPU prefers: STIBP always on             = false
      IBRS preferred over software solution    = false
      IBRS provides same mode protection       = false
      EFER[LMSLE] not supported                = false
      INVLPGB supports TLB flush guest nested  = false
      ppin processor id number supported       = false
      SSBD: speculative store bypass disable   = false
      virtualized SSBD                         = false
      SSBD fixed in hardware                   = false
      CPPC: collaborative processor perf ctrl  = false
      PSFD: predictive store forward disable   = false
      not vulnerable to branch type confusion  = false
      branch sampling feature support          = false
      (vuln to branch type confusion synth)    = true
   Size Identifiers (0x80000008/ecx):
      number of CPU cores                 = 0x2 (2)
      ApicIdCoreIdSize                    = 0x4 (4)
      performance time-stamp counter size = 40 bits (0)
   Feature Extended Size (0x80000008/edx):
      max page count for INVLPGB instruction = 0x0 (0)
      RDPRU instruction max input support    = 0x0 (0)
   SVM Secure Virtual Machine (0x8000000a/eax):
      SvmRev: SVM revision = 0x1 (1)
   SVM Secure Virtual Machine (0x8000000a/edx):
      nested paging                           = true
      LBR virtualization                      = true
      SVM lock                                = true
      NRIP save                               = true
      MSR based TSC rate control              = true
      VMCB clean bits support                 = true
      flush by ASID                           = true
      decode assists                          = true
      SSSE3/SSE5 opcode set disable           = false
      pause intercept filter                  = true
      pause filter threshold                  = true
      AVIC: AMD virtual interrupt controller  = false
      virtualized VMLOAD/VMSAVE               = false
      virtualized global interrupt flag (GIF) = false
      GMET: guest mode execute trap           = false
      X2AVIC: virtualized X2APIC              = false
      supervisor shadow stack                 = false
      guest Spec_ctl support                  = false
      ROGPT: read-only guest page table       = false
      host MCE override                       = false
      INVLPGB/TLBSYNC hyperv interc enable    = false
      VNMI: NMI virtualization                = false
      IBS virtualization                      = false
      guest SVME addr check                   = false
   NASID: number of address space identifiers = 0x10000 (65536):
   L1 TLB information: 1G pages (0x80000019/eax):
      instruction # entries     = 0x18 (24)
      instruction associativity = full (15)
      data # entries            = 0x40 (64)
      data associativity        = full (15)
   L2 TLB information: 1G pages (0x80000019/ebx):
      instruction # entries     = 0x400 (1024)
      instruction associativity = 8 to 15-way (6)
      data # entries            = 0x400 (1024)
      data associativity        = 8 to 15-way (6)
   Performance Optimization Identifiers (0x8000001a/eax):
      128-bit SSE executed full-width = true
      MOVU* better than MOVL*/MOVH*   = true
      256-bit SSE executed full-width = false
   Instruction Based Sampling Identifiers (0x8000001b/eax):
      IBS feature flags valid                  = true
      IBS fetch sampling                       = true
      IBS execution sampling                   = true
      read write of op counter                 = true
      op counting mode                         = true
      branch target address reporting          = true
      IbsOpCurCnt and IbsOpMaxCnt extend 7     = true
      invalid RIP indication support           = true
      fused branch micro-op indication support = false
      IBS fetch control extended MSR support   = false
      IBS op data 4 MSR support                = false
      IBS L3 miss filtering support            = false
   Lightweight Profiling Capabilities: Availability (0x8000001c/eax):
      lightweight profiling                  = false
      LWPVAL instruction                     = false
      instruction retired event              = false
      branch retired event                   = false
      DC miss event                          = false
      core clocks not halted event           = false
      core reference clocks not halted event = false
      continuous mode sampling               = false
      tsc in event record                    = false
      interrupt on threshold overflow        = false
   Lightweight Profiling Capabilities: Supported (0x8000001c/edx):
      lightweight profiling                  = true
      LWPVAL instruction                     = true
      instruction retired event              = true
      branch retired event                   = true
      DC miss event                          = false
      core clocks not halted event           = false
      core reference clocks not halted event = false
      continuous mode sampling               = false
      tsc in event record                    = false
      interrupt on threshold overflow        = true
   Lightweight Profiling Capabilities (0x8000001c/ebx):
      LWPCB byte size             = 0x13 (19)
      event record byte size      = 0x20 (32)
      maximum EventId             = 0x3 (3)
      EventInterval1 field offset = 0x80 (128)
   Lightweight Profiling Capabilities (0x8000001c/ecx):
      latency counter bit size          = 0x0 (0)
      data cache miss address valid     = false
      amount cache latency is rounded   = 0x0 (0)
      LWP implementation version        = 0x1 (1)
      event ring buffer size in records = 0x1 (1)
      branch prediction filtering       = false
      IP filtering                      = false
      cache level filtering             = false
      cache latency filteing            = false
   Cache Properties (0x8000001d):
      --- cache 0 ---
      type                            = data (1)
      level                           = 0x1 (1)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x0 (0)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x4 (4)
      number of sets                  = 64
      write-back invalidate           = false
      cache inclusive of lower levels = false
      (synth size)                    = 16384 (16 KB)
      --- cache 1 ---
      type                            = instruction (2)
      level                           = 0x1 (1)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x1 (1)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x2 (2)
      number of sets                  = 512
      write-back invalidate           = false
      cache inclusive of lower levels = false
      (synth size)                    = 65536 (64 KB)
      --- cache 2 ---
      type                            = unified (3)
      level                           = 0x2 (2)
      self-initializing               = true
      fully associative               = false
      extra cores sharing this cache  = 0x1 (1)
      line size in bytes              = 0x40 (64)
      physical line partitions        = 0x1 (1)
      number of ways                  = 0x10 (16)
      number of sets                  = 1024
      write-back invalidate           = true
      cache inclusive of lower levels = false
      (synth size)                    = 1048576 (1024 KB)
   extended APIC ID = 17
   Compute Unit Identifiers (0x8000001e/ebx):
      compute unit ID        = 0x0 (0)
      cores per compute unit = 0x2 (2)
   Node Identifiers (0x8000001e/ecx):
      node ID             = 0x0 (0)
      nodes per processor = 0x1 (1)
   (instruction supported synth):
      CMPXCHG8B                = true
      conditional move/compare = true
      PREFETCH/PREFETCHW       = true
   (multi-processing synth) = multi-core (c=2)
   (multi-processing method) = AMD
   (APIC widths synth): CORE_width=1 SMT_width=0
   (APIC synth): PKG_ID=0 CORE_ID=1 SMT_ID=0
   (uarch synth) = AMD Piledriver, 32nm
   (synth) = AMD A-Series (Richland RL-A1) [Piledriver], 32nm
Thomas Gleixner April 19, 2023, 9:38 a.m. UTC | #11
Paul!

On Tue, Apr 18 2023 at 22:10, Paul Menzel wrote:
> Am 18.04.23 um 10:40 schrieb Thomas Gleixner:
>> Can you please provide the output of cpuid?
>
> Of course. Here the top, and the whole output is attached.

Thanks for the data. Can you please apply the debug patch below and
provide the dmesg output? Just the line which is added by the patch is
enough. You can boot with cpuhp.parallel=off so you don't have wait for
10 seconds.

Thanks,

        tglx
---
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -814,6 +814,7 @@ static int wakeup_secondary_cpu_via_init
 	unsigned long send_status = 0, accept_status = 0;
 	int maxlvt, num_starts, j;
 
+	pr_info("Kicking AP alive: %d\n", phys_apicid);
 	preempt_disable();
 	maxlvt = lapic_get_maxlvt();
Thomas Gleixner April 19, 2023, 12:38 p.m. UTC | #12
On Wed, Apr 19 2023 at 11:38, Thomas Gleixner wrote:
> On Tue, Apr 18 2023 at 22:10, Paul Menzel wrote:
>> Am 18.04.23 um 10:40 schrieb Thomas Gleixner:
>>> Can you please provide the output of cpuid?
>>
>> Of course. Here the top, and the whole output is attached.
>
> Thanks for the data. Can you please apply the debug patch below and
> provide the dmesg output? Just the line which is added by the patch is
> enough. You can boot with cpuhp.parallel=off so you don't have wait for
> 10 seconds.

Borislav found some a machine which also refuses to boot. It turns of
the debug patch was spot on:

[    0.462724] .... node  #0, CPUs:      #1
[    0.462731] smpboot: Kicking AP alive: 17
[    0.465723]  #2
[    0.465732] smpboot: Kicking AP alive: 18
[    0.467641]  #3
[    0.467641] smpboot: Kicking AP alive: 19

So the kernel gets APICID 17, 18, 19 from ACPI but CPUID leaf 0x1
ebx[31:24], which is the initial APICID has:

CPU1		0x01
CPU2		0x02
CPU3		0x03

Which means the APICID to Linux CPU number lookup based on CPUID 0x01
fails for all of them and stops them dead in the low level startup code.

IOW, the BIOS assignes random numbers to the AP APICs for whatever
raisins, which leaves the parallel startup low level code up a creek
without a paddle, except for actually reading the APICID back from the
APIC. *SHUDDER*

I'm leaning towards disabling the CPUID lead 0x01 based discovery and be
done with it.

Thanks,

        tglx
David Woodhouse April 19, 2023, 1:32 p.m. UTC | #13
On Wed, 2023-04-19 at 14:38 +0200, Thomas Gleixner wrote:
> 
> I'm leaning towards disabling the CPUID lead 0x01 based discovery and be
> done with it.

Makes sense. The large machines where users really want the parallel
startup all ought to have X2APIC and hence CPUID 0x0b.
Thomas Gleixner April 19, 2023, 1:43 p.m. UTC | #14
On Wed, Apr 19 2023 at 14:38, Thomas Gleixner wrote:
> On Wed, Apr 19 2023 at 11:38, Thomas Gleixner wrote:
> IOW, the BIOS assignes random numbers to the AP APICs for whatever
> raisins, which leaves the parallel startup low level code up a creek
> without a paddle, except for actually reading the APICID back from the
> APIC. *SHUDDER*

So Andrew just pointed out on IRC that this might be related to the
ancient issue of the 3-wire APIC bus where IO/APIC and APIC shared the
ID space, but that system is definitely post 3-wire APIC :)

Thanks,

        tglx
Andrew Cooper April 19, 2023, 1:50 p.m. UTC | #15
On 19/04/2023 2:43 pm, Thomas Gleixner wrote:
> On Wed, Apr 19 2023 at 14:38, Thomas Gleixner wrote:
>> On Wed, Apr 19 2023 at 11:38, Thomas Gleixner wrote:
>> IOW, the BIOS assignes random numbers to the AP APICs for whatever
>> raisins, which leaves the parallel startup low level code up a creek
>> without a paddle, except for actually reading the APICID back from the
>> APIC. *SHUDDER*
> So Andrew just pointed out on IRC that this might be related to the
> ancient issue of the 3-wire APIC bus where IO/APIC and APIC shared the
> ID space, but that system is definitely post 3-wire APIC :)

Doesn't mean the BIOS code was updated adequately following that.

What I'm confused by is why this system boots in the first place.  I can
only think that's is a system which only has 4-bit APIC IDs, and happens
to function when bit 4 gets truncated off the top of the SIPI destination...

~Andrew
Andrew Cooper April 19, 2023, 4:21 p.m. UTC | #16
On 19/04/2023 2:50 pm, Andrew Cooper wrote:
> On 19/04/2023 2:43 pm, Thomas Gleixner wrote:
>> On Wed, Apr 19 2023 at 14:38, Thomas Gleixner wrote:
>>> On Wed, Apr 19 2023 at 11:38, Thomas Gleixner wrote:
>>> IOW, the BIOS assignes random numbers to the AP APICs for whatever
>>> raisins, which leaves the parallel startup low level code up a creek
>>> without a paddle, except for actually reading the APICID back from the
>>> APIC. *SHUDDER*
>> So Andrew just pointed out on IRC that this might be related to the
>> ancient issue of the 3-wire APIC bus where IO/APIC and APIC shared the
>> ID space, but that system is definitely post 3-wire APIC :)
> Doesn't mean the BIOS code was updated adequately following that.
>
> What I'm confused by is why this system boots in the first place.  I can
> only think that's is a system which only has 4-bit APIC IDs, and happens
> to function when bit 4 gets truncated off the top of the SIPI destination...

https://www.amd.com/system/files/TechDocs/42300_15h_Mod_10h-1Fh_BKDG.pdf

This system does still require the IO-APICs to be at 0, and the LAPICs
to start at some offset, which is clearly 16 in this case.  Also, this
system has configurable 4-bit or 8-bit wide APIC IDs, and I can't tell
which mode is active just from the manual.

But, it does mean that the BIOS has genuinely modified the APIC IDs of
the logic processors.  This does highlight an error in reasoning with
the parallel bringup code.

For xAPIC, the APIC_ID register is writeable (at least, model
specifically), and CPUID is only the value it would have had at reset. 
So the AP bringup logic can't actually use CPUID reliably.

This was changed in x2APIC, which made the x2APIC_ID immutable.

I don't see an option other than the AP bringup code query for xAPIC vs
x2APIC mode, and either looking at the real APIC_ID register, or falling
back to CPUID.

~Andrew
Thomas Gleixner April 20, 2023, 8:32 a.m. UTC | #17
On Wed, Apr 19 2023 at 17:21, Andrew Cooper wrote:
> On 19/04/2023 2:50 pm, Andrew Cooper wrote:
>> What I'm confused by is why this system boots in the first place.  I can
>> only think that's is a system which only has 4-bit APIC IDs, and happens
>> to function when bit 4 gets truncated off the top of the SIPI destination...
>
> https://www.amd.com/system/files/TechDocs/42300_15h_Mod_10h-1Fh_BKDG.pdf
>
> This system does still require the IO-APICs to be at 0, and the LAPICs
> to start at some offset, which is clearly 16 in this case.  Also, this
> system has configurable 4-bit or 8-bit wide APIC IDs, and I can't tell
> which mode is active just from the manual.

That document contradicts itself:

  "The ApicId of core j must be enumerated/assigned as:
   ApicId[core=j] = (OFFSET_IDX) * MNC + j

   Where OFFSET_IDX is an integer offset (0 to N) used to shift up the
   core ApicId values to allow room for IOAPIC devices.

   It is recommended that BIOS use the following APIC ID assignments for
   the broadest operating system sup- port. Given N = MNC and M =
   Number_Of_IOAPICs:

   • Assign the core ApicId’s first from 0 to N-1, and the IOAPIC IDs
     from N to N+(M-1)."

Oh well. If the rest of these docs is of the same quality then it's not
a surprise that BIOSes are trainwrecks.

> But, it does mean that the BIOS has genuinely modified the APIC IDs of
> the logic processors.  This does highlight an error in reasoning with
> the parallel bringup code.

Yes.

> For xAPIC, the APIC_ID register is writeable (at least, model
> specifically), and CPUID is only the value it would have had at reset. 
> So the AP bringup logic can't actually use CPUID reliably.
>
> This was changed in x2APIC, which made the x2APIC_ID immutable.
>
> I don't see an option other than the AP bringup code query for xAPIC vs
> x2APIC mode, and either looking at the real APIC_ID register, or falling
> back to CPUID.

I'm pondering to simply deny parallel mode if x2APIC is not there.

Thanks,

        tglx
Andrew Cooper April 20, 2023, 9:23 a.m. UTC | #18
On 20/04/2023 9:32 am, Thomas Gleixner wrote:
> On Wed, Apr 19 2023 at 17:21, Andrew Cooper wrote:
>> On 19/04/2023 2:50 pm, Andrew Cooper wrote:
>> For xAPIC, the APIC_ID register is writeable (at least, model
>> specifically), and CPUID is only the value it would have had at reset. 
>> So the AP bringup logic can't actually use CPUID reliably.
>>
>> This was changed in x2APIC, which made the x2APIC_ID immutable.
>>
>> I don't see an option other than the AP bringup code query for xAPIC vs
>> x2APIC mode, and either looking at the real APIC_ID register, or falling
>> back to CPUID.
> I'm pondering to simply deny parallel mode if x2APIC is not there.

I'm not sure if that will help much.

Just because x2APIC is there doesn't mean it's in use.  There are
several generations of Intel system which have x2APIC but also use the
opt-out bit in ACPI tables.  There are some machines which have
mismatched APIC-ness settings in the BIOS->OS handover.

There's very little you can do on the BSP alone to know for certain that
the APs come out of wait-for-SIPI already in x2APIC mode.

One way is the ÆPIC Leak "locked into x2APIC mode" giant security
bodge.  If the system really does have a CPU with an APIC ID above 0xfe,
then chances are good that the APs come out consistently...

~Andrew
Thomas Gleixner April 20, 2023, 11:17 a.m. UTC | #19
On Thu, Apr 20 2023 at 10:23, Andrew Cooper wrote:
> On 20/04/2023 9:32 am, Thomas Gleixner wrote:
>> I'm pondering to simply deny parallel mode if x2APIC is not there.
>
> I'm not sure if that will help much.

Spoilsport.

> Just because x2APIC is there doesn't mean it's in use.  There are
> several generations of Intel system which have x2APIC but also use the
> opt-out bit in ACPI tables.  There are some machines which have
> mismatched APIC-ness settings in the BIOS->OS handover.
>
> There's very little you can do on the BSP alone to know for certain that
> the APs come out of wait-for-SIPI already in x2APIC mode.

Yeah. Reading the APIC that early is going to be entertaining too :)

> One way is the ÆPIC Leak "locked into x2APIC mode" giant security
> bodge. 

Bah.

> If the system really does have a CPU with an APIC ID above 0xfe, then
> chances are good that the APs come out consistently...

Anything else would be really magic :)
Sean Christopherson April 20, 2023, 2:51 p.m. UTC | #20
On Thu, Apr 20, 2023, Thomas Gleixner wrote:
> On Thu, Apr 20 2023 at 10:23, Andrew Cooper wrote:
> > On 20/04/2023 9:32 am, Thomas Gleixner wrote:
> > > On Wed, Apr 19, 2023, Andrew Cooper wrote:
> > > > This was changed in x2APIC, which made the x2APIC_ID immutable.
>
> >> I'm pondering to simply deny parallel mode if x2APIC is not there.
> >
> > I'm not sure if that will help much.
> 
> Spoilsport.

LOL, well let me pile on then.  x2APIC IDs aren't immutable on AMD hardware.  The
ID is read-only when the CPU is in x2APIC mode, but any changes made to the ID
while the CPU is in xAPIC mode survive the transition to x2APIC.  From the APM:

  A value previously written by software to the 8-bit APIC_ID register (MMIO offset
  30h) is converted by hardware into the appropriate format and reflected into the
  32-bit x2APIC_ID register (MSR 802h).

FWIW, my observations from testing on bare metal are that the xAPIC ID is effectively
read-only (writes are dropped) on Intel CPUs as far back as Haswell, while the above
behavior described in the APM holds true on at least Rome and Milan.

My guess is that Intel's uArch specific behavior of the xAPIC ID being read-only
was introduced when x2APIC came along, but I didn't test farther back than Haswell.
Thomas Gleixner April 20, 2023, 3:57 p.m. UTC | #21
On Thu, Apr 20 2023 at 07:51, Sean Christopherson wrote:
> On Thu, Apr 20, 2023, Thomas Gleixner wrote:
>> On Thu, Apr 20 2023 at 10:23, Andrew Cooper wrote:
>> > On 20/04/2023 9:32 am, Thomas Gleixner wrote:
>> > > On Wed, Apr 19, 2023, Andrew Cooper wrote:
>> > > > This was changed in x2APIC, which made the x2APIC_ID immutable.
>>
>> >> I'm pondering to simply deny parallel mode if x2APIC is not there.
>> >
>> > I'm not sure if that will help much.
>> 
>> Spoilsport.
>
> LOL, well let me pile on then.  x2APIC IDs aren't immutable on AMD hardware.  The
> ID is read-only when the CPU is in x2APIC mode, but any changes made to the ID
> while the CPU is in xAPIC mode survive the transition to x2APIC.  From the APM:
>
>   A value previously written by software to the 8-bit APIC_ID register (MMIO offset
>   30h) is converted by hardware into the appropriate format and reflected into the
>   32-bit x2APIC_ID register (MSR 802h).
>
> FWIW, my observations from testing on bare metal are that the xAPIC ID is effectively
> read-only (writes are dropped) on Intel CPUs as far back as Haswell, while the above
> behavior described in the APM holds true on at least Rome and Milan.
>
> My guess is that Intel's uArch specific behavior of the xAPIC ID being read-only
> was introduced when x2APIC came along, but I didn't test farther back than Haswell.

I'm not so worried about modern hardware. The horrorshow is the old muck
as demonstrated and of course there is virt :)

Something like the completely untested below should just work whatever
APIC ID the BIOS decided to dice.

That might just work on SEV too without that GHCB muck, but what do I
know.

Thanks,

        tglx
---
--- a/arch/x86/include/asm/apicdef.h
+++ b/arch/x86/include/asm/apicdef.h
@@ -138,7 +138,8 @@
 #define		APIC_EILVT_MASKED	(1 << 16)
 
 #define APIC_BASE (fix_to_virt(FIX_APIC_BASE))
-#define APIC_BASE_MSR	0x800
+#define APIC_BASE_MSR		0x800
+#define APIC_X2APIC_ID_MSR	0x802
 #define XAPIC_ENABLE	(1UL << 11)
 #define X2APIC_ENABLE	(1UL << 10)
 
@@ -162,6 +163,7 @@
 #define APIC_CPUID(apicid)	((apicid) & XAPIC_DEST_CPUS_MASK)
 #define NUM_APIC_CLUSTERS	((BAD_APICID + 1) >> XAPIC_DEST_CPUS_SHIFT)
 
+#ifndef __ASSEMBLY__
 /*
  * the local APIC register structure, memory mapped. Not terribly well
  * tested, but we might eventually use this one in the future - the
@@ -435,4 +437,5 @@ enum apic_delivery_modes {
 	APIC_DELIVERY_MODE_EXTINT	= 7,
 };
 
+#endif /* !__ASSEMBLY__ */
 #endif /* _ASM_X86_APICDEF_H */
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -195,14 +195,13 @@ extern void nmi_selftest(void);
 #endif
 
 extern unsigned int smpboot_control;
+extern unsigned long apic_mmio_base;
 
 #endif /* !__ASSEMBLY__ */
 
 /* Control bits for startup_64 */
-#define STARTUP_APICID_CPUID_1F 0x80000000
-#define STARTUP_APICID_CPUID_0B 0x40000000
-#define STARTUP_APICID_CPUID_01 0x20000000
-#define STARTUP_APICID_SEV_ES	0x10000000
+#define STARTUP_READ_APICID	0x80000000
+#define STARTUP_APICID_SEV_ES	0x40000000
 
 /* Top 8 bits are reserved for control */
 #define STARTUP_PARALLEL_MASK	0xFF000000
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -101,6 +101,8 @@ static int apic_extnmi __ro_after_init =
  */
 static bool virt_ext_dest_id __ro_after_init;
 
+unsigned long apic_mmio_base __ro_after_init;
+
 /*
  * Map cpu index to physical APIC ID
  */
@@ -2164,6 +2166,7 @@ void __init register_lapic_address(unsig
 
 	if (!x2apic_mode) {
 		set_fixmap_nocache(FIX_APIC_BASE, address);
+		apic_mmio_base = APIC_BASE;
 		apic_printk(APIC_VERBOSE, "mapped APIC to %16lx (%16lx)\n",
 			    APIC_BASE, address);
 	}
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -24,8 +24,10 @@
 #include "../entry/calling.h"
 #include <asm/export.h>
 #include <asm/nospec-branch.h>
+#include <asm/apicdef.h>
 #include <asm/fixmap.h>
 #include <asm/smp.h>
+
 #include <asm/sev-common.h>
 
 /*
@@ -237,37 +239,24 @@ SYM_INNER_LABEL(secondary_startup_64_no_
 
 #ifdef CONFIG_SMP
 	/*
-	 * For parallel boot, the APIC ID is retrieved from CPUID, and then
-	 * used to look up the CPU number.  For booting a single CPU, the
-	 * CPU number is encoded in smpboot_control.
+	 * For parallel boot, the APIC ID is either retrieved the APIC or
+	 * from CPUID, and then used to look up the CPU number.
+	 * For booting a single CPU, the CPU number is encoded in
+	 * smpboot_control.
 	 *
-	 * Bit 31	STARTUP_APICID_CPUID_1F flag (use CPUID 0x1f)
-	 * Bit 30	STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b)
-	 * Bit 29	STARTUP_APICID_CPUID_01 flag (use CPUID 0x01)
-	 * Bit 28	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
+	 * Bit 31	STARTUP_APICID_READ (Read APICID from APIC)
+	 * Bit 30	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
 	 * Bit 0-23	CPU# if STARTUP_APICID_CPUID_xx flags are not set
 	 */
 	movl	smpboot_control(%rip), %ecx
+	testl	$STARTUP_READ_APICID, %ecx
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 	testl	$STARTUP_APICID_SEV_ES, %ecx
 	jnz	.Luse_sev_cpuid_0b
 #endif
-	testl	$STARTUP_APICID_CPUID_1F, %ecx
-	jnz	.Luse_cpuid_1f
-	testl	$STARTUP_APICID_CPUID_0B, %ecx
-	jnz	.Luse_cpuid_0b
-	testl	$STARTUP_APICID_CPUID_01, %ecx
-	jnz	.Luse_cpuid_01
 	andl	$(~STARTUP_PARALLEL_MASK), %ecx
 	jmp	.Lsetup_cpu
 
-.Luse_cpuid_01:
-	mov	$0x01, %eax
-	cpuid
-	mov	%ebx, %edx
-	shr	$24, %edx
-	jmp	.Lsetup_AP
-
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 .Luse_sev_cpuid_0b:
 	/* Set the GHCB MSR to request CPUID 0x0B_EDX */
@@ -292,24 +281,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_
 	jmp	.Lsetup_AP
 #endif
 
-.Luse_cpuid_0b:
-	mov	$0x0B, %eax
-	xorl	%ecx, %ecx
-	cpuid
-	jmp	.Lsetup_AP
+.Lread_apicid:
+	mov	$MSR_IA32_APICBASE, %ecx
+	rdmsr
+	testl	$X2APIC_ENABLE, %eax
+	jnz	read_apicid_msr
+
+	/* Read the APIC ID from the fix-mapped MMIO space. */
+	movq	apic_mmio_base(%rip), %rcx
+	addq	$APIC_ID, %rcx
+	movl	(%rcx), %eax
+	shr	$24, %eax
+	jnz	.Lread_apicid
 
-.Luse_cpuid_1f:
-	mov	$0x1f, %eax
-	xorl	%ecx, %ecx
-	cpuid
+.Lread_apicid_msr:
+	mov	$APIC_X2APIC_ID_MSR, %ecx
+	rdmsr
 
 .Lsetup_AP:
-	/* EDX contains the APIC ID of the current CPU */
+	/* EAX contains the APIC ID of the current CPU */
 	xorq	%rcx, %rcx
 	leaq	cpuid_to_apicid(%rip), %rbx
 
 .Lfind_cpunr:
-	cmpl	(%rbx,%rcx,4), %edx
+	cmpl	(%rbx,%rcx,4), %eax
 	jz	.Lsetup_cpu
 	inc	%ecx
 #ifdef CONFIG_FORCE_NR_CPUS
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1253,41 +1253,22 @@ bool __init arch_cpuhp_init_parallel_bri
 		return false;
 	}
 
-	/* Encrypted guests require special CPUID handling. */
+	/* Encrypted guests require special handling. */
 	if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) {
 		switch (cc_get_vendor()) {
 		case CC_VENDOR_AMD:
 			ctrl = STARTUP_APICID_SEV_ES;
 			if (topology_extended_leaf == 0x0b)
-				goto setup;
+				break;
 			fallthrough;
 		default:
 			pr_info("Parallel CPU startup disabled due to guest state encryption\n");
 			return false;
 		}
+	} else {
+		ctrl = STARTUP_READ_APICID;
 	}
 
-	switch (topology_extended_leaf) {
-	case 0x0b:
-		ctrl = STARTUP_APICID_CPUID_0B;
-		break;
-	case 0x1f:
-		ctrl = STARTUP_APICID_CPUID_1F;
-		break;
-	case 0x00:
-		/* For !x2APIC mode 8 bits from leaf 0x01 are sufficient. */
-		if (!x2apic_mode) {
-			ctrl = STARTUP_APICID_CPUID_01;
-			break;
-		}
-		fallthrough;
-	default:
-		pr_info("Parallel CPU startup disabled. Unsupported topology leaf %u\n",
-			topology_extended_leaf);
-		return false;
-	}
-
-setup:
 	pr_debug("Parallel CPU startup enabled: 0x%08x\n", ctrl);
 	smpboot_control = ctrl;
 	return true;
Paul Menzel April 20, 2023, 4:47 p.m. UTC | #22
Dear Thomas,


Am 20.04.23 um 17:57 schrieb Thomas Gleixner:
> On Thu, Apr 20 2023 at 07:51, Sean Christopherson wrote:
>> On Thu, Apr 20, 2023, Thomas Gleixner wrote:
>>> On Thu, Apr 20 2023 at 10:23, Andrew Cooper wrote:
>>>> On 20/04/2023 9:32 am, Thomas Gleixner wrote:
>>>>> On Wed, Apr 19, 2023, Andrew Cooper wrote:
>>>>>> This was changed in x2APIC, which made the x2APIC_ID immutable.
>>>
>>>>> I'm pondering to simply deny parallel mode if x2APIC is not there.
>>>>
>>>> I'm not sure if that will help much.
>>>
>>> Spoilsport.
>>
>> LOL, well let me pile on then.  x2APIC IDs aren't immutable on AMD hardware.  The
>> ID is read-only when the CPU is in x2APIC mode, but any changes made to the ID
>> while the CPU is in xAPIC mode survive the transition to x2APIC.  From the APM:
>>
>>    A value previously written by software to the 8-bit APIC_ID register (MMIO offset
>>    30h) is converted by hardware into the appropriate format and reflected into the
>>    32-bit x2APIC_ID register (MSR 802h).
>>
>> FWIW, my observations from testing on bare metal are that the xAPIC ID is effectively
>> read-only (writes are dropped) on Intel CPUs as far back as Haswell, while the above
>> behavior described in the APM holds true on at least Rome and Milan.
>>
>> My guess is that Intel's uArch specific behavior of the xAPIC ID being read-only
>> was introduced when x2APIC came along, but I didn't test farther back than Haswell.
> 
> I'm not so worried about modern hardware. The horrorshow is the old muck
> as demonstrated and of course there is virt :)
> 
> Something like the completely untested below should just work whatever
> APIC ID the BIOS decided to dice.
> 
> That might just work on SEV too without that GHCB muck, but what do I
> know.
> 
> Thanks,
> 
>          tglx
> ---
> --- a/arch/x86/include/asm/apicdef.h
> +++ b/arch/x86/include/asm/apicdef.h
> @@ -138,7 +138,8 @@
>   #define		APIC_EILVT_MASKED	(1 << 16)
>   
>   #define APIC_BASE (fix_to_virt(FIX_APIC_BASE))
> -#define APIC_BASE_MSR	0x800
> +#define APIC_BASE_MSR		0x800
> +#define APIC_X2APIC_ID_MSR	0x802
>   #define XAPIC_ENABLE	(1UL << 11)
>   #define X2APIC_ENABLE	(1UL << 10)
>   
> @@ -162,6 +163,7 @@
>   #define APIC_CPUID(apicid)	((apicid) & XAPIC_DEST_CPUS_MASK)
>   #define NUM_APIC_CLUSTERS	((BAD_APICID + 1) >> XAPIC_DEST_CPUS_SHIFT)
>   
> +#ifndef __ASSEMBLY__
>   /*
>    * the local APIC register structure, memory mapped. Not terribly well
>    * tested, but we might eventually use this one in the future - the
> @@ -435,4 +437,5 @@ enum apic_delivery_modes {
>   	APIC_DELIVERY_MODE_EXTINT	= 7,
>   };
>   
> +#endif /* !__ASSEMBLY__ */
>   #endif /* _ASM_X86_APICDEF_H */
> --- a/arch/x86/include/asm/smp.h
> +++ b/arch/x86/include/asm/smp.h
> @@ -195,14 +195,13 @@ extern void nmi_selftest(void);
>   #endif
>   
>   extern unsigned int smpboot_control;
> +extern unsigned long apic_mmio_base;
>   
>   #endif /* !__ASSEMBLY__ */
>   
>   /* Control bits for startup_64 */
> -#define STARTUP_APICID_CPUID_1F 0x80000000
> -#define STARTUP_APICID_CPUID_0B 0x40000000
> -#define STARTUP_APICID_CPUID_01 0x20000000
> -#define STARTUP_APICID_SEV_ES	0x10000000
> +#define STARTUP_READ_APICID	0x80000000
> +#define STARTUP_APICID_SEV_ES	0x40000000
>   
>   /* Top 8 bits are reserved for control */
>   #define STARTUP_PARALLEL_MASK	0xFF000000
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -101,6 +101,8 @@ static int apic_extnmi __ro_after_init =
>    */
>   static bool virt_ext_dest_id __ro_after_init;
>   
> +unsigned long apic_mmio_base __ro_after_init;
> +
>   /*
>    * Map cpu index to physical APIC ID
>    */
> @@ -2164,6 +2166,7 @@ void __init register_lapic_address(unsig
>   
>   	if (!x2apic_mode) {
>   		set_fixmap_nocache(FIX_APIC_BASE, address);
> +		apic_mmio_base = APIC_BASE;
>   		apic_printk(APIC_VERBOSE, "mapped APIC to %16lx (%16lx)\n",
>   			    APIC_BASE, address);
>   	}
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -24,8 +24,10 @@
>   #include "../entry/calling.h"
>   #include <asm/export.h>
>   #include <asm/nospec-branch.h>
> +#include <asm/apicdef.h>
>   #include <asm/fixmap.h>
>   #include <asm/smp.h>
> +
>   #include <asm/sev-common.h>
>   
>   /*
> @@ -237,37 +239,24 @@ SYM_INNER_LABEL(secondary_startup_64_no_
>   
>   #ifdef CONFIG_SMP
>   	/*
> -	 * For parallel boot, the APIC ID is retrieved from CPUID, and then
> -	 * used to look up the CPU number.  For booting a single CPU, the
> -	 * CPU number is encoded in smpboot_control.
> +	 * For parallel boot, the APIC ID is either retrieved the APIC or
> +	 * from CPUID, and then used to look up the CPU number.
> +	 * For booting a single CPU, the CPU number is encoded in
> +	 * smpboot_control.
>   	 *
> -	 * Bit 31	STARTUP_APICID_CPUID_1F flag (use CPUID 0x1f)
> -	 * Bit 30	STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b)
> -	 * Bit 29	STARTUP_APICID_CPUID_01 flag (use CPUID 0x01)
> -	 * Bit 28	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
> +	 * Bit 31	STARTUP_APICID_READ (Read APICID from APIC)
> +	 * Bit 30	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
>   	 * Bit 0-23	CPU# if STARTUP_APICID_CPUID_xx flags are not set
>   	 */
>   	movl	smpboot_control(%rip), %ecx
> +	testl	$STARTUP_READ_APICID, %ecx
>   #ifdef CONFIG_AMD_MEM_ENCRYPT
>   	testl	$STARTUP_APICID_SEV_ES, %ecx
>   	jnz	.Luse_sev_cpuid_0b
>   #endif
> -	testl	$STARTUP_APICID_CPUID_1F, %ecx
> -	jnz	.Luse_cpuid_1f
> -	testl	$STARTUP_APICID_CPUID_0B, %ecx
> -	jnz	.Luse_cpuid_0b
> -	testl	$STARTUP_APICID_CPUID_01, %ecx
> -	jnz	.Luse_cpuid_01
>   	andl	$(~STARTUP_PARALLEL_MASK), %ecx
>   	jmp	.Lsetup_cpu
>   
> -.Luse_cpuid_01:
> -	mov	$0x01, %eax
> -	cpuid
> -	mov	%ebx, %edx
> -	shr	$24, %edx
> -	jmp	.Lsetup_AP
> -
>   #ifdef CONFIG_AMD_MEM_ENCRYPT
>   .Luse_sev_cpuid_0b:
>   	/* Set the GHCB MSR to request CPUID 0x0B_EDX */
> @@ -292,24 +281,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_
>   	jmp	.Lsetup_AP
>   #endif
>   
> -.Luse_cpuid_0b:
> -	mov	$0x0B, %eax
> -	xorl	%ecx, %ecx
> -	cpuid
> -	jmp	.Lsetup_AP
> +.Lread_apicid:
> +	mov	$MSR_IA32_APICBASE, %ecx
> +	rdmsr
> +	testl	$X2APIC_ENABLE, %eax
> +	jnz	read_apicid_msr
> +
> +	/* Read the APIC ID from the fix-mapped MMIO space. */
> +	movq	apic_mmio_base(%rip), %rcx
> +	addq	$APIC_ID, %rcx
> +	movl	(%rcx), %eax
> +	shr	$24, %eax
> +	jnz	.Lread_apicid
>   
> -.Luse_cpuid_1f:
> -	mov	$0x1f, %eax
> -	xorl	%ecx, %ecx
> -	cpuid
> +.Lread_apicid_msr:
> +	mov	$APIC_X2APIC_ID_MSR, %ecx
> +	rdmsr
>   
>   .Lsetup_AP:
> -	/* EDX contains the APIC ID of the current CPU */
> +	/* EAX contains the APIC ID of the current CPU */
>   	xorq	%rcx, %rcx
>   	leaq	cpuid_to_apicid(%rip), %rbx
>   
>   .Lfind_cpunr:
> -	cmpl	(%rbx,%rcx,4), %edx
> +	cmpl	(%rbx,%rcx,4), %eax
>   	jz	.Lsetup_cpu
>   	inc	%ecx
>   #ifdef CONFIG_FORCE_NR_CPUS
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1253,41 +1253,22 @@ bool __init arch_cpuhp_init_parallel_bri
>   		return false;
>   	}
>   
> -	/* Encrypted guests require special CPUID handling. */
> +	/* Encrypted guests require special handling. */
>   	if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) {
>   		switch (cc_get_vendor()) {
>   		case CC_VENDOR_AMD:
>   			ctrl = STARTUP_APICID_SEV_ES;
>   			if (topology_extended_leaf == 0x0b)
> -				goto setup;
> +				break;
>   			fallthrough;
>   		default:
>   			pr_info("Parallel CPU startup disabled due to guest state encryption\n");
>   			return false;
>   		}
> +	} else {
> +		ctrl = STARTUP_READ_APICID;
>   	}
>   
> -	switch (topology_extended_leaf) {
> -	case 0x0b:
> -		ctrl = STARTUP_APICID_CPUID_0B;
> -		break;
> -	case 0x1f:
> -		ctrl = STARTUP_APICID_CPUID_1F;
> -		break;
> -	case 0x00:
> -		/* For !x2APIC mode 8 bits from leaf 0x01 are sufficient. */
> -		if (!x2apic_mode) {
> -			ctrl = STARTUP_APICID_CPUID_01;
> -			break;
> -		}
> -		fallthrough;
> -	default:
> -		pr_info("Parallel CPU startup disabled. Unsupported topology leaf %u\n",
> -			topology_extended_leaf);
> -		return false;
> -	}
> -
> -setup:
>   	pr_debug("Parallel CPU startup enabled: 0x%08x\n", ctrl);
>   	smpboot_control = ctrl;
>   	return true;

I quickly applied it on top of your branch, but I am getting:

```
$ wget https://lore.kernel.org/lkml/87v8hq35sk.ffs@tglx/raw
$ patch -p1 < raw
$ make
[…]
   LD      .tmp_vmlinux.kallsyms1
ld: arch/x86/kernel/head_64.o: in function `secondary_startup_64_no_verify':
(.head.text+0xbf): undefined reference to `read_apicid_msr'
make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 1
make: *** [Makefile:1249: vmlinux] Error 2
```


Kind regards,

Paul
Thomas Gleixner April 20, 2023, 7:10 p.m. UTC | #23
On Thu, Apr 20 2023 at 18:47, Paul Menzel wrote:
> Am 20.04.23 um 17:57 schrieb Thomas Gleixner:
> I quickly applied it on top of your branch, but I am getting:

As I said it was untested. I was traveling and did not have access to a
machine to even build it completely. Fixed up and tested version below.

Thanks,

        tglx
---
--- a/arch/x86/include/asm/apicdef.h
+++ b/arch/x86/include/asm/apicdef.h
@@ -138,7 +138,8 @@
 #define		APIC_EILVT_MASKED	(1 << 16)
 
 #define APIC_BASE (fix_to_virt(FIX_APIC_BASE))
-#define APIC_BASE_MSR	0x800
+#define APIC_BASE_MSR		0x800
+#define APIC_X2APIC_ID_MSR	0x802
 #define XAPIC_ENABLE	(1UL << 11)
 #define X2APIC_ENABLE	(1UL << 10)
 
@@ -162,6 +163,7 @@
 #define APIC_CPUID(apicid)	((apicid) & XAPIC_DEST_CPUS_MASK)
 #define NUM_APIC_CLUSTERS	((BAD_APICID + 1) >> XAPIC_DEST_CPUS_SHIFT)
 
+#ifndef __ASSEMBLY__
 /*
  * the local APIC register structure, memory mapped. Not terribly well
  * tested, but we might eventually use this one in the future - the
@@ -435,4 +437,5 @@ enum apic_delivery_modes {
 	APIC_DELIVERY_MODE_EXTINT	= 7,
 };
 
+#endif /* !__ASSEMBLY__ */
 #endif /* _ASM_X86_APICDEF_H */
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -195,14 +195,13 @@ extern void nmi_selftest(void);
 #endif
 
 extern unsigned int smpboot_control;
+extern unsigned long apic_mmio_base;
 
 #endif /* !__ASSEMBLY__ */
 
 /* Control bits for startup_64 */
-#define STARTUP_APICID_CPUID_1F 0x80000000
-#define STARTUP_APICID_CPUID_0B 0x40000000
-#define STARTUP_APICID_CPUID_01 0x20000000
-#define STARTUP_APICID_SEV_ES	0x10000000
+#define STARTUP_READ_APICID	0x80000000
+#define STARTUP_APICID_SEV_ES	0x40000000
 
 /* Top 8 bits are reserved for control */
 #define STARTUP_PARALLEL_MASK	0xFF000000
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -101,6 +101,8 @@ static int apic_extnmi __ro_after_init =
  */
 static bool virt_ext_dest_id __ro_after_init;
 
+unsigned long apic_mmio_base __ro_after_init;
+
 /*
  * Map cpu index to physical APIC ID
  */
@@ -2164,6 +2166,7 @@ void __init register_lapic_address(unsig
 
 	if (!x2apic_mode) {
 		set_fixmap_nocache(FIX_APIC_BASE, address);
+		apic_mmio_base = APIC_BASE;
 		apic_printk(APIC_VERBOSE, "mapped APIC to %16lx (%16lx)\n",
 			    APIC_BASE, address);
 	}
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -24,8 +24,10 @@
 #include "../entry/calling.h"
 #include <asm/export.h>
 #include <asm/nospec-branch.h>
+#include <asm/apicdef.h>
 #include <asm/fixmap.h>
 #include <asm/smp.h>
+
 #include <asm/sev-common.h>
 
 /*
@@ -237,37 +239,25 @@ SYM_INNER_LABEL(secondary_startup_64_no_
 
 #ifdef CONFIG_SMP
 	/*
-	 * For parallel boot, the APIC ID is retrieved from CPUID, and then
-	 * used to look up the CPU number.  For booting a single CPU, the
-	 * CPU number is encoded in smpboot_control.
+	 * For parallel boot, the APIC ID is either retrieved the APIC or
+	 * from CPUID, and then used to look up the CPU number.
+	 * For booting a single CPU, the CPU number is encoded in
+	 * smpboot_control.
 	 *
-	 * Bit 31	STARTUP_APICID_CPUID_1F flag (use CPUID 0x1f)
-	 * Bit 30	STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b)
-	 * Bit 29	STARTUP_APICID_CPUID_01 flag (use CPUID 0x01)
-	 * Bit 28	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
+	 * Bit 31	STARTUP_APICID_READ (Read APICID from APIC)
+	 * Bit 30	STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
 	 * Bit 0-23	CPU# if STARTUP_APICID_CPUID_xx flags are not set
 	 */
 	movl	smpboot_control(%rip), %ecx
+	testl	$STARTUP_READ_APICID, %ecx
+	jnz	.Lread_apicid
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 	testl	$STARTUP_APICID_SEV_ES, %ecx
 	jnz	.Luse_sev_cpuid_0b
 #endif
-	testl	$STARTUP_APICID_CPUID_1F, %ecx
-	jnz	.Luse_cpuid_1f
-	testl	$STARTUP_APICID_CPUID_0B, %ecx
-	jnz	.Luse_cpuid_0b
-	testl	$STARTUP_APICID_CPUID_01, %ecx
-	jnz	.Luse_cpuid_01
 	andl	$(~STARTUP_PARALLEL_MASK), %ecx
 	jmp	.Lsetup_cpu
 
-.Luse_cpuid_01:
-	mov	$0x01, %eax
-	cpuid
-	mov	%ebx, %edx
-	shr	$24, %edx
-	jmp	.Lsetup_AP
-
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 .Luse_sev_cpuid_0b:
 	/* Set the GHCB MSR to request CPUID 0x0B_EDX */
@@ -292,24 +282,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_
 	jmp	.Lsetup_AP
 #endif
 
-.Luse_cpuid_0b:
-	mov	$0x0B, %eax
-	xorl	%ecx, %ecx
-	cpuid
+.Lread_apicid:
+	mov	$MSR_IA32_APICBASE, %ecx
+	rdmsr
+	testl	$X2APIC_ENABLE, %eax
+	jnz	.Lread_apicid_msr
+
+	/* Read the APIC ID from the fix-mapped MMIO space. */
+	movq	apic_mmio_base(%rip), %rcx
+	addq	$APIC_ID, %rcx
+	movl	(%rcx), %eax
+	shr	$24, %eax
 	jmp	.Lsetup_AP
 
-.Luse_cpuid_1f:
-	mov	$0x1f, %eax
-	xorl	%ecx, %ecx
-	cpuid
+.Lread_apicid_msr:
+	mov	$APIC_X2APIC_ID_MSR, %ecx
+	rdmsr
 
 .Lsetup_AP:
-	/* EDX contains the APIC ID of the current CPU */
+	/* EAX contains the APIC ID of the current CPU */
 	xorq	%rcx, %rcx
 	leaq	cpuid_to_apicid(%rip), %rbx
 
 .Lfind_cpunr:
-	cmpl	(%rbx,%rcx,4), %edx
+	cmpl	(%rbx,%rcx,4), %eax
 	jz	.Lsetup_cpu
 	inc	%ecx
 #ifdef CONFIG_FORCE_NR_CPUS
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1253,41 +1253,22 @@ bool __init arch_cpuhp_init_parallel_bri
 		return false;
 	}
 
-	/* Encrypted guests require special CPUID handling. */
+	/* Encrypted guests require special handling. */
 	if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) {
 		switch (cc_get_vendor()) {
 		case CC_VENDOR_AMD:
 			ctrl = STARTUP_APICID_SEV_ES;
 			if (topology_extended_leaf == 0x0b)
-				goto setup;
+				break;
 			fallthrough;
 		default:
 			pr_info("Parallel CPU startup disabled due to guest state encryption\n");
 			return false;
 		}
+	} else {
+		ctrl = STARTUP_READ_APICID;
 	}
 
-	switch (topology_extended_leaf) {
-	case 0x0b:
-		ctrl = STARTUP_APICID_CPUID_0B;
-		break;
-	case 0x1f:
-		ctrl = STARTUP_APICID_CPUID_1F;
-		break;
-	case 0x00:
-		/* For !x2APIC mode 8 bits from leaf 0x01 are sufficient. */
-		if (!x2apic_mode) {
-			ctrl = STARTUP_APICID_CPUID_01;
-			break;
-		}
-		fallthrough;
-	default:
-		pr_info("Parallel CPU startup disabled. Unsupported topology leaf %u\n",
-			topology_extended_leaf);
-		return false;
-	}
-
-setup:
 	pr_debug("Parallel CPU startup enabled: 0x%08x\n", ctrl);
 	smpboot_control = ctrl;
 	return true;
Thomas Gleixner April 21, 2023, 4:36 p.m. UTC | #24
On Thu, Apr 20 2023 at 21:10, Thomas Gleixner wrote:
> On Thu, Apr 20 2023 at 18:47, Paul Menzel wrote:
>> Am 20.04.23 um 17:57 schrieb Thomas Gleixner:
>> I quickly applied it on top of your branch, but I am getting:
>
> As I said it was untested. I was traveling and did not have access to a
> machine to even build it completely. Fixed up and tested version below.

I've updated

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug

for your conveniance.

Thanks,

        tglx
Paul Menzel April 24, 2023, 6:46 p.m. UTC | #25
Dear Thomas,


Am 20.04.23 um 21:10 schrieb Thomas Gleixner:
> On Thu, Apr 20 2023 at 18:47, Paul Menzel wrote:
>> Am 20.04.23 um 17:57 schrieb Thomas Gleixner:
>> I quickly applied it on top of your branch, but I am getting:
> 
> As I said it was untested. I was traveling and did not have access to a
> machine to even build it completely. Fixed up and tested version below.

Sorry, if it sounded like a complaint. I just wanted to give a quick 
feedback.

[…]

I tested your new version even on Friday, and it worked fine – no ten 
seconds delay. Please find the messages attached.

Thank you all for your great work.


Kind regards,

Paul


PS: I am going to try to test your updated branch at the end of the week.
[    0.000000] Linux version 6.3.0-rc3-00046-g8ba643d7e1c7 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #452 SMP PREEMPT_DYNAMIC Thu Apr 20 20:15:01 UTC 2023
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00046-g8ba643d7e1c7 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] signal: max sigframe size: 1776
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005fe4cfff] usable
[    0.000000] BIOS-e820: [mem 0x000000005fe4d000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000017effffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0.0 present.
[    0.000000] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-gb640ed51b2 04/17/2023
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 3900.440 MHz processor
[    0.000756] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000759] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x400000000
[    0.000768] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000938] last_pfn = 0x5fe4d max_arch_pfn = 0x400000000
[    0.004000] Using GB pages for direct mapping
[    0.004000] ACPI: Early table checksum verification disabled
[    0.004000] ACPI: RSDP 0x00000000000F6830 000024 (v02 COREv4)
[    0.004000] ACPI: XSDT 0x000000005FE5A0E0 000074 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: FACP 0x000000005FE5BBC0 000114 (v06 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: DSDT 0x000000005FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925)
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: FACS 0x000000005FE5A240 000040
[    0.004000] ACPI: SSDT 0x000000005FE5BCE0 00008A (v02 COREv4 COREBOOT 0000002A CORE 20200925)
[    0.004000] ACPI: MCFG 0x000000005FE5BD70 00003C (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: APIC 0x000000005FE5BDB0 000062 (v03 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HPET 0x000000005FE5BE20 000038 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: HEST 0x000000005FE5BE60 0001D0 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: IVRS 0x000000005FE5C030 000070 (v02 AMD    AMDIOMMU 00000001 AMD  00000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C0A0 00051F (v02 AMD    ALIB     00000001 MSFT 04000000)
[    0.004000] ACPI: SSDT 0x000000005FE5C5C0 0006B2 (v01 AMD    POWERNOW 00000001 AMD  00000001)
[    0.004000] ACPI: VFCT 0x000000005FE5CC80 00F269 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[    0.004000] ACPI: Reserving FACP table memory at [mem 0x5fe5bbc0-0x5fe5bcd3]
[    0.004000] ACPI: Reserving DSDT table memory at [mem 0x5fe5a280-0x5fe5bbb9]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5bce0-0x5fe5bd69]
[    0.004000] ACPI: Reserving MCFG table memory at [mem 0x5fe5bd70-0x5fe5bdab]
[    0.004000] ACPI: Reserving APIC table memory at [mem 0x5fe5bdb0-0x5fe5be11]
[    0.004000] ACPI: Reserving HPET table memory at [mem 0x5fe5be20-0x5fe5be57]
[    0.004000] ACPI: Reserving HEST table memory at [mem 0x5fe5be60-0x5fe5c02f]
[    0.004000] ACPI: Reserving IVRS table memory at [mem 0x5fe5c030-0x5fe5c09f]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c0a0-0x5fe5c5be]
[    0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5c5c0-0x5fe5cc71]
[    0.004000] ACPI: Reserving VFCT table memory at [mem 0x5fe5cc80-0x5fe6bee8]
[    0.004000] No NUMA configuration found
[    0.004000] Faking a node at [mem 0x0000000000000000-0x000000017effffff]
[    0.004000] NODE_DATA(0) allocated [mem 0x17efe9000-0x17effffff]
[    0.004000] Zone ranges:
[    0.004000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.004000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.004000]   Normal   [mem 0x0000000100000000-0x000000017effffff]
[    0.004000]   Device   empty
[    0.004000] Movable zone start for each node
[    0.004000] Early memory node ranges
[    0.004000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.004000]   node   0: [mem 0x0000000000100000-0x000000005fe4cfff]
[    0.004000]   node   0: [mem 0x0000000100000000-0x000000017effffff]
[    0.004000] Initmem setup node 0 [mem 0x0000000000001000-0x000000017effffff]
[    0.004000] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.004000] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 435 pages in unavailable ranges
[    0.004000] On node 0, zone Normal: 4096 pages in unavailable ranges
[    0.004000] ACPI: PM-Timer IO Port: 0x818
[    0.004000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.004000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.004000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.004000] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.004000] ACPI: HPET id: 0x10228210 base: 0xfed00000
[    0.004000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.004000] [mem 0x80000000-0xf7ffffff] available for PCI devices
[    0.004000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.004000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.004000] percpu: Embedded 55 pages/cpu s188392 r8192 d28696 u1048576
[    0.004000] pcpu-alloc: s188392 r8192 d28696 u1048576 alloc=1*2097152
[    0.004000] pcpu-alloc: [0] 0 1 
[    0.004000] Fallback order for Node 0: 0 
[    0.004000] Built 1 zonelists, mobility grouping on.  Total pages: 898451
[    0.004000] Policy zone: Normal
[    0.004000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00046-g8ba643d7e1c7 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0
[    0.004000] Unknown kernel command line parameters "noisapnp BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00046-g8ba643d7e1c7", will be passed to user space.
[    0.004000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.004000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.004000] stackdepot: allocating hash table via alloc_large_system_hash
[    0.004000] stackdepot hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.004000] software IO TLB: area num 2.
[    0.004000] Memory: 3477168K/3651500K available (14336K kernel code, 2340K rwdata, 5308K rodata, 2908K init, 11064K bss, 174072K reserved, 0K cma-reserved)
[    0.004000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.004000] ftrace: allocating 38652 entries in 151 pages
[    0.004000] ftrace: allocated 151 pages with 5 groups
[    0.004000] Dynamic Preempt: full
[    0.004000] rcu: Preemptible hierarchical RCU implementation.
[    0.004000] rcu: 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[    0.004000] 	Trampoline variant of Tasks RCU enabled.
[    0.004000] 	Rude variant of Tasks RCU enabled.
[    0.004000] 	Tracing variant of Tasks RCU enabled.
[    0.004000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.004000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.004000] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
[    0.004000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.004000] spurious 8259A interrupt: IRQ7.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] printk: console [tty0] enabled
[    0.004000] ACPI: Core revision 20221020
[    0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[    0.004000] APIC: Switch to symmetric I/O mode setup
[    0.004000] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[    0.004000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.004000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x7071f4ed18f, max_idle_ns: 881590420850 ns
[    0.145332] Calibrating delay loop (skipped), value calculated using timer frequency.. 7800.88 BogoMIPS (lpj=15601760)
[    0.145336] pid_max: default: 32768 minimum: 301
[    0.145430] LSM: initializing lsm=capability
[    0.145526] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145542] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.145940] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512
[    0.145943] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512, 1GB 0
[    0.145947] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.145950] Spectre V2 : Mitigation: Retpolines
[    0.145951] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.145952] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.145952] Spectre V2 : Enabling Speculation Barrier for firmware calls
[    0.145953] RETBleed: Mitigation: untrained return thunk
[    0.145955] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.145957] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[    0.150469] Freeing SMP alternatives memory: 32K
[    0.258608] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[    0.258845] cblist_init_generic: Setting adjustable number of callback queues.
[    0.258846] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258878] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258905] cblist_init_generic: Setting shift to 1 and lim to 1.
[    0.258933] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    0.258955] ... version:                0
[    0.258956] ... bit width:              48
[    0.258957] ... generic registers:      6
[    0.258958] ... value mask:             0000ffffffffffff
[    0.258959] ... max period:             00007fffffffffff
[    0.258960] ... fixed-purpose events:   0
[    0.258961] ... event mask:             000000000000003f
[    0.259083] rcu: Hierarchical SRCU implementation.
[    0.259084] rcu: 	Max phase no-delay instances is 1000.
[    0.259676] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    0.259750] smp: Bringing up secondary CPUs ...
[    0.259952] x86: Booting SMP configuration:
[    0.259953] .... node  #0, CPUs:      #1
[    0.259958] smpboot: Kicking AP alive: 17
[    0.260088] smp: Brought up 1 node, 2 CPUs
[    0.260088] smpboot: Max logical packages: 1
[    0.260088] smpboot: Total of 2 processors activated (15601.76 BogoMIPS)
[    0.261513] devtmpfs: initialized
[    0.261513] x86/mm: Memory block size: 128MB
[    0.262387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.262387] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.262387] pinctrl core: initialized pinctrl subsystem
[    0.262387] PM: RTC time: 20:56:07, date: 2023-04-21
[    0.262387] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.262600] audit: initializing netlink subsys (disabled)
[    0.262617] audit: type=2000 audit(1682110567.140:1): state=initialized audit_enabled=0 res=1
[    0.262617] thermal_sys: Registered thermal governor 'fair_share'
[    0.262617] thermal_sys: Registered thermal governor 'bang_bang'
[    0.262617] thermal_sys: Registered thermal governor 'step_wise'
[    0.262617] thermal_sys: Registered thermal governor 'user_space'
[    0.262617] cpuidle: using governor ladder
[    0.262617] cpuidle: using governor menu
[    0.262617] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
[    0.262617] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved as E820 entry
[    0.262617] PCI: Using configuration type 1 for base access
[    0.262617] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.273356] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.273356] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.273356] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.273356] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.280509] cryptd: max_cpu_qlen set to 1000
[    0.280509] ACPI: Added _OSI(Module Device)
[    0.280509] ACPI: Added _OSI(Processor Device)
[    0.280509] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.280509] ACPI: Added _OSI(Processor Aggregator Device)
[    0.287039] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    0.288072] ACPI: Interpreter enabled
[    0.288072] ACPI: PM: (supports S0 S1 S3 S5)
[    0.288072] ACPI: Using IOAPIC for interrupt routing
[    0.288072] HEST: Table parsing has been initialized.
[    0.288072] GHES: Failed to enable APEI firmware first mode.
[    0.288072] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.288072] PCI: Ignoring E820 reservations for host bridge windows
[    0.288072] ACPI: Enabled 8 GPEs in block 00 to 1F
[    0.290922] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.290932] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.291018] acpi PNP0A03:00: _OSC: OS now controls [PME AER PCIeCapability LTR]
[    0.291034] acpi PNP0A03:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge
[    0.291117] acpi PNP0A03:00: host bridge window expanded to [io  0x0000-0x0cf7 window]; [io  0x03b0-0x03df window] ignored
[    0.291369] PCI host bridge to bus 0000:00
[    0.291371] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.291374] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.291376] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000dffff]
[    0.291379] pci_bus 0000:00: root bus resource [mem 0x80000000-0xffffffff]
[    0.291381] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.291406] pci 0000:00:00.0: [1022:1410] type 00 class 0x060000
[    0.291559] pci 0000:00:00.2: [1022:1419] type 00 class 0x080600
[    0.291648] pci 0000:00:01.0: [1002:9996] type 00 class 0x030000
[    0.291656] pci 0000:00:01.0: reg 0x10: [mem 0xe0000000-0xefffffff pref]
[    0.291661] pci 0000:00:01.0: reg 0x14: [io  0x1000-0x10ff]
[    0.291665] pci 0000:00:01.0: reg 0x18: [mem 0xf0180000-0xf01bffff]
[    0.291681] pci 0000:00:01.0: enabling Extended Tags
[    0.291694] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.291711] pci 0000:00:01.0: supports D1 D2
[    0.291776] pci 0000:00:01.1: [1002:9902] type 00 class 0x040300
[    0.291784] pci 0000:00:01.1: reg 0x10: [mem 0xf01c0000-0xf01c3fff]
[    0.291805] pci 0000:00:01.1: enabling Extended Tags
[    0.291828] pci 0000:00:01.1: supports D1 D2
[    0.291918] pci 0000:00:11.0: [1022:7801] type 00 class 0x010601
[    0.291931] pci 0000:00:11.0: reg 0x10: [io  0x1410-0x1417]
[    0.291938] pci 0000:00:11.0: reg 0x14: [io  0x1420-0x1423]
[    0.291946] pci 0000:00:11.0: reg 0x18: [io  0x1418-0x141f]
[    0.291953] pci 0000:00:11.0: reg 0x1c: [io  0x1424-0x1427]
[    0.291960] pci 0000:00:11.0: reg 0x20: [io  0x1400-0x140f]
[    0.291968] pci 0000:00:11.0: reg 0x24: [mem 0xf01cc000-0xf01cc7ff]
[    0.292125] pci 0000:00:12.0: [1022:7807] type 00 class 0x0c0310
[    0.292139] pci 0000:00:12.0: reg 0x10: [mem 0xf01c8000-0xf01c8fff]
[    0.292322] pci 0000:00:12.2: [1022:7808] type 00 class 0x0c0320
[    0.292335] pci 0000:00:12.2: reg 0x10: [mem 0xf01cd000-0xf01cd0ff]
[    0.292400] pci 0000:00:12.2: supports D1 D2
[    0.292402] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    0.292542] pci 0000:00:13.0: [1022:7807] type 00 class 0x0c0310
[    0.292555] pci 0000:00:13.0: reg 0x10: [mem 0xf01c9000-0xf01c9fff]
[    0.292737] pci 0000:00:13.2: [1022:7808] type 00 class 0x0c0320
[    0.292750] pci 0000:00:13.2: reg 0x10: [mem 0xf01ce000-0xf01ce0ff]
[    0.292815] pci 0000:00:13.2: supports D1 D2
[    0.292816] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    0.292955] pci 0000:00:14.0: [1022:780b] type 00 class 0x0c0500
[    0.293139] pci 0000:00:14.2: [1022:780d] type 00 class 0x040300
[    0.293156] pci 0000:00:14.2: reg 0x10: [mem 0xf01c4000-0xf01c7fff 64bit]
[    0.293210] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[    0.293355] pci 0000:00:14.3: [1022:780e] type 00 class 0x060100
[    0.293541] pci 0000:00:14.4: [1022:780f] type 01 class 0x060401
[    0.293703] pci 0000:00:14.5: [1022:7809] type 00 class 0x0c0310
[    0.293717] pci 0000:00:14.5: reg 0x10: [mem 0xf01ca000-0xf01cafff]
[    0.293906] pci 0000:00:15.0: [1022:43a0] type 01 class 0x060400
[    0.293935] pci 0000:00:15.0: enabling Extended Tags
[    0.293975] pci 0000:00:15.0: supports D1 D2
[    0.294151] pci 0000:00:15.1: [1022:43a1] type 01 class 0x060400
[    0.294182] pci 0000:00:15.1: enabling Extended Tags
[    0.294221] pci 0000:00:15.1: supports D1 D2
[    0.294398] pci 0000:00:15.2: [1022:43a2] type 01 class 0x060400
[    0.294426] pci 0000:00:15.2: enabling Extended Tags
[    0.294468] pci 0000:00:15.2: supports D1 D2
[    0.294547] pci 0000:00:16.0: [1022:7807] type 00 class 0x0c0310
[    0.294560] pci 0000:00:16.0: reg 0x10: [mem 0xf01cb000-0xf01cbfff]
[    0.294737] pci 0000:00:16.2: [1022:7808] type 00 class 0x0c0320
[    0.294751] pci 0000:00:16.2: reg 0x10: [mem 0xf01cf000-0xf01cf0ff]
[    0.294815] pci 0000:00:16.2: supports D1 D2
[    0.294817] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[    0.294958] pci 0000:00:18.0: [1022:1400] type 00 class 0x060000
[    0.295024] pci 0000:00:18.1: [1022:1401] type 00 class 0x060000
[    0.295086] pci 0000:00:18.2: [1022:1402] type 00 class 0x060000
[    0.295152] pci 0000:00:18.3: [1022:1403] type 00 class 0x060000
[    0.295289] pci 0000:00:18.4: [1022:1404] type 00 class 0x060000
[    0.295353] pci 0000:00:18.5: [1022:1405] type 00 class 0x060000
[    0.295436] pci_bus 0000:01: extended config space not accessible
[    0.295499] pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
[    0.295510] pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
[    0.295513] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff window] (subtractive decode)
[    0.295515] pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000dffff] (subtractive decode)
[    0.295517] pci 0000:00:14.4:   bridge window [mem 0x80000000-0xffffffff] (subtractive decode)
[    0.295571] pci 0000:00:15.0: PCI bridge to [bus 02]
[    0.295653] pci 0000:03:00.0: [1b21:1042] type 00 class 0x0c0330
[    0.295689] pci 0000:03:00.0: reg 0x10: [mem 0xf0000000-0xf0007fff 64bit]
[    0.295863] pci 0000:03:00.0: PME# supported from D3hot D3cold
[    0.295907] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:15.1 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    0.309393] pci 0000:00:15.1: PCI bridge to [bus 03]
[    0.309405] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[    0.309414] pci 0000:00:15.2: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.309545] pci 0000:04:00.0: [10ec:8168] type 00 class 0x020000
[    0.309563] pci 0000:04:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.309585] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit pref]
[    0.309598] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[    0.309705] pci 0000:04:00.0: supports D1 D2
[    0.309707] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.325403] pci 0000:00:15.2: PCI bridge to [bus 04-ff]
[    0.325414] pci 0000:00:15.2:   bridge window [io  0x0000-0x0fff]
[    0.325417] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff]
[    0.325422] pci 0000:00:15.2:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    0.325425] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[    0.325919] ACPI: PCI: Interrupt link INTA configured for IRQ 0
[    0.326011] ACPI: PCI: Interrupt link INTB configured for IRQ 0
[    0.326102] ACPI: PCI: Interrupt link INTC configured for IRQ 0
[    0.326193] ACPI: PCI: Interrupt link INTD configured for IRQ 0
[    0.326284] ACPI: PCI: Interrupt link INTE configured for IRQ 0
[    0.326375] ACPI: PCI: Interrupt link INTF configured for IRQ 0
[    0.326465] ACPI: PCI: Interrupt link INTG configured for IRQ 0
[    0.326556] ACPI: PCI: Interrupt link INTH configured for IRQ 0
[    0.326777] iommu: Default domain type: Translated 
[    0.326778] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.326974] SCSI subsystem initialized
[    0.329389] libata version 3.00 loaded.
[    0.329395] ACPI: bus type USB registered
[    0.329418] usbcore: registered new interface driver usbfs
[    0.329428] usbcore: registered new interface driver hub
[    0.329437] usbcore: registered new device driver usb
[    0.329564] PCI: Using ACPI for IRQ routing
[    0.331132] PCI: pci_cache_line_size set to 64 bytes
[    0.331184] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    0.331187] e820: reserve RAM buffer [mem 0x5fe4d000-0x5fffffff]
[    0.331189] e820: reserve RAM buffer [mem 0x17f000000-0x17fffffff]
[    0.331235] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.331240] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    0.334410] clocksource: Switched to clocksource tsc-early
[    0.334656] VFS: Disk quotas dquot_6.6.0
[    0.334682] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.334803] pnp: PnP ACPI init
[    0.335147] system 00:00: [mem 0xfec10002-0xfec11001] could not be reserved
[    0.335525] pnp: PnP ACPI: found 2 devices
[    0.342072] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.342260] NET: Registered PF_INET protocol family
[    0.342434] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.344041] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.344058] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.344070] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.344141] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    0.344505] TCP: Hash tables configured (established 32768 bind 32768)
[    0.344576] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.344601] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.344722] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.344757] pci 0000:00:15.2: BAR 15: assigned [mem 0x80000000-0x800fffff 64bit pref]
[    0.344763] pci 0000:00:15.2: BAR 13: assigned [io  0x2000-0x2fff]
[    0.344769] pci 0000:00:14.4: PCI bridge to [bus 01]
[    0.344781] pci 0000:00:15.0: PCI bridge to [bus 02]
[    0.344790] pci 0000:00:15.1: PCI bridge to [bus 03]
[    0.344793] pci 0000:00:15.1:   bridge window [mem 0xf0000000-0xf00fffff]
[    0.344807] pci 0000:04:00.0: BAR 4: assigned [mem 0x80000000-0x80003fff 64bit pref]
[    0.344821] pci 0000:04:00.0: BAR 2: assigned [mem 0x80004000-0x80004fff 64bit pref]
[    0.344833] pci 0000:04:00.0: BAR 0: assigned [io  0x2000-0x20ff]
[    0.344840] pci 0000:00:15.2: PCI bridge to [bus 04]
[    0.344842] pci 0000:00:15.2:   bridge window [io  0x2000-0x2fff]
[    0.344847] pci 0000:00:15.2:   bridge window [mem 0x80000000-0x800fffff 64bit pref]
[    0.344854] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.344857] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.344860] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000dffff]
[    0.344862] pci_bus 0000:00: resource 7 [mem 0x80000000-0xffffffff]
[    0.344864] pci_bus 0000:01: resource 4 [io  0x0000-0x0cf7 window]
[    0.344867] pci_bus 0000:01: resource 5 [io  0x0d00-0xffff window]
[    0.344869] pci_bus 0000:01: resource 6 [mem 0x000a0000-0x000dffff]
[    0.344871] pci_bus 0000:01: resource 7 [mem 0x80000000-0xffffffff]
[    0.344873] pci_bus 0000:03: resource 1 [mem 0xf0000000-0xf00fffff]
[    0.344875] pci_bus 0000:04: resource 0 [io  0x2000-0x2fff]
[    0.344877] pci_bus 0000:04: resource 2 [mem 0x80000000-0x800fffff 64bit pref]
[    0.345003] pci 0000:00:01.1: D0 power state depends on 0000:00:01.0
[    0.345887] pci 0000:00:12.2: PME# does not work under D3, disabling it
[    0.346671] pci 0000:00:13.2: PME# does not work under D3, disabling it
[    0.347980] pci 0000:00:16.2: PME# does not work under D3, disabling it
[    0.348353] PCI: CLS 64 bytes, default 64
[    0.348449] pci 0000:00:00.2: AMD-Vi: Applying erratum 746 workaround
[    0.348560] pci 0000:00:01.0: Adding to iommu group 0
[    0.348584] pci 0000:00:01.1: Adding to iommu group 0
[    0.348616] pci 0000:00:11.0: Adding to iommu group 1
[    0.348659] pci 0000:00:12.0: Adding to iommu group 2
[    0.348681] pci 0000:00:12.2: Adding to iommu group 2
[    0.348723] pci 0000:00:13.0: Adding to iommu group 3
[    0.348743] pci 0000:00:13.2: Adding to iommu group 3
[    0.348792] pci 0000:00:14.0: Adding to iommu group 4
[    0.348817] pci 0000:00:14.2: Adding to iommu group 4
[    0.348840] pci 0000:00:14.3: Adding to iommu group 4
[    0.348879] pci 0000:00:14.4: Adding to iommu group 5
[    0.348903] pci 0000:00:14.5: Adding to iommu group 6
[    0.348946] pci 0000:00:15.0: Adding to iommu group 7
[    0.348974] pci 0000:00:15.1: Adding to iommu group 7
[    0.349000] pci 0000:00:15.2: Adding to iommu group 7
[    0.349040] pci 0000:00:16.0: Adding to iommu group 8
[    0.349063] pci 0000:00:16.2: Adding to iommu group 8
[    0.349128] pci 0000:00:18.0: Adding to iommu group 9
[    0.349150] pci 0000:00:18.1: Adding to iommu group 9
[    0.349188] pci 0000:00:18.2: Adding to iommu group 9
[    0.349210] pci 0000:00:18.3: Adding to iommu group 9
[    0.349234] pci 0000:00:18.4: Adding to iommu group 9
[    0.349255] pci 0000:00:18.5: Adding to iommu group 9
[    0.349269] pci 0000:03:00.0: Adding to iommu group 7
[    0.349279] pci 0000:04:00.0: Adding to iommu group 7
[    0.352005] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.352011] AMD-Vi: Extended features (0x800000853, 0x0): PreF PPR GT IA
[    0.352020] AMD-Vi: Interrupt remapping enabled
[    0.352238] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.352240] software IO TLB: mapped [mem 0x000000005be4d000-0x000000005fe4d000] (64MB)
[    0.352297] LVT offset 0 assigned for vector 0x400
[    0.352343] perf: AMD IBS detected (0x000000ff)
[    0.352351] amd_uncore: 4  amd_nb counters detected
[    0.356598] workingset: timestamp_bits=37 max_order=20 bucket_order=0
[    0.356626] zbud: loaded
[    0.357112] NET: Registered PF_ALG protocol family
[    0.357118] Key type asymmetric registered
[    0.357119] Asymmetric key parser 'x509' registered
[    0.357471] alg: self-tests disabled
[    0.357567] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.357610] io scheduler mq-deadline registered
[    0.357612] io scheduler kyber registered
[    0.358782] pcieport 0000:00:15.0: PME: Signaling with IRQ 25
[    0.358948] pcieport 0000:00:15.1: PME: Signaling with IRQ 26
[    0.359013] pcieport 0000:00:15.2: enabling device (0000 -> 0003)
[    0.359228] pcieport 0000:00:15.2: PME: Signaling with IRQ 27
[    0.359503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.359567] ACPI: button: Power Button [PWRF]
[    0.359621] ACPI: \_SB_.P000: Found 2 idle states
[    0.359743] ACPI: \_SB_.P001: Found 2 idle states
[    0.360670] thermal LNXTHERM:00: registered as thermal_zone0
[    0.360673] ACPI: thermal: Thermal Zone [TZ00] (15 C)
[    0.360984] Non-volatile memory driver v1.3
[    0.361051] AMD-Vi: AMD IOMMUv2 loaded and initialized
[    0.361079] ACPI: bus type drm_connector registered
[    0.361292] ahci 0000:00:11.0: version 3.0
[    0.361580] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 8 ports 6 Gbps 0x40 impl SATA mode
[    0.361584] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck led clo pio 
[    0.362741] scsi host0: ahci
[    0.362967] scsi host1: ahci
[    0.363157] scsi host2: ahci
[    0.363372] scsi host3: ahci
[    0.363571] scsi host4: ahci
[    0.363773] scsi host5: ahci
[    0.363984] scsi host6: ahci
[    0.364182] scsi host7: ahci
[    0.364275] ata1: DUMMY
[    0.364277] ata2: DUMMY
[    0.364278] ata3: DUMMY
[    0.364278] ata4: DUMMY
[    0.364279] ata5: DUMMY
[    0.364280] ata6: DUMMY
[    0.364282] ata7: SATA max UDMA/133 abar m2048@0xf01cc000 port 0xf01cc400 irq 19
[    0.364284] ata8: DUMMY
[    0.364564] i8042: PNP: No PS/2 controller found.
[    0.364565] i8042: Probing ports directly.
[    0.366981] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.366988] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.367123] mousedev: PS/2 mouse device common for all mice
[    0.367188] rtc_cmos 00:01: RTC can wake from S4
[    0.367487] rtc_cmos 00:01: registered as rtc0
[    0.367510] rtc_cmos 00:01: setting system clock to 2023-04-21T20:56:07 UTC (1682110567)
[    0.367553] rtc_cmos 00:01: alarms up to one day, y3k, 114 bytes nvram, hpet irqs
[    0.367590] device-mapper: uevent: version 1.0.3
[    0.367665] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com
[    0.367827] hid: raw HID events driver (C) Jiri Kosina
[    0.367913] usbcore: registered new interface driver usbhid
[    0.367914] usbhid: USB HID core driver
[    0.368010] Initializing XFRM netlink socket
[    0.368020] NET: Registered PF_PACKET protocol family
[    0.368021] x86/pm: family 0x15 cpu detected, MSR saving is needed during suspending.
[    0.368340] microcode: CPU1: patch_level=0x0600111f
[    0.368342] microcode: CPU0: patch_level=0x0600111f
[    0.368353] microcode: Microcode Update Driver: v2.2.
[    0.368358] IPI shorthand broadcast: enabled
[    0.368370] AVX version of gcm_enc/dec engaged.
[    0.368401] AES CTR mode by8 optimization enabled
[    0.372495] sched_clock: Marking stable (254511749, 117328013)->(374590709, -2750947)
[    0.372745] registered taskstats version 1
[    0.372987] zswap: loaded using pool lzo/zbud
[    0.377461] kmemleak: Kernel memory leak detector initialized (mem pool available: 15679)
[    0.377464] kmemleak: Automatic memory scanning thread started
[    0.377465] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.382250] Key type encrypted registered
[    0.385395] PM:   Magic number: 3:579:953
[    0.385449] memory memory6: hash matches
[    0.483426] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    0.483587] ata7.00: ATA-9: SanDisk SDSSDP064G, 2.0.0, max UDMA/133
[    0.483590] ata7.00: 125045424 sectors, multi 1: LBA48 NCQ (depth 32)
[    0.483792] ata7.00: configured for UDMA/133
[    0.484004] scsi 6:0:0:0: Direct-Access     ATA      SanDisk SDSSDP06 0    PQ: 0 ANSI: 5
[    0.485006] sd 6:0:0:0: [sda] 125045424 512-byte logical blocks: (64.0 GB/59.6 GiB)
[    0.485028] sd 6:0:0:0: [sda] Write Protect is off
[    0.485033] sd 6:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    0.485062] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.485103] sd 6:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[    0.486554]  sda: sda1 sda2 sda3
[    0.487033] sd 6:0:0:0: [sda] Attached SCSI disk
[    0.508543] EXT4-fs (sda3): mounted filesystem fe29e0dc-6303-4401-987c-8472bc1b9516 with ordered data mode. Quota mode: none.
[    0.508591] VFS: Mounted root (ext4 filesystem) on device 8:3.
[    0.510618] devtmpfs: mounted
[    0.514506] Freeing unused kernel image (initmem) memory: 2908K
[    0.532141] Write protecting the kernel read-only data: 20480k
[    0.532425] Freeing unused kernel image (rodata/data gap) memory: 836K
[    0.569736] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    0.569742] rodata_test: all tests were successful
[    0.569773] Run /sbin/init as init process
[    0.569775]   with arguments:
[    0.569776]     /sbin/init
[    0.569777]     noisapnp
[    0.569778]   with environment:
[    0.569779]     HOME=/
[    0.569780]     TERM=linux
[    0.569781]     BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00046-g8ba643d7e1c7
[    0.751269] systemd[1]: Inserted module 'autofs4'
[    0.777767] NET: Registered PF_INET6 protocol family
[    0.778674] Segment Routing with IPv6
[    0.778702] In-situ OAM (IOAM) with IPv6
[    0.804133] systemd[1]: systemd 252.6-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.804143] systemd[1]: Detected architecture x86-64.
[    0.808903] systemd[1]: Hostname set to <kodi>.
[    1.081619] systemd[1]: Queued start job for default target graphical.target.
[    1.102404] systemd[1]: Created slice system-getty.slice - Slice /system/getty.
[    1.103505] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[    1.104345] systemd[1]: Created slice user.slice - User and Session Slice.
[    1.104531] systemd[1]: Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[    1.104655] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[    1.105082] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    1.105122] systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[    1.105171] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[    1.105213] systemd[1]: Reached target paths.target - Path Units.
[    1.105248] systemd[1]: Reached target remote-fs.target - Remote File Systems.
[    1.105276] systemd[1]: Reached target slices.target - Slice Units.
[    1.105302] systemd[1]: Reached target swap.target - Swaps.
[    1.105344] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[    1.107911] systemd[1]: Listening on systemd-coredump.socket - Process Core Dump Socket.
[    1.108154] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[    1.108318] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[    1.108624] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[    1.108886] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    1.109157] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[    1.109414] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[    1.110215] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[    1.110475] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[    1.113168] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    1.116585] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    1.121085] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    1.124574] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    1.128227] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[    1.132312] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[    1.135610] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[    1.138876] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[    1.147425] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[    1.150601] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[    1.153921] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
[    1.154024] systemd[1]: systemd-firstboot.service - First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    1.154112] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathIsReadWrite=!/).
[    1.154158] systemd[1]: Reached target local-fs.target - Local File Systems.
[    1.154259] systemd[1]: apparmor.service - Load AppArmor profiles was skipped because of an unmet condition check (ConditionSecurity=apparmor).
[    1.167112] fuse: init (API version 7.38)
[    1.167255] loop: module loaded
[    1.168586] systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[    1.174035] systemd[1]: Starting systemd-journald.service - Journal Service...
[    1.177353] systemd[1]: Starting systemd-random-seed.service - Load/Save Random Seed...
[    1.180848] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[    1.195025] systemd[1]: Starting systemd-sysusers.service - Create System Users...
[    1.201956] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[    1.217695] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    1.218409] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    1.218683] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    1.218957] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    1.225546] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[    1.226534] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    1.226893] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[    1.227645] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    1.228209] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    1.230383] systemd[1]: modprobe@drm.service: Deactivated successfully.
[    1.230725] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[    1.231459] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[    1.231774] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[    1.232474] systemd[1]: modprobe@fuse.service: Deactivated successfully.
[    1.232795] systemd[1]: Finished modprobe@fuse.service - Load Kernel Module fuse.
[    1.234105] systemd[1]: modprobe@loop.service: Deactivated successfully.
[    1.234462] systemd[1]: Finished modprobe@loop.service - Load Kernel Module loop.
[    1.235099] systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 140 (systemd-binfmt)
[    1.261703] systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[    1.270160] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[    1.273274] systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
[    1.273402] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[    1.273574] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[    1.290481] systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.
[    1.294159] systemd[1]: Finished systemd-sysusers.service - Create System Users.
[    1.294555] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[    1.303356] systemd[1]: Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[    1.304817] systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
[    1.320909] systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[    1.323560] systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[    1.369356] tsc: Refined TSC clocksource calibration: 3900.224 MHz
[    1.369367] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x70705a6472c, max_idle_ns: 881590586812 ns
[    1.370036] systemd[1]: Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
[    1.373207] clocksource: Switched to clocksource tsc
[    1.389831] systemd[1]: Starting systemd-udevd.service - Rule-based Manager for Device Events and Files...
[    1.427455] systemd[1]: Started systemd-journald.service - Journal Service.
[    1.472737] systemd-journald[142]: Received client request to flush runtime journal.
[    1.742223] sd 6:0:0:0: Attached scsi generic sg0 type 0
[    1.866042] acpi_cpufreq: overriding BIOS provided _PSD data
[    2.005347] random: crng init done
[    2.137832] QUIRK: Enable AMD PLL fix
[    2.137832] QUIRK: Enable AMD PLL fix
[    2.137887] ehci-pci 0000:00:13.2: EHCI Host Controller
[    2.137922] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 1
[    2.137936] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.137945] ehci-pci 0000:00:13.2: debug port 1
[    2.138141] ehci-pci 0000:00:13.2: irq 17, io mem 0xf01ce000
[    2.139491] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[    2.139505] piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection
[    2.140107] piix4_smbus 0000:00:14.0: Auxiliary SMBus Host Controller at 0xb20
[    2.154632] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00
[    2.155284] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.155291] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.155294] usb usb1: Product: EHCI Host Controller
[    2.155296] usb usb1: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ehci_hcd
[    2.155299] usb usb1: SerialNumber: 0000:00:13.2
[    2.174521] hub 1-0:1.0: USB hub found
[    2.174560] hub 1-0:1.0: 5 ports detected
[    2.175582] ehci-pci 0000:00:12.2: EHCI Host Controller
[    2.175610] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 2
[    2.175623] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.175632] ehci-pci 0000:00:12.2: debug port 1
[    2.175789] ehci-pci 0000:00:12.2: irq 17, io mem 0xf01cd000
[    2.189373] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00
[    2.189720] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.189725] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.189727] usb usb2: Product: EHCI Host Controller
[    2.189730] usb usb2: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ehci_hcd
[    2.189732] usb usb2: SerialNumber: 0000:00:12.2
[    2.190283] hub 2-0:1.0: USB hub found
[    2.190323] hub 2-0:1.0: 5 ports detected
[    2.191801] ehci-pci 0000:00:16.2: EHCI Host Controller
[    2.191831] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 3
[    2.191845] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.191855] ehci-pci 0000:00:16.2: debug port 1
[    2.192018] ehci-pci 0000:00:16.2: irq 17, io mem 0xf01cf000
[    2.205373] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00
[    2.205651] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.205655] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.205658] usb usb3: Product: EHCI Host Controller
[    2.205660] usb usb3: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ehci_hcd
[    2.205662] usb usb3: SerialNumber: 0000:00:16.2
[    2.206200] hub 3-0:1.0: USB hub found
[    2.206236] hub 3-0:1.0: 4 ports detected
[    2.221053] ohci-pci 0000:00:12.0: OHCI PCI host controller
[    2.221085] ohci-pci 0000:00:12.0: new USB bus registered, assigned bus number 4
[    2.221264] ohci-pci 0000:00:12.0: irq 18, io mem 0xf01c8000
[    2.237796] r8169 0000:04:00.0: enabling device (0000 -> 0003)
[    2.261308] xhci_hcd 0000:03:00.0: xHCI Host Controller
[    2.261394] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 5
[    2.272125] r8169 0000:04:00.0 eth0: RTL8168f/8111f, 08:60:6e:74:7a:51, XID 480, IRQ 28
[    2.272143] r8169 0000:04:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    2.282633] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.282644] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.282650] usb usb4: Product: OHCI PCI host controller
[    2.282655] usb usb4: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ohci_hcd
[    2.282659] usb usb4: SerialNumber: 0000:00:12.0
[    2.283372] hub 4-0:1.0: USB hub found
[    2.283431] hub 4-0:1.0: 5 ports detected
[    2.297576] ohci-pci 0000:00:13.0: OHCI PCI host controller
[    2.297609] ohci-pci 0000:00:13.0: new USB bus registered, assigned bus number 6
[    2.297727] ohci-pci 0000:00:13.0: irq 18, io mem 0xf01c9000
[    2.320955] xhci_hcd 0000:03:00.0: hcc params 0x0200f180 hci version 0x96 quirks 0x0000000000080010
[    2.322099] xhci_hcd 0000:03:00.0: xHCI Host Controller
[    2.322121] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 7
[    2.322136] xhci_hcd 0000:03:00.0: Host supports USB 3.0 SuperSpeed
[    2.324029] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    2.324036] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.324039] usb usb5: Product: xHCI Host Controller
[    2.324041] usb usb5: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 xhci-hcd
[    2.324043] usb usb5: SerialNumber: 0000:03:00.0
[    2.324687] hub 5-0:1.0: USB hub found
[    2.324727] hub 5-0:1.0: 2 ports detected
[    2.325467] usb usb7: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.325624] usb usb7: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.03
[    2.325628] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.325630] usb usb7: Product: xHCI Host Controller
[    2.325632] usb usb7: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 xhci-hcd
[    2.325634] usb usb7: SerialNumber: 0000:03:00.0
[    2.326195] hub 7-0:1.0: USB hub found
[    2.327034] hub 7-0:1.0: 2 ports detected
[    2.349205] snd_hda_intel 0000:00:01.1: Force to non-snoop mode
[    2.366399] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.366406] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.366409] usb usb6: Product: OHCI PCI host controller
[    2.366411] usb usb6: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ohci_hcd
[    2.366413] usb usb6: SerialNumber: 0000:00:13.0
[    2.367210] hub 6-0:1.0: USB hub found
[    2.367303] hub 6-0:1.0: 5 ports detected
[    2.391447] r8169 0000:04:00.0 enp4s0: renamed from eth0
[    2.420037] ohci-pci 0000:00:14.5: OHCI PCI host controller
[    2.420070] ohci-pci 0000:00:14.5: new USB bus registered, assigned bus number 8
[    2.421556] ohci-pci 0000:00:14.5: irq 18, io mem 0xf01ca000
[    2.445281] r8169 0000:04:00.0: Direct firmware load for rtl_nic/rtl8168f-1.fw failed with error -2
[    2.445292] r8169 0000:04:00.0: Unable to load firmware rtl_nic/rtl8168f-1.fw (-2)
[    2.445868] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
[    2.481250] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/sound/card0/input1
[    2.481880] input: HDA ATI HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.1/sound/card0/input2
[    2.488733] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.488740] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.488743] usb usb8: Product: OHCI PCI host controller
[    2.488745] usb usb8: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ohci_hcd
[    2.488747] usb usb8: SerialNumber: 0000:00:14.5
[    2.492816] snd_hda_codec_realtek hdaudioC1D0: ALC892: SKU not ready 0x00000100
[    2.492902] hub 8-0:1.0: USB hub found
[    2.492953] hub 8-0:1.0: 2 ports detected
[    2.493612] snd_hda_codec_realtek hdaudioC1D0: autoconfig for ALC892: line_outs=4 (0x14/0x16/0x15/0x17/0x0) type:line
[    2.493620] snd_hda_codec_realtek hdaudioC1D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    2.493623] snd_hda_codec_realtek hdaudioC1D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    2.493626] snd_hda_codec_realtek hdaudioC1D0:    mono: mono_out=0x0
[    2.493628] snd_hda_codec_realtek hdaudioC1D0:    dig-out=0x1e/0x0
[    2.493629] snd_hda_codec_realtek hdaudioC1D0:    inputs:
[    2.493631] snd_hda_codec_realtek hdaudioC1D0:      Rear Mic=0x18
[    2.493633] snd_hda_codec_realtek hdaudioC1D0:      Front Mic=0x19
[    2.493635] snd_hda_codec_realtek hdaudioC1D0:      Line=0x1a
[    2.493636] snd_hda_codec_realtek hdaudioC1D0:      CD=0x1c
[    2.495003] ohci-pci 0000:00:16.0: OHCI PCI host controller
[    2.495036] ohci-pci 0000:00:16.0: new USB bus registered, assigned bus number 9
[    2.495172] ohci-pci 0000:00:16.0: irq 18, io mem 0xf01cb000
[    2.510556] [drm] radeon kernel modesetting enabled.
[    2.513510] [drm] initializing kernel modesetting (ARUBA 0x1002:0x9996 0x1002:0x9996 0x00).
[    2.513588] ATOM BIOS: 113
[    2.513893] radeon 0000:00:01.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
[    2.513898] radeon 0000:00:01.0: GTT: 1024M 0x0000000020000000 - 0x000000005FFFFFFF
[    2.513911] [drm] Detected VRAM RAM=512M, BAR=256M
[    2.513913] [drm] RAM width 64bits DDR
[    2.514805] [drm] radeon: 512M of VRAM memory ready
[    2.514815] [drm] radeon: 1024M of GTT memory ready.
[    2.514888] [drm] Loading ARUBA Microcode
[    2.517185] r8169 0000:04:00.0 enp4s0: Link is Down
[    2.519927] input: HD-Audio Generic Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input3
[    2.520307] input: HD-Audio Generic Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input4
[    2.520824] input: HD-Audio Generic Line as /devices/pci0000:00/0000:00:14.2/sound/card1/input5
[    2.521180] input: HD-Audio Generic Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card1/input6
[    2.521594] input: HD-Audio Generic Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card1/input7
[    2.522160] input: HD-Audio Generic Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card1/input8
[    2.522536] input: HD-Audio Generic Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card1/input9
[    2.522888] input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card1/input10
[    2.532990] [drm] Internal thermal controller without fan control
[    2.558842] usb usb9: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.03
[    2.558849] usb usb9: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.558852] usb usb9: Product: OHCI PCI host controller
[    2.558855] usb usb9: Manufacturer: Linux 6.3.0-rc3-00046-g8ba643d7e1c7 ohci_hcd
[    2.558857] usb usb9: SerialNumber: 0000:00:16.0
[    2.558984] [drm] radeon: dpm initialized
[    2.564667] [drm] Found VCE firmware/feedback version 50.0.1 / 17!
[    2.564728] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    2.594051] [drm] PCIE GART of 1024M enabled (table at 0x00000000001D6000).
[    2.594314] radeon 0000:00:01.0: WB enabled
[    2.594318] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00
[    2.594697] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 0x0000000000075a18
[    2.595711] hub 9-0:1.0: USB hub found
[    2.595753] hub 9-0:1.0: 4 ports detected
[    2.618866] radeon 0000:00:01.0: fence driver on ring 6 use gpu addr 0x0000000020000c18
[    2.618874] radeon 0000:00:01.0: fence driver on ring 7 use gpu addr 0x0000000020000c1c
[    2.618877] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 0x0000000020000c04
[    2.618879] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 0x0000000020000c08
[    2.618881] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 0x0000000020000c0c
[    2.618883] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 0x0000000020000c10
[    2.647823] radeon 0000:00:01.0: radeon: MSI limited to 32-bit
[    2.648048] radeon 0000:00:01.0: radeon: using MSI.
[    2.648125] [drm] radeon: irq initialized.
[    2.667958] [drm] ring test on 0 succeeded in 3 usecs
[    2.667970] [drm] ring test on 3 succeeded in 4 usecs
[    2.667977] [drm] ring test on 4 succeeded in 4 usecs
[    2.713784] [drm] ring test on 5 succeeded in 2 usecs
[    2.733666] [drm] UVD initialized successfully.
[    2.741341] usb 4-1: new low-speed USB device number 2 using ohci-pci
[    2.843040] [drm] ring test on 6 succeeded in 18 usecs
[    2.843052] [drm] ring test on 7 succeeded in 3 usecs
[    2.843053] [drm] VCE initialized successfully.
[    2.843202] snd_hda_intel 0000:00:01.1: bound 0000:00:01.0 (ops radeon_audio_component_bind_ops [radeon])
[    2.843370] [drm] ib test on ring 0 succeeded in 0 usecs
[    2.843424] [drm] ib test on ring 3 succeeded in 0 usecs
[    2.843474] [drm] ib test on ring 4 succeeded in 0 usecs
[    2.942595] usb 4-1: New USB device found, idVendor=413c, idProduct=2106, bcdDevice= 1.01
[    2.942607] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.942611] usb 4-1: Product: Dell QuietKey Keyboard
[    2.942615] usb 4-1: Manufacturer: DELL
[    2.951006] input: DELL Dell QuietKey Keyboard as /devices/pci0000:00/0000:00:12.0/usb4/4-1/4-1:1.0/0003:413C:2106.0001/input/input11
[    3.010775] hid-generic 0003:413C:2106.0001: input,hidraw0: USB HID v1.10 Keyboard [DELL Dell QuietKey Keyboard] on usb-0000:00:12.0-1/input0
[    3.385454] [drm] ib test on ring 5 succeeded
[    3.405349] usb 4-2: new low-speed USB device number 3 using ohci-pci
[    3.600597] usb 4-2: New USB device found, idVendor=046d, idProduct=c016, bcdDevice= 3.40
[    3.600609] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.600613] usb 4-2: Product: Optical USB Mouse
[    3.600617] usb 4-2: Manufacturer: Logitech
[    3.610963] input: Logitech Optical USB Mouse as /devices/pci0000:00/0000:00:12.0/usb4/4-2/4-2:1.0/0003:046D:C016.0002/input/input12
[    3.611917] hid-generic 0003:046D:C016.0002: input,hidraw1: USB HID v1.10 Mouse [Logitech Optical USB Mouse] on usb-0000:00:12.0-2/input0
[    3.929454] [drm] ib test on ring 6 succeeded
[    4.441455] [drm] ib test on ring 7 succeeded
[    4.446983] [drm] Radeon Display Connectors
[    4.446989] [drm] Connector 0:
[    4.446992] [drm]   DP-1
[    4.446995] [drm]   HPD1
[    4.446997] [drm]   DDC: 0x6530 0x6530 0x6534 0x6534 0x6538 0x6538 0x653c 0x653c
[    4.447003] [drm]   Encoders:
[    4.447005] [drm]     DFP1: INTERNAL_UNIPHY2
[    4.447007] [drm] Connector 1:
[    4.447009] [drm]   VGA-1
[    4.447011] [drm]   HPD2
[    4.447013] [drm]   DDC: 0x6540 0x6540 0x6544 0x6544 0x6548 0x6548 0x654c 0x654c
[    4.447018] [drm]   Encoders:
[    4.447020] [drm]     CRT1: INTERNAL_UNIPHY2
[    4.447022] [drm]     CRT1: NUTMEG
[    4.447024] [drm] Connector 2:
[    4.447026] [drm]   HDMI-A-1
[    4.447028] [drm]   HPD3
[    4.447030] [drm]   DDC: 0x6550 0x6550 0x6554 0x6554 0x6558 0x6558 0x655c 0x655c
[    4.447035] [drm]   Encoders:
[    4.447036] [drm]     DFP2: INTERNAL_UNIPHY
[    4.716171] [drm] fb mappable at 0xE03E9000
[    4.716179] [drm] vram apper at 0xE0000000
[    4.716181] [drm] size 5242880
[    4.716183] [drm] fb depth is 24
[    4.716185] [drm]    pitch is 5120
[    4.716671] fbcon: radeondrmfb (fb0) is primary device
[    4.908197] Console: switching to colour frame buffer device 160x64
[    4.909974] radeon 0000:00:01.0: [drm] fb0: radeondrmfb frame buffer device
[    4.933682] [drm] Initialized radeon 2.50.0 20080528 for 0000:00:01.0 on minor 0
[    5.125043] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
[    5.125059] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0: link becomes ready
[    7.669093] [drm] amdgpu kernel modesetting enabled.
[   11.216542] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=272 'systemd'
Thomas Gleixner April 25, 2023, 8:07 p.m. UTC | #26
On Thu, Apr 20 2023 at 17:57, Thomas Gleixner wrote:
> On Thu, Apr 20 2023 at 07:51, Sean Christopherson wrote:
> Something like the completely untested below should just work whatever
> APIC ID the BIOS decided to dice.
>
> That might just work on SEV too without that GHCB muck, but what do I
> know.

It does not.

RDMSR(X2APIC_ID) is trapped via #VC which cannot be handled at that
point. Unfortunately the GHCB protocol does not provide a RDMSR
mechanism similar to the CPUID mechanism. Neither does the secure
firmware enforce CPUID(0xb):APICID to real APIC ID consistency.

So the hypervisor can dice the APIC IDs as long as they are consistent
with the provided ACPI/MADT table.

So no parallel startup for SEV for now.

Thanks,

        tglx
Michael Kelley (LINUX) April 27, 2023, 2:48 p.m. UTC | #27
From: Thomas Gleixner <tglx@linutronix.de> Sent: Friday, April 14, 2023 4:44 PM

[snip]

> 
> Conclusion
> ----------
> 
> Adding the basic parallel bringup mechanism as provided by this series
> makes a lot of sense. Improving particular issues as pointed out in the
> analysis makes sense too.
> 
> But trying to solve an application specific problem fully in the kernel
> with tons of complexity, without exploring straight forward and simple
> approaches first, does not make any sense at all.
> 
> Thanks,
> 
> 	tglx
> 
> ---
>  Documentation/admin-guide/kernel-parameters.txt |   20
>  Documentation/core-api/cpu_hotplug.rst          |   13
>  arch/Kconfig                                    |   23 +
>  arch/arm/Kconfig                                |    1
>  arch/arm/include/asm/smp.h                      |    2
>  arch/arm/kernel/smp.c                           |   18
>  arch/arm64/Kconfig                              |    1
>  arch/arm64/include/asm/smp.h                    |    2
>  arch/arm64/kernel/smp.c                         |   14
>  arch/csky/Kconfig                               |    1
>  arch/csky/include/asm/smp.h                     |    2
>  arch/csky/kernel/smp.c                          |    8
>  arch/mips/Kconfig                               |    1
>  arch/mips/cavium-octeon/smp.c                   |    1
>  arch/mips/include/asm/smp-ops.h                 |    1
>  arch/mips/kernel/smp-bmips.c                    |    1
>  arch/mips/kernel/smp-cps.c                      |   14
>  arch/mips/kernel/smp.c                          |    8
>  arch/mips/loongson64/smp.c                      |    1
>  arch/parisc/Kconfig                             |    1
>  arch/parisc/kernel/process.c                    |    4
>  arch/parisc/kernel/smp.c                        |    7
>  arch/riscv/Kconfig                              |    1
>  arch/riscv/include/asm/smp.h                    |    2
>  arch/riscv/kernel/cpu-hotplug.c                 |   14
>  arch/x86/Kconfig                                |   45 --
>  arch/x86/include/asm/apic.h                     |    5
>  arch/x86/include/asm/cpu.h                      |    5
>  arch/x86/include/asm/cpumask.h                  |    5
>  arch/x86/include/asm/processor.h                |    1
>  arch/x86/include/asm/realmode.h                 |    3
>  arch/x86/include/asm/sev-common.h               |    3
>  arch/x86/include/asm/smp.h                      |   26 -
>  arch/x86/include/asm/topology.h                 |   23 -
>  arch/x86/include/asm/tsc.h                      |    2
>  arch/x86/kernel/acpi/sleep.c                    |    9
>  arch/x86/kernel/apic/apic.c                     |   22 -
>  arch/x86/kernel/callthunks.c                    |    4
>  arch/x86/kernel/cpu/amd.c                       |    2
>  arch/x86/kernel/cpu/cacheinfo.c                 |   21
>  arch/x86/kernel/cpu/common.c                    |   50 --
>  arch/x86/kernel/cpu/topology.c                  |    3
>  arch/x86/kernel/head_32.S                       |   14
>  arch/x86/kernel/head_64.S                       |  121 +++++
>  arch/x86/kernel/sev.c                           |    2
>  arch/x86/kernel/smp.c                           |    3
>  arch/x86/kernel/smpboot.c                       |  508 ++++++++----------------
>  arch/x86/kernel/topology.c                      |   98 ----
>  arch/x86/kernel/tsc.c                           |   20
>  arch/x86/kernel/tsc_sync.c                      |   36 -
>  arch/x86/power/cpu.c                            |   37 -
>  arch/x86/realmode/init.c                        |    3
>  arch/x86/realmode/rm/trampoline_64.S            |   27 +
>  arch/x86/xen/enlighten_hvm.c                    |   11
>  arch/x86/xen/smp_hvm.c                          |   16
>  arch/x86/xen/smp_pv.c                           |   56 +-
>  drivers/acpi/processor_idle.c                   |    4
>  include/linux/cpu.h                             |    4
>  include/linux/cpuhotplug.h                      |   17
>  kernel/cpu.c                                    |  397 +++++++++++++++++-
>  kernel/smp.c                                    |    2
>  kernel/smpboot.c                                |  163 -------
>  62 files changed, 953 insertions(+), 976 deletions(-)
> 

I smoke-tested several Linux guest configurations running on Hyper-V,
using the "kernel/git/tglx/devel.git hotplug" tree as updated on April 26th.
No functional issues, but encountered one cosmetic issue (details below).

Configurations tested:
*  16 vCPUs and 32 vCPUs
*  1 NUMA node and 2 NUMA nodes
*  Parallel bring-up enabled and disabled via kernel boot line
*  "Normal" VMs and SEV-SNP VMs running with a paravisor on Hyper-V.
    This config can use parallel bring-up because most of the SNP-ness is
    hidden in the paravisor.  I was glad to see this work properly.

There's not much difference in performance with and without parallel
bring-up on the 32 vCPU VM.   Without parallel, the time is about 26
milliseconds.  With parallel, it's about 24 ms.   So bring-up is already
fast in the virtual environment.

The cosmetic issue is in the dmesg log, and arises because Hyper-V
enumerates SMT CPUs differently from many other environments.  In
a Hyper-V guest, the SMT threads in a core are numbered as <even, odd>
pairs.  Guest CPUs #0 & #1 are SMT threads in core, as are #2 & #3, etc.  With
parallel bring-up, here's the dmesg output:

[    0.444345] smp: Bringing up secondary CPUs ...
[    0.445139] .... node  #0, CPUs:    #2  #4  #6  #8 #10 #12 #14 #16 #18 #20 #22 #24 #26 #28 #30
[    0.454112] x86: Booting SMP configuration:
[    0.456035]       #1  #3  #5  #7  #9 #11 #13 #15 #17 #19 #21 #23 #25 #27 #29 #31
[    0.466120] smp: Brought up 1 node, 32 CPUs
[    0.467036] smpboot: Max logical packages: 1
[    0.468035] smpboot: Total of 32 processors activated (153240.06 BogoMIPS)

The function announce_cpu() is specifically testing for CPU #1 to output the
"Booting SMP configuration" message.  In a Hyper-V guest, CPU #1 is the second
SMT thread in a core, so it isn't started until all the even-numbered CPUs are
started.

I don't know if this cosmetic issue is worth fixing, but I thought I'd point it out.

In any case,

Tested-by: Michael Kelley <mikelley@microsoft.com>
Thomas Gleixner May 4, 2023, 6:46 p.m. UTC | #28
Michael!

On Thu, Apr 27 2023 at 14:48, Michael Kelley wrote:
> From: Thomas Gleixner <tglx@linutronix.de> Sent: Friday, April 14, 2023 4:44 PM
>
> I smoke-tested several Linux guest configurations running on Hyper-V,
> using the "kernel/git/tglx/devel.git hotplug" tree as updated on April 26th.
> No functional issues, but encountered one cosmetic issue (details below).
>
> Configurations tested:
> *  16 vCPUs and 32 vCPUs
> *  1 NUMA node and 2 NUMA nodes
> *  Parallel bring-up enabled and disabled via kernel boot line
> *  "Normal" VMs and SEV-SNP VMs running with a paravisor on Hyper-V.
>     This config can use parallel bring-up because most of the SNP-ness is
>     hidden in the paravisor.  I was glad to see this work properly.
>
> There's not much difference in performance with and without parallel
> bring-up on the 32 vCPU VM.   Without parallel, the time is about 26
> milliseconds.  With parallel, it's about 24 ms.   So bring-up is already
> fast in the virtual environment.

Depends on the environment :)

> The cosmetic issue is in the dmesg log, and arises because Hyper-V
> enumerates SMT CPUs differently from many other environments.  In
> a Hyper-V guest, the SMT threads in a core are numbered as <even, odd>
> pairs.  Guest CPUs #0 & #1 are SMT threads in core, as are #2 & #3, etc.  With
> parallel bring-up, here's the dmesg output:
>
> [    0.444345] smp: Bringing up secondary CPUs ...
> [    0.445139] .... node  #0, CPUs:    #2  #4  #6  #8 #10 #12 #14 #16 #18 #20 #22 #24 #26 #28 #30
> [    0.454112] x86: Booting SMP configuration:
> [    0.456035]       #1  #3  #5  #7  #9 #11 #13 #15 #17 #19 #21 #23 #25 #27 #29 #31
> [    0.466120] smp: Brought up 1 node, 32 CPUs
> [    0.467036] smpboot: Max logical packages: 1
> [    0.468035] smpboot: Total of 32 processors activated (153240.06 BogoMIPS)
>
> The function announce_cpu() is specifically testing for CPU #1 to output the
> "Booting SMP configuration" message.  In a Hyper-V guest, CPU #1 is the second
> SMT thread in a core, so it isn't started until all the even-numbered CPUs are
> started.

Ah. Didn't notice that because SMT siblings are usually enumerated after
all primary ones in ACPI.

> I don't know if this cosmetic issue is worth fixing, but I thought I'd point it out.

That's trivial enough to fix. I'll amend the topmost patch before
posting V2.

Thanks for giving it a ride!

       tglx