mbox series

[0/3] x86/pvh: Support relocating dom0 kernel

Message ID 20240306185032.103216-1-jason.andryuk@amd.com (mailing list archive)
Headers show
Series x86/pvh: Support relocating dom0 kernel | expand

Message

Jason Andryuk March 6, 2024, 6:50 p.m. UTC
Xen tries to load a PVH dom0 kernel at the fixed guest physical address
from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
it can be configured.

Unfortunately there exist firmwares that have reserved regions at this
address, so Xen fails to load the dom0 kernel since it's not RAM.

The other issue is that the Linux PVH entry point is not
position-independent.  It expects to run at the compiled
CONFIG_PHYSICAL_ADDRESS.

This patch set expands the PVH dom0 builder to try to relocate the
kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
kernels to indicate they are relocatable.  However, we may want to
switch to an additional ELF note with the kernel alignment.  Linux
specifies a kernel alignment in the bzImage boot_params.setup_header,
but that is not present the ELF vmlinux file.

The first patch fixes some whitespace in features.h

The second patch enhances bzimage_parse to pull the kernel_alignment into
an optional align pointer.

The third patch expands the pvh dom0 kernel placement code.

I'll post an additional patch showing the Linux changes to make PVH
relocatable.  

Jason Andryuk (3):
  features.h: Replace hard tabs
  xen/x86: bzImage parse kernel_alignment
  x86/PVH: Support relocatable dom0 kernels

 xen/arch/x86/bzimage.c             |   4 +-
 xen/arch/x86/hvm/dom0_build.c      | 113 ++++++++++++++++++++++++++++-
 xen/arch/x86/include/asm/bzimage.h |   3 +-
 xen/arch/x86/pv/dom0_build.c       |   2 +-
 xen/include/public/features.h      |   9 ++-
 5 files changed, 124 insertions(+), 7 deletions(-)

Comments

Roger Pau Monné March 7, 2024, 10 a.m. UTC | #1
On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote:
> Xen tries to load a PVH dom0 kernel at the fixed guest physical address
> from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
> it can be configured.
> 
> Unfortunately there exist firmwares that have reserved regions at this
> address, so Xen fails to load the dom0 kernel since it's not RAM.
> 
> The other issue is that the Linux PVH entry point is not
> position-independent.  It expects to run at the compiled
> CONFIG_PHYSICAL_ADDRESS.
> 
> This patch set expands the PVH dom0 builder to try to relocate the
> kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
> kernels to indicate they are relocatable.  However, we may want to
> switch to an additional ELF note with the kernel alignment.  Linux
> specifies a kernel alignment in the bzImage boot_params.setup_header,
> but that is not present the ELF vmlinux file.

I wonder whether we need a pair of notes, to signal the min/max
addresses the kernel supports being relocated to.

Thanks, Roger.
Jan Beulich March 7, 2024, 10:08 a.m. UTC | #2
On 07.03.2024 11:00, Roger Pau Monné wrote:
> On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote:
>> Xen tries to load a PVH dom0 kernel at the fixed guest physical address
>> from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
>> it can be configured.
>>
>> Unfortunately there exist firmwares that have reserved regions at this
>> address, so Xen fails to load the dom0 kernel since it's not RAM.
>>
>> The other issue is that the Linux PVH entry point is not
>> position-independent.  It expects to run at the compiled
>> CONFIG_PHYSICAL_ADDRESS.
>>
>> This patch set expands the PVH dom0 builder to try to relocate the
>> kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
>> kernels to indicate they are relocatable.  However, we may want to
>> switch to an additional ELF note with the kernel alignment.  Linux
>> specifies a kernel alignment in the bzImage boot_params.setup_header,
>> but that is not present the ELF vmlinux file.
> 
> I wonder whether we need a pair of notes, to signal the min/max
> addresses the kernel supports being relocated to.

Plus, as per your other reply, a 3rd one to specify alignment needs.

Then again - a single note can have multiple values. So perhaps it
doesn't need to be more than one new notes (except if dealing with
multi-value ones is deemed too complicated).

Jan
Roger Pau Monné March 7, 2024, 10:20 a.m. UTC | #3
On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote:
> On 07.03.2024 11:00, Roger Pau Monné wrote:
> > On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote:
> >> Xen tries to load a PVH dom0 kernel at the fixed guest physical address
> >> from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
> >> it can be configured.
> >>
> >> Unfortunately there exist firmwares that have reserved regions at this
> >> address, so Xen fails to load the dom0 kernel since it's not RAM.
> >>
> >> The other issue is that the Linux PVH entry point is not
> >> position-independent.  It expects to run at the compiled
> >> CONFIG_PHYSICAL_ADDRESS.
> >>
> >> This patch set expands the PVH dom0 builder to try to relocate the
> >> kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
> >> kernels to indicate they are relocatable.  However, we may want to
> >> switch to an additional ELF note with the kernel alignment.  Linux
> >> specifies a kernel alignment in the bzImage boot_params.setup_header,
> >> but that is not present the ELF vmlinux file.
> > 
> > I wonder whether we need a pair of notes, to signal the min/max
> > addresses the kernel supports being relocated to.
> 
> Plus, as per your other reply, a 3rd one to specify alignment needs.

Alignment we could in theory get from the ELF program header, if OSes
fill those reliably.  FreeBSD seems to do so, haven't checked Linux.

> Then again - a single note can have multiple values. So perhaps it
> doesn't need to be more than one new notes (except if dealing with
> multi-value ones is deemed too complicated).

I've never dealt with a multi-value note, if that's not overly
complicated I would be fine with it, otherwise multiple notes are OK
IMO.

Thanks, Roger.
Jason Andryuk March 7, 2024, 5:33 p.m. UTC | #4
On 2024-03-07 05:20, Roger Pau Monné wrote:
> On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote:
>> On 07.03.2024 11:00, Roger Pau Monné wrote:
>>> On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote:
>>>> Xen tries to load a PVH dom0 kernel at the fixed guest physical address
>>>> from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
>>>> it can be configured.
>>>>
>>>> Unfortunately there exist firmwares that have reserved regions at this
>>>> address, so Xen fails to load the dom0 kernel since it's not RAM.
>>>>
>>>> The other issue is that the Linux PVH entry point is not
>>>> position-independent.  It expects to run at the compiled
>>>> CONFIG_PHYSICAL_ADDRESS.
>>>>
>>>> This patch set expands the PVH dom0 builder to try to relocate the
>>>> kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
>>>> kernels to indicate they are relocatable.  However, we may want to
>>>> switch to an additional ELF note with the kernel alignment.  Linux
>>>> specifies a kernel alignment in the bzImage boot_params.setup_header,
>>>> but that is not present the ELF vmlinux file.
>>>
>>> I wonder whether we need a pair of notes, to signal the min/max
>>> addresses the kernel supports being relocated to.
>>
>> Plus, as per your other reply, a 3rd one to specify alignment needs.
> 
> Alignment we could in theory get from the ELF program header, if OSes
> fill those reliably.  FreeBSD seems to do so, haven't checked Linux.

I will look into this more, but at first glance, I don't see a 
connection between Linux's CONFIG_PHYSICAL_ALIGN and the values in the 
elf headers.  As a quick test, I set CONFIG_PHYSICAL_ALIGN=0x800000, but 
the elf align values are still 0x200000.

The elf header values may be a suitable fallback though?

>> Then again - a single note can have multiple values. So perhaps it
>> doesn't need to be more than one new notes (except if dealing with
>> multi-value ones is deemed too complicated).
> 
> I've never dealt with a multi-value note, if that's not overly
> complicated I would be fine with it, otherwise multiple notes are OK
> IMO.

It looks like a single note can be followed by arbitrary data, so 
putting three values in should be fine.  PVH is defined with a 32bit 
entry point, so are 32bit min, max & align values acceptable, or should 
64bit be specified?  Linux uses _ASM_PTR for PHYS32_ENTRY so its size 
matches the bitness of the target (32 on 32/ 64 on 64).  I guess I'll 
follow that unless I hear a preference otherwise.

Regards,
Jason
Jan Beulich March 8, 2024, 7:03 a.m. UTC | #5
On 07.03.2024 18:33, Jason Andryuk wrote:
> On 2024-03-07 05:20, Roger Pau Monné wrote:
>> On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote:
>>> On 07.03.2024 11:00, Roger Pau Monné wrote:
>>>> On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote:
>>>>> Xen tries to load a PVH dom0 kernel at the fixed guest physical address
>>>>> from the elf headers.  For Linux, this defaults to 0x1000000 (16MB), but
>>>>> it can be configured.
>>>>>
>>>>> Unfortunately there exist firmwares that have reserved regions at this
>>>>> address, so Xen fails to load the dom0 kernel since it's not RAM.
>>>>>
>>>>> The other issue is that the Linux PVH entry point is not
>>>>> position-independent.  It expects to run at the compiled
>>>>> CONFIG_PHYSICAL_ADDRESS.
>>>>>
>>>>> This patch set expands the PVH dom0 builder to try to relocate the
>>>>> kernel if needed and possible.  XENFEAT_pvh_relocatable is added for
>>>>> kernels to indicate they are relocatable.  However, we may want to
>>>>> switch to an additional ELF note with the kernel alignment.  Linux
>>>>> specifies a kernel alignment in the bzImage boot_params.setup_header,
>>>>> but that is not present the ELF vmlinux file.
>>>>
>>>> I wonder whether we need a pair of notes, to signal the min/max
>>>> addresses the kernel supports being relocated to.
>>>
>>> Plus, as per your other reply, a 3rd one to specify alignment needs.
>>
>> Alignment we could in theory get from the ELF program header, if OSes
>> fill those reliably.  FreeBSD seems to do so, haven't checked Linux.
> 
> I will look into this more, but at first glance, I don't see a 
> connection between Linux's CONFIG_PHYSICAL_ALIGN and the values in the 
> elf headers.  As a quick test, I set CONFIG_PHYSICAL_ALIGN=0x800000, but 
> the elf align values are still 0x200000.
> 
> The elf header values may be a suitable fallback though?

Imo, given the above, explicit values should be required. Better not
load a kernel than doing so and then getting hard to debug crashes.

Jan