| Message ID | 1385251666-27532-3-git-send-email-santosh.shilimkar@ti.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: > Boot-CPU entry into the HYP mode is managed in boot-loader but > the secondary CPUs directly jumps to kernel during boot. Same > path is also used for CPU hotplug as well during suspend for > secondary CPU. > > Hence patch the secondary CPU boot path for hyp mode etry. > > Cc: Marc Zyngier <marc.zyngier@arm.com> > Cc: Christoffer Dall <christoffer.dall@linaro.org> > Cc: Tony Lindgren <tony@atomide.com> > Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> > --- > arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S > index 75e9295..4844dd8 100644 > --- a/arch/arm/mach-omap2/omap-headsmp.S > +++ b/arch/arm/mach-omap2/omap-headsmp.S > @@ -22,6 +22,7 @@ > > /* Physical address needed since MMU not enabled yet on secondary core */ > #define AUX_CORE_BOOT0_PA 0x48281800 > +#define API_HYP_ENTRY 0x102 > > /* > * OMAP5 specific entry point for secondary CPU to jump from ROM > @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 > and r4, r4, #0x0f > cmp r0, r4 > bne wait > +#ifdef CONFIG_KVM_ARM_HOST > + ldr r12, =API_HYP_ENTRY > + adr r0, hyp_boot > + smc #0 > +hyp_boot: > +#endif > b secondary_startup > END(omap5_secondary_startup) > /* hmm, this means that currently running this in a guest will fail to bring-up SMP, right? Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel? -Christoffer
On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: > On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >> Boot-CPU entry into the HYP mode is managed in boot-loader but >> the secondary CPUs directly jumps to kernel during boot. Same >> path is also used for CPU hotplug as well during suspend for >> secondary CPU. >> >> Hence patch the secondary CPU boot path for hyp mode etry. >> >> Cc: Marc Zyngier <marc.zyngier@arm.com> >> Cc: Christoffer Dall <christoffer.dall@linaro.org> >> Cc: Tony Lindgren <tony@atomide.com> >> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >> --- >> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >> index 75e9295..4844dd8 100644 >> --- a/arch/arm/mach-omap2/omap-headsmp.S >> +++ b/arch/arm/mach-omap2/omap-headsmp.S >> @@ -22,6 +22,7 @@ >> >> /* Physical address needed since MMU not enabled yet on secondary core */ >> #define AUX_CORE_BOOT0_PA 0x48281800 >> +#define API_HYP_ENTRY 0x102 >> >> /* >> * OMAP5 specific entry point for secondary CPU to jump from ROM >> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >> and r4, r4, #0x0f >> cmp r0, r4 >> bne wait >> +#ifdef CONFIG_KVM_ARM_HOST >> + ldr r12, =API_HYP_ENTRY >> + adr r0, hyp_boot >> + smc #0 >> +hyp_boot: >> +#endif >> b secondary_startup >> END(omap5_secondary_startup) >> /* > > hmm, this means that currently running this in a guest will fail to > bring-up SMP, right? > Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ? > Couldn't you create a little wrapper-pen in U-Boot instead, which > replicates the omap boot protocol and takes care of the hyp-mode > startup there instead, keeping this completely out of the kernel? > Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode. In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must. Regards, Santosh Regards, Santosh
On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: > On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: >> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>> Boot-CPU entry into the HYP mode is managed in boot-loader but >>> the secondary CPUs directly jumps to kernel during boot. Same >>> path is also used for CPU hotplug as well during suspend for >>> secondary CPU. >>> >>> Hence patch the secondary CPU boot path for hyp mode etry. >>> >>> Cc: Marc Zyngier <marc.zyngier@arm.com> >>> Cc: Christoffer Dall <christoffer.dall@linaro.org> >>> Cc: Tony Lindgren <tony@atomide.com> >>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >>> --- >>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >>> index 75e9295..4844dd8 100644 >>> --- a/arch/arm/mach-omap2/omap-headsmp.S >>> +++ b/arch/arm/mach-omap2/omap-headsmp.S >>> @@ -22,6 +22,7 @@ >>> >>> /* Physical address needed since MMU not enabled yet on secondary core */ >>> #define AUX_CORE_BOOT0_PA 0x48281800 >>> +#define API_HYP_ENTRY 0x102 >>> >>> /* >>> * OMAP5 specific entry point for secondary CPU to jump from ROM >>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >>> and r4, r4, #0x0f >>> cmp r0, r4 >>> bne wait >>> +#ifdef CONFIG_KVM_ARM_HOST >>> + ldr r12, =API_HYP_ENTRY >>> + adr r0, hyp_boot >>> + smc #0 >>> +hyp_boot: >>> +#endif >>> b secondary_startup >>> END(omap5_secondary_startup) >>> /* >> >> hmm, this means that currently running this in a guest will fail to >> bring-up SMP, right? >> > Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build > will not enable CONFIG_KVM_ARM_HOST and things should be fine then. > Right ? > That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution. >> Couldn't you create a little wrapper-pen in U-Boot instead, which >> replicates the omap boot protocol and takes care of the hyp-mode >> startup there instead, keeping this completely out of the kernel? >> > Its not just booting but CPU hotplug also follows the same path > so we need the mechanism in kernel to switch mode. > > In general, I think its important to consider the aspect with > CPU PM. CPUs are not going to go through the boot-loaders in > those paths and hence need of HYP entry in the kernel will > be must. > I agree, and PSCI is the obvious only correct answer to this. We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas. However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction. -Christoffer
On 25/11/13 16:28, Santosh Shilimkar wrote: > On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: >> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>> Boot-CPU entry into the HYP mode is managed in boot-loader but >>> the secondary CPUs directly jumps to kernel during boot. Same >>> path is also used for CPU hotplug as well during suspend for >>> secondary CPU. >>> >>> Hence patch the secondary CPU boot path for hyp mode etry. >>> >>> Cc: Marc Zyngier <marc.zyngier@arm.com> >>> Cc: Christoffer Dall <christoffer.dall@linaro.org> >>> Cc: Tony Lindgren <tony@atomide.com> >>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >>> --- >>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >>> index 75e9295..4844dd8 100644 >>> --- a/arch/arm/mach-omap2/omap-headsmp.S >>> +++ b/arch/arm/mach-omap2/omap-headsmp.S >>> @@ -22,6 +22,7 @@ >>> >>> /* Physical address needed since MMU not enabled yet on secondary core */ >>> #define AUX_CORE_BOOT0_PA 0x48281800 >>> +#define API_HYP_ENTRY 0x102 >>> >>> /* >>> * OMAP5 specific entry point for secondary CPU to jump from ROM >>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >>> and r4, r4, #0x0f >>> cmp r0, r4 >>> bne wait >>> +#ifdef CONFIG_KVM_ARM_HOST >>> + ldr r12, =API_HYP_ENTRY >>> + adr r0, hyp_boot >>> + smc #0 >>> +hyp_boot: >>> +#endif >>> b secondary_startup >>> END(omap5_secondary_startup) >>> /* >> >> hmm, this means that currently running this in a guest will fail to >> bring-up SMP, right? >> > Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build > will not enable CONFIG_KVM_ARM_HOST and things should be fine then. > Right ? Absolutely not. We're way past the "one image per usage" model, and I fully expect the same kernel to boot both as a guest and a host. Otherwise, multiplatform is just a joke. >> Couldn't you create a little wrapper-pen in U-Boot instead, which >> replicates the omap boot protocol and takes care of the hyp-mode >> startup there instead, keeping this completely out of the kernel? >> > Its not just booting but CPU hotplug also follows the same path > so we need the mechanism in kernel to switch mode. > > In general, I think its important to consider the aspect with > CPU PM. CPUs are not going to go through the boot-loaders in > those paths and hence need of HYP entry in the kernel will > be must. I definitely agree that you need to consider PM as well. But we've decided a long time ago that we wouldn't take that kind of code in the kernel. Your CPU has to reach the kernel in HYP, and that is down to your boot protocol to take care off that. There are a few options. Christoffer mentioned having a pen, and you also could implement PSCI fairly easily (I've recently posted a u-boot patch series to take care of this, though there is still a fair bit of work required). Whatever the approach is, the bottom line remains: your CPU enters the kernel in HYP. Cheers, M.
On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: > On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: >>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>>> Boot-CPU entry into the HYP mode is managed in boot-loader but >>>> the secondary CPUs directly jumps to kernel during boot. Same >>>> path is also used for CPU hotplug as well during suspend for >>>> secondary CPU. >>>> >>>> Hence patch the secondary CPU boot path for hyp mode etry. >>>> >>>> Cc: Marc Zyngier <marc.zyngier@arm.com> >>>> Cc: Christoffer Dall <christoffer.dall@linaro.org> >>>> Cc: Tony Lindgren <tony@atomide.com> >>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >>>> --- >>>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >>>> 1 file changed, 7 insertions(+) >>>> >>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >>>> index 75e9295..4844dd8 100644 >>>> --- a/arch/arm/mach-omap2/omap-headsmp.S >>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S >>>> @@ -22,6 +22,7 @@ >>>> >>>> /* Physical address needed since MMU not enabled yet on secondary core */ >>>> #define AUX_CORE_BOOT0_PA 0x48281800 >>>> +#define API_HYP_ENTRY 0x102 >>>> >>>> /* >>>> * OMAP5 specific entry point for secondary CPU to jump from ROM >>>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >>>> and r4, r4, #0x0f >>>> cmp r0, r4 >>>> bne wait >>>> +#ifdef CONFIG_KVM_ARM_HOST >>>> + ldr r12, =API_HYP_ENTRY >>>> + adr r0, hyp_boot >>>> + smc #0 >>>> +hyp_boot: >>>> +#endif >>>> b secondary_startup >>>> END(omap5_secondary_startup) >>>> /* >>> >>> hmm, this means that currently running this in a guest will fail to >>> bring-up SMP, right? >>> >> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build >> will not enable CONFIG_KVM_ARM_HOST and things should be fine then. >> Right ? >> > > That really goes against the whole single binary on all platforms > thing. With multi-platform support you really shouldn't have to > compile your kernel any differently for running as a guest as when > you're running on a host. Someone may even emulate an OMAP5 in QEMU > and you'd certainly want your kvm-enabled kernel to run as both guest > and host. After all, this is not a paravirtualization solution. > Fair enough. >>> Couldn't you create a little wrapper-pen in U-Boot instead, which >>> replicates the omap boot protocol and takes care of the hyp-mode >>> startup there instead, keeping this completely out of the kernel? >>> >> Its not just booting but CPU hotplug also follows the same path >> so we need the mechanism in kernel to switch mode. >> >> In general, I think its important to consider the aspect with >> CPU PM. CPUs are not going to go through the boot-loaders in >> those paths and hence need of HYP entry in the kernel will >> be must. >> > I agree, and PSCI is the obvious only correct answer to this. > > We have discussed this a bit earlier (I think Will Deacon brought this > up - cc'ed), but I don't think anyone had any bright ideas. > > However, we broadly agreed on the fact that for KVM/hyp support, you > need to boot your kernel in that mode, and this is definitely pulling > in the wrong direction. > What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels. CC'ing few more folks and changing the subject line Regards, Santosh
On Monday 25 November 2013 11:42 AM, Marc Zyngier wrote: > On 25/11/13 16:28, Santosh Shilimkar wrote: >> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: >>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>>> Boot-CPU entry into the HYP mode is managed in boot-loader but >>>> the secondary CPUs directly jumps to kernel during boot. Same >>>> path is also used for CPU hotplug as well during suspend for >>>> secondary CPU. >>>> >>>> Hence patch the secondary CPU boot path for hyp mode etry. >>>> >>>> Cc: Marc Zyngier <marc.zyngier@arm.com> >>>> Cc: Christoffer Dall <christoffer.dall@linaro.org> >>>> Cc: Tony Lindgren <tony@atomide.com> >>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >>>> --- >>>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >>>> 1 file changed, 7 insertions(+) >>>> >>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >>>> index 75e9295..4844dd8 100644 >>>> --- a/arch/arm/mach-omap2/omap-headsmp.S >>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S >>>> @@ -22,6 +22,7 @@ >>>> >>>> /* Physical address needed since MMU not enabled yet on secondary core */ >>>> #define AUX_CORE_BOOT0_PA 0x48281800 >>>> +#define API_HYP_ENTRY 0x102 >>>> >>>> /* >>>> * OMAP5 specific entry point for secondary CPU to jump from ROM >>>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >>>> and r4, r4, #0x0f >>>> cmp r0, r4 >>>> bne wait >>>> +#ifdef CONFIG_KVM_ARM_HOST >>>> + ldr r12, =API_HYP_ENTRY >>>> + adr r0, hyp_boot >>>> + smc #0 >>>> +hyp_boot: >>>> +#endif >>>> b secondary_startup >>>> END(omap5_secondary_startup) >>>> /* >>> >>> hmm, this means that currently running this in a guest will fail to >>> bring-up SMP, right? >>> >> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build >> will not enable CONFIG_KVM_ARM_HOST and things should be fine then. >> Right ? > > Absolutely not. We're way past the "one image per usage" model, and I > fully expect the same kernel to boot both as a guest and a host. > Otherwise, multiplatform is just a joke. > >>> Couldn't you create a little wrapper-pen in U-Boot instead, which >>> replicates the omap boot protocol and takes care of the hyp-mode >>> startup there instead, keeping this completely out of the kernel? >>> >> Its not just booting but CPU hotplug also follows the same path >> so we need the mechanism in kernel to switch mode. >> >> In general, I think its important to consider the aspect with >> CPU PM. CPUs are not going to go through the boot-loaders in >> those paths and hence need of HYP entry in the kernel will >> be must. > > I definitely agree that you need to consider PM as well. But we've > decided a long time ago that we wouldn't take that kind of code in the > kernel. Your CPU has to reach the kernel in HYP, and that is down to > your boot protocol to take care off that. > > There are a few options. Christoffer mentioned having a pen, and you > also could implement PSCI fairly easily (I've recently posted a u-boot > patch series to take care of this, though there is still a fair bit of > work required). > > Whatever the approach is, the bottom line remains: your CPU enters the > kernel in HYP. > Our emails crossed. I understand the stand from you guys. Lets continue discussion on other thread.
On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: > On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: > > On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: > >> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: > >>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: > >>>> Boot-CPU entry into the HYP mode is managed in boot-loader but > >>>> the secondary CPUs directly jumps to kernel during boot. Same > >>>> path is also used for CPU hotplug as well during suspend for > >>>> secondary CPU. > >>>> > >>>> Hence patch the secondary CPU boot path for hyp mode etry. > >>>> > >>>> Cc: Marc Zyngier <marc.zyngier@arm.com> > >>>> Cc: Christoffer Dall <christoffer.dall@linaro.org> > >>>> Cc: Tony Lindgren <tony@atomide.com> > >>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> > >>>> --- > >>>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ > >>>> 1 file changed, 7 insertions(+) > >>>> > >>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S > >>>> index 75e9295..4844dd8 100644 > >>>> --- a/arch/arm/mach-omap2/omap-headsmp.S > >>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S > >>>> @@ -22,6 +22,7 @@ > >>>> > >>>> /* Physical address needed since MMU not enabled yet on secondary core */ > >>>> #define AUX_CORE_BOOT0_PA 0x48281800 > >>>> +#define API_HYP_ENTRY 0x102 > >>>> > >>>> /* > >>>> * OMAP5 specific entry point for secondary CPU to jump from ROM > >>>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 > >>>> and r4, r4, #0x0f > >>>> cmp r0, r4 > >>>> bne wait > >>>> +#ifdef CONFIG_KVM_ARM_HOST > >>>> + ldr r12, =API_HYP_ENTRY > >>>> + adr r0, hyp_boot > >>>> + smc #0 > >>>> +hyp_boot: > >>>> +#endif > >>>> b secondary_startup > >>>> END(omap5_secondary_startup) > >>>> /* > >>> > >>> hmm, this means that currently running this in a guest will fail to > >>> bring-up SMP, right? > >>> > >> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build > >> will not enable CONFIG_KVM_ARM_HOST and things should be fine then. > >> Right ? > >> > > > > That really goes against the whole single binary on all platforms > > thing. With multi-platform support you really shouldn't have to > > compile your kernel any differently for running as a guest as when > > you're running on a host. Someone may even emulate an OMAP5 in QEMU > > and you'd certainly want your kvm-enabled kernel to run as both guest > > and host. After all, this is not a paravirtualization solution. > > Fair enough. > > >>> Couldn't you create a little wrapper-pen in U-Boot instead, which > >>> replicates the omap boot protocol and takes care of the hyp-mode > >>> startup there instead, keeping this completely out of the kernel? > >>> > >> Its not just booting but CPU hotplug also follows the same path > >> so we need the mechanism in kernel to switch mode. > >> > >> In general, I think its important to consider the aspect with > >> CPU PM. CPUs are not going to go through the boot-loaders in > >> those paths and hence need of HYP entry in the kernel will > >> be must. > >> > > I agree, and PSCI is the obvious only correct answer to this. > > > > We have discussed this a bit earlier (I think Will Deacon brought this > > up - cc'ed), but I don't think anyone had any bright ideas. > > > > However, we broadly agreed on the fact that for KVM/hyp support, you > > need to boot your kernel in that mode, and this is definitely pulling > > in the wrong direction. > > What I am saying is the platforms like OMAP5 already support PM in > mainline kernel and we can't break that for KVM. Boot-loaders > would be thrashed after boot so you need something which runs > in Kernel or along with Kernel to have equivalent of hyp > switching. > > Am not challenging the agreed direction but we need to solve the > PM problem as well before making "all CPU runs boot-loader for > HYP kernels" as a must have. At least its is a change in boot > strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?
On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: > On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: >> On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: >>> On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>>> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: >>>>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote: >>>>>> Boot-CPU entry into the HYP mode is managed in boot-loader but >>>>>> the secondary CPUs directly jumps to kernel during boot. Same >>>>>> path is also used for CPU hotplug as well during suspend for >>>>>> secondary CPU. >>>>>> >>>>>> Hence patch the secondary CPU boot path for hyp mode etry. >>>>>> >>>>>> Cc: Marc Zyngier <marc.zyngier@arm.com> >>>>>> Cc: Christoffer Dall <christoffer.dall@linaro.org> >>>>>> Cc: Tony Lindgren <tony@atomide.com> >>>>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> >>>>>> --- >>>>>> arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ >>>>>> 1 file changed, 7 insertions(+) >>>>>> >>>>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S >>>>>> index 75e9295..4844dd8 100644 >>>>>> --- a/arch/arm/mach-omap2/omap-headsmp.S >>>>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S >>>>>> @@ -22,6 +22,7 @@ >>>>>> >>>>>> /* Physical address needed since MMU not enabled yet on secondary core */ >>>>>> #define AUX_CORE_BOOT0_PA 0x48281800 >>>>>> +#define API_HYP_ENTRY 0x102 >>>>>> >>>>>> /* >>>>>> * OMAP5 specific entry point for secondary CPU to jump from ROM >>>>>> @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 >>>>>> and r4, r4, #0x0f >>>>>> cmp r0, r4 >>>>>> bne wait >>>>>> +#ifdef CONFIG_KVM_ARM_HOST >>>>>> + ldr r12, =API_HYP_ENTRY >>>>>> + adr r0, hyp_boot >>>>>> + smc #0 >>>>>> +hyp_boot: >>>>>> +#endif >>>>>> b secondary_startup >>>>>> END(omap5_secondary_startup) >>>>>> /* >>>>> >>>>> hmm, this means that currently running this in a guest will fail to >>>>> bring-up SMP, right? >>>>> >>>> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build >>>> will not enable CONFIG_KVM_ARM_HOST and things should be fine then. >>>> Right ? >>>> >>> >>> That really goes against the whole single binary on all platforms >>> thing. With multi-platform support you really shouldn't have to >>> compile your kernel any differently for running as a guest as when >>> you're running on a host. Someone may even emulate an OMAP5 in QEMU >>> and you'd certainly want your kvm-enabled kernel to run as both guest >>> and host. After all, this is not a paravirtualization solution. >> >> Fair enough. >> >>>>> Couldn't you create a little wrapper-pen in U-Boot instead, which >>>>> replicates the omap boot protocol and takes care of the hyp-mode >>>>> startup there instead, keeping this completely out of the kernel? >>>>> >>>> Its not just booting but CPU hotplug also follows the same path >>>> so we need the mechanism in kernel to switch mode. >>>> >>>> In general, I think its important to consider the aspect with >>>> CPU PM. CPUs are not going to go through the boot-loaders in >>>> those paths and hence need of HYP entry in the kernel will >>>> be must. >>>> >>> I agree, and PSCI is the obvious only correct answer to this. >>> >>> We have discussed this a bit earlier (I think Will Deacon brought this >>> up - cc'ed), but I don't think anyone had any bright ideas. >>> >>> However, we broadly agreed on the fact that for KVM/hyp support, you >>> need to boot your kernel in that mode, and this is definitely pulling >>> in the wrong direction. >> >> What I am saying is the platforms like OMAP5 already support PM in >> mainline kernel and we can't break that for KVM. Boot-loaders >> would be thrashed after boot so you need something which runs >> in Kernel or along with Kernel to have equivalent of hyp >> switching. >> >> Am not challenging the agreed direction but we need to solve the >> PM problem as well before making "all CPU runs boot-loader for >> HYP kernels" as a must have. At least its is a change in boot >> strategy from existing kernels. > > Of course I recommend PSCI which covers both hotplug and suspend ;), but > I guess it's not the case for OMAP5. Since OMAP has its own secondary > booting protocol and CPU hotplug re-entry, can you not just use > different entry point when the primary CPU was initially started in Hyp > mode (e.g. omap5_hyp_secondary_startup)? > How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. I will look at PSCI more closely and see what can be done here. Regards, Santosh
On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote: > On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: > > On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: > >> What I am saying is the platforms like OMAP5 already support PM in > >> mainline kernel and we can't break that for KVM. Boot-loaders > >> would be thrashed after boot so you need something which runs > >> in Kernel or along with Kernel to have equivalent of hyp > >> switching. > >> > >> Am not challenging the agreed direction but we need to solve the > >> PM problem as well before making "all CPU runs boot-loader for > >> HYP kernels" as a must have. At least its is a change in boot > >> strategy from existing kernels. > > > > Of course I recommend PSCI which covers both hotplug and suspend ;), but > > I guess it's not the case for OMAP5. Since OMAP has its own secondary > > booting protocol and CPU hotplug re-entry, can you not just use > > different entry point when the primary CPU was initially started in Hyp > > mode (e.g. omap5_hyp_secondary_startup)? > > > How will that solve the guest secondary boot failure case when using > the same kernel binary for guest-boot ? Even for primary CPU which > will be suspended it needs to resume already in HYP mode and its not > going to go through boot-loader. So the low power code needs to have > HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware. > I will look at PSCI more closely and see what can be done here. PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)?
On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: > On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote: >> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: >>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: >>>> What I am saying is the platforms like OMAP5 already support PM in >>>> mainline kernel and we can't break that for KVM. Boot-loaders >>>> would be thrashed after boot so you need something which runs >>>> in Kernel or along with Kernel to have equivalent of hyp >>>> switching. >>>> >>>> Am not challenging the agreed direction but we need to solve the >>>> PM problem as well before making "all CPU runs boot-loader for >>>> HYP kernels" as a must have. At least its is a change in boot >>>> strategy from existing kernels. >>> >>> Of course I recommend PSCI which covers both hotplug and suspend ;), but >>> I guess it's not the case for OMAP5. Since OMAP has its own secondary >>> booting protocol and CPU hotplug re-entry, can you not just use >>> different entry point when the primary CPU was initially started in Hyp >>> mode (e.g. omap5_hyp_secondary_startup)? >>> >> How will that solve the guest secondary boot failure case when using >> the same kernel binary for guest-boot ? Even for primary CPU which >> will be suspended it needs to resume already in HYP mode and its not >> going to go through boot-loader. So the low power code needs to have >> HYP switch code so that CPU resumes in HYP mode. > > Is it late to rewrite the OMAP5 firmware? > Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used. > What I meant is that you have the same kernel binary but with two sets > of entry points, or maybe a single set of entry points and some global > variable that you set during cold boot if the primary CPU is entered in > Hyp mode. So you change the boot loader to switch to Hyp before it > starts the kernel and leave the additional switching to be done by the > secondary entry points (or PM code) based on the variable you set during > cold boot. But I would recommend rewriting the firmware. > >> I will look at PSCI more closely and see what can be done here. > > PSCI has a different secondary start-up and hotplug/suspend protocol and > CPUs wake up in Hyp if available. But this requires changing your > firmware, PSCI is not a kernel-only thing. Do you need to support both > older and newer kernels with the OMAP5 firmware (and possibly same DT)? > Hmmm.. So thats rules out PSCI for OMAP5 then. Keystone actually has nice way to deal with the problem since the boot-monitor code can be patched up and hence I had no trouble implementing the "CPU starts in HYP mode" requirement on it. Regards, Santosh
On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: > On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: > > On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote: > >> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: > >>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: > >>>> What I am saying is the platforms like OMAP5 already support PM in > >>>> mainline kernel and we can't break that for KVM. Boot-loaders > >>>> would be thrashed after boot so you need something which runs > >>>> in Kernel or along with Kernel to have equivalent of hyp > >>>> switching. > >>>> > >>>> Am not challenging the agreed direction but we need to solve the > >>>> PM problem as well before making "all CPU runs boot-loader for > >>>> HYP kernels" as a must have. At least its is a change in boot > >>>> strategy from existing kernels. > >>> > >>> Of course I recommend PSCI which covers both hotplug and suspend ;), but > >>> I guess it's not the case for OMAP5. Since OMAP has its own secondary > >>> booting protocol and CPU hotplug re-entry, can you not just use > >>> different entry point when the primary CPU was initially started in Hyp > >>> mode (e.g. omap5_hyp_secondary_startup)? > >>> > >> How will that solve the guest secondary boot failure case when using > >> the same kernel binary for guest-boot ? Even for primary CPU which > >> will be suspended it needs to resume already in HYP mode and its not > >> going to go through boot-loader. So the low power code needs to have > >> HYP switch code so that CPU resumes in HYP mode. > > > > Is it late to rewrite the OMAP5 firmware? > > > Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement > a secure API which lets you enter into HYP mode and thats the only > thing can be used. If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload. Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5? Cheers ---Dave
On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote: > On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: >> On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: >>> On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote: >>>> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: >>>>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: >>>>>> What I am saying is the platforms like OMAP5 already support PM in >>>>>> mainline kernel and we can't break that for KVM. Boot-loaders >>>>>> would be thrashed after boot so you need something which runs >>>>>> in Kernel or along with Kernel to have equivalent of hyp >>>>>> switching. >>>>>> >>>>>> Am not challenging the agreed direction but we need to solve the >>>>>> PM problem as well before making "all CPU runs boot-loader for >>>>>> HYP kernels" as a must have. At least its is a change in boot >>>>>> strategy from existing kernels. >>>>> >>>>> Of course I recommend PSCI which covers both hotplug and suspend ;), but >>>>> I guess it's not the case for OMAP5. Since OMAP has its own secondary >>>>> booting protocol and CPU hotplug re-entry, can you not just use >>>>> different entry point when the primary CPU was initially started in Hyp >>>>> mode (e.g. omap5_hyp_secondary_startup)? >>>>> >>>> How will that solve the guest secondary boot failure case when using >>>> the same kernel binary for guest-boot ? Even for primary CPU which >>>> will be suspended it needs to resume already in HYP mode and its not >>>> going to go through boot-loader. So the low power code needs to have >>>> HYP switch code so that CPU resumes in HYP mode. >>> >>> Is it late to rewrite the OMAP5 firmware? >>> >> Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement >> a secure API which lets you enter into HYP mode and thats the only >> thing can be used. > > If the ROM is capable of loading some additional signed Secure World > firmware after the ROM itself has booted, PSCI could be implemented > in the second, resident firmware payload. > > Some SoCs ship with boot ROMs that can do that -- is this not the case > for OMAP5? > On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs. regards, Santosh
On Tue, Nov 26, 2013 at 09:49:10PM +0000, Santosh Shilimkar wrote: > On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote: > > On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: > >> On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: > >>> On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote: > >>>> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: > >>>>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote: > >>>>>> What I am saying is the platforms like OMAP5 already support PM in > >>>>>> mainline kernel and we can't break that for KVM. Boot-loaders > >>>>>> would be thrashed after boot so you need something which runs > >>>>>> in Kernel or along with Kernel to have equivalent of hyp > >>>>>> switching. > >>>>>> > >>>>>> Am not challenging the agreed direction but we need to solve the > >>>>>> PM problem as well before making "all CPU runs boot-loader for > >>>>>> HYP kernels" as a must have. At least its is a change in boot > >>>>>> strategy from existing kernels. > >>>>> > >>>>> Of course I recommend PSCI which covers both hotplug and suspend ;), but > >>>>> I guess it's not the case for OMAP5. Since OMAP has its own secondary > >>>>> booting protocol and CPU hotplug re-entry, can you not just use > >>>>> different entry point when the primary CPU was initially started in Hyp > >>>>> mode (e.g. omap5_hyp_secondary_startup)? > >>>>> > >>>> How will that solve the guest secondary boot failure case when using > >>>> the same kernel binary for guest-boot ? Even for primary CPU which > >>>> will be suspended it needs to resume already in HYP mode and its not > >>>> going to go through boot-loader. So the low power code needs to have > >>>> HYP switch code so that CPU resumes in HYP mode. > >>> > >>> Is it late to rewrite the OMAP5 firmware? > >>> > >> Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement > >> a secure API which lets you enter into HYP mode and thats the only > >> thing can be used. > > > > If the ROM is capable of loading some additional signed Secure World > > firmware after the ROM itself has booted, PSCI could be implemented > > in the second, resident firmware payload. > > > > Some SoCs ship with boot ROMs that can do that -- is this not the case > > for OMAP5? > > > On OMAP, the secure devices there is a way to do this but not for > general purpose devices. General purpose devices once you > exit the ROM code, we exit out of security and only way to > re-enter secure word is via ROM implemented monitor/secure > APIs. As last resort, you can still write a bunch of instructions acting as warm-bootloader, reserve a page and add them there. Code required is just a shim layer and should not require more than few bytes, after all it has just to check some variables and jump accordingly (+ calling smc if required). Not pretty, agreed, but still better than patching the kernel IMHO (basically you will end up patching the bootloader - and possibly the dtb - instead, to install the shim mentioned above). Lorenzo
diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S index 75e9295..4844dd8 100644 --- a/arch/arm/mach-omap2/omap-headsmp.S +++ b/arch/arm/mach-omap2/omap-headsmp.S @@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM @@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU. Hence patch the secondary CPU boot path for hyp mode etry. Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Tony Lindgren <tony@atomide.com> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> --- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)