| Message ID | 1467963871-31556-1-git-send-email-dirk.behme@de.bosch.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi Dirk, On 08/07/16 08:44, Dirk Behme wrote: > Xen hypervisor drivers might replace native OS drivers. The result is > that some important clocks that are enabled by the OS in the non-Xen > case are not properly enabled in the presence of Xen. The clocks > property enumerates the clocks that must be enabled by the Xen clock > consumer. > > An example is a serial driver enabled by the hypervisor. Xen must I would say "An example is the UART used by the hypervisor." > consume and enable these clocks in the OS to ensure behavior continues > after firmware configures the UART hardware and corresponding clock > harder. What do you mean by "harder"? Also, relying on DOM0 to enable the clock looks very wrong to me and you give an example which prove that. The UART will be used before hand by Xen, however it will not be possible to use it if you expect DOM0 to enable the clock (or even modify the clock frequency). The clock should be enabled either by the firmware or Xen. But not DOM0. DOM0 should not touch this clock at all. Furthermore, this property could be used for clock associated to device that will be passthrough-ed to a guest. In this case, the clock would be enabled even if the device is not in use which will result more power consumption. > > Up to now, the workaround for this has been to use the Linux kernel > command line parameter 'clk_ignore_unused'. See Xen bug > > http://bugs.xenproject.org/xen/bug/45 > > too. > > To fix this, we will add the "unused" clocks in Xen to the hypervisor > node. The OS has to consume and enable the clocks from the hypervisor > node, then. > > Therefore, check if there is a "clocks" entry in the hypervisor node > and if so consume and enable the given clocks. This prevents the clocks > from being disabled by the OS. > > Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com> > --- > Changes in v3: Use the xen.txt description proposed by Michael. Thanks! > > Changes in v2: Drop the Linux implementation details like clk_disable_unused > in xen.txt. > > Documentation/devicetree/bindings/arm/xen.txt | 13 ++++++++ > arch/arm/xen/enlighten.c | 47 +++++++++++++++++++++++++++ > 2 files changed, 60 insertions(+) > > diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt > index c9b9321..00f2165 100644 > --- a/Documentation/devicetree/bindings/arm/xen.txt > +++ b/Documentation/devicetree/bindings/arm/xen.txt > @@ -17,6 +17,19 @@ the following properties: > A GIC node is also required. > This property is unnecessary when booting Dom0 using ACPI. > > +Optional properties: > + > +clocks: one or more clocks to be enabled > + Xen hypervisor drivers might replace native OS drivers. The result is "native OS" has no meaning on Xen. This seems to be a cumbersome way to say that the device will be used by the hypervisor and hidden to DOM0 (aka hardware domain). > + that some important clocks that are enabled by the OS in the non-Xen > + case are not properly enabled in the presence of Xen. The clocks > + property enumerates the clocks that must be enabled by the Xen clock > + consumer. > + An example is a serial driver enabled by the hypervisor. Xen must > + consume and enable these clocks in the OS to ensure behavior continues > + after firmware configures the UART hardware and corresponding clock > + harder. > + > To support UEFI on Xen ARM virtual platforms, Xen populates the FDT "uefi" node > under /hypervisor with following parameters: Regards,
Hi Michael and Julien, On 08.07.2016 11:34, Julien Grall wrote: > Hi Dirk, > > On 08/07/16 08:44, Dirk Behme wrote: >> Xen hypervisor drivers might replace native OS drivers. The result is >> that some important clocks that are enabled by the OS in the non-Xen >> case are not properly enabled in the presence of Xen. The clocks >> property enumerates the clocks that must be enabled by the Xen clock >> consumer. >> >> An example is a serial driver enabled by the hypervisor. Xen must > > I would say "An example is the UART used by the hypervisor." > >> consume and enable these clocks in the OS to ensure behavior continues >> after firmware configures the UART hardware and corresponding clock >> harder. > > What do you mean by "harder"? > > Also, relying on DOM0 to enable the clock looks very wrong to me and you > give an example which prove that. The UART will be used before hand by > Xen, however it will not be possible to use it if you expect DOM0 to > enable the clock (or even modify the clock frequency). > > The clock should be enabled either by the firmware or Xen. But not DOM0. > DOM0 should not touch this clock at all. > > Furthermore, this property could be used for clock associated to device > that will be passthrough-ed to a guest. In this case, the clock would be > enabled even if the device is not in use which will result more power > consumption. I took the description directly from Michael's proposal http://www.spinics.net/lists/arm-kernel/msg516576.html Would it be possible that you two experts agree on the exact wording you like to see? Best regards Dirk >> Up to now, the workaround for this has been to use the Linux kernel >> command line parameter 'clk_ignore_unused'. See Xen bug >> >> http://bugs.xenproject.org/xen/bug/45 >> >> too. >> >> To fix this, we will add the "unused" clocks in Xen to the hypervisor >> node. The OS has to consume and enable the clocks from the hypervisor >> node, then. >> >> Therefore, check if there is a "clocks" entry in the hypervisor node >> and if so consume and enable the given clocks. This prevents the clocks >> from being disabled by the OS. >> >> Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com> >> --- >> Changes in v3: Use the xen.txt description proposed by Michael. Thanks! >> >> Changes in v2: Drop the Linux implementation details like >> clk_disable_unused >> in xen.txt. >> >> Documentation/devicetree/bindings/arm/xen.txt | 13 ++++++++ >> arch/arm/xen/enlighten.c | 47 >> +++++++++++++++++++++++++++ >> 2 files changed, 60 insertions(+) >> >> diff --git a/Documentation/devicetree/bindings/arm/xen.txt >> b/Documentation/devicetree/bindings/arm/xen.txt >> index c9b9321..00f2165 100644 >> --- a/Documentation/devicetree/bindings/arm/xen.txt >> +++ b/Documentation/devicetree/bindings/arm/xen.txt >> @@ -17,6 +17,19 @@ the following properties: >> A GIC node is also required. >> This property is unnecessary when booting Dom0 using ACPI. >> >> +Optional properties: >> + >> +clocks: one or more clocks to be enabled >> + Xen hypervisor drivers might replace native OS drivers. The result is > > "native OS" has no meaning on Xen. This seems to be a cumbersome way to > say that the device will be used by the hypervisor and hidden to DOM0 > (aka hardware domain). > >> + that some important clocks that are enabled by the OS in the non-Xen >> + case are not properly enabled in the presence of Xen. The clocks >> + property enumerates the clocks that must be enabled by the Xen clock >> + consumer. >> + An example is a serial driver enabled by the hypervisor. Xen must >> + consume and enable these clocks in the OS to ensure behavior continues >> + after firmware configures the UART hardware and corresponding clock >> + harder. >> + >> To support UEFI on Xen ARM virtual platforms, Xen populates the FDT >> "uefi" node >> under /hypervisor with following parameters: > > Regards, >
On 08/07/16 11:40, Dirk Behme wrote: > Hi Michael and Julien, > > On 08.07.2016 11:34, Julien Grall wrote: >> Hi Dirk, >> >> On 08/07/16 08:44, Dirk Behme wrote: >>> Xen hypervisor drivers might replace native OS drivers. The result is >>> that some important clocks that are enabled by the OS in the non-Xen >>> case are not properly enabled in the presence of Xen. The clocks >>> property enumerates the clocks that must be enabled by the Xen clock >>> consumer. >>> >>> An example is a serial driver enabled by the hypervisor. Xen must >> >> I would say "An example is the UART used by the hypervisor." >> >>> consume and enable these clocks in the OS to ensure behavior continues >>> after firmware configures the UART hardware and corresponding clock >>> harder. >> >> What do you mean by "harder"? >> >> Also, relying on DOM0 to enable the clock looks very wrong to me and you >> give an example which prove that. The UART will be used before hand by >> Xen, however it will not be possible to use it if you expect DOM0 to >> enable the clock (or even modify the clock frequency). >> >> The clock should be enabled either by the firmware or Xen. But not DOM0. >> DOM0 should not touch this clock at all. >> >> Furthermore, this property could be used for clock associated to device >> that will be passthrough-ed to a guest. In this case, the clock would be >> enabled even if the device is not in use which will result more power >> consumption. > > > I took the description directly from Michael's proposal > > http://www.spinics.net/lists/arm-kernel/msg516576.html > > Would it be possible that you two experts agree on the exact wording you > like to see? I think the wording suggested by Mark in [1] represents better what we would like to achieve with this property. Regards, [1] https://www.spinics.net/lists/arm-kernel/msg516158.html
Quoting Julien Grall (2016-07-08 03:49:33) > > > On 08/07/16 11:40, Dirk Behme wrote: > > Hi Michael and Julien, > > > > On 08.07.2016 11:34, Julien Grall wrote: > >> Hi Dirk, > >> > >> On 08/07/16 08:44, Dirk Behme wrote: > >>> Xen hypervisor drivers might replace native OS drivers. The result is > >>> that some important clocks that are enabled by the OS in the non-Xen > >>> case are not properly enabled in the presence of Xen. The clocks > >>> property enumerates the clocks that must be enabled by the Xen clock > >>> consumer. > >>> > >>> An example is a serial driver enabled by the hypervisor. Xen must > >> > >> I would say "An example is the UART used by the hypervisor." > >> > >>> consume and enable these clocks in the OS to ensure behavior continues > >>> after firmware configures the UART hardware and corresponding clock > >>> harder. > >> > >> What do you mean by "harder"? > >> > >> Also, relying on DOM0 to enable the clock looks very wrong to me and you > >> give an example which prove that. The UART will be used before hand by > >> Xen, however it will not be possible to use it if you expect DOM0 to > >> enable the clock (or even modify the clock frequency). > >> > >> The clock should be enabled either by the firmware or Xen. But not DOM0. > >> DOM0 should not touch this clock at all. > >> > >> Furthermore, this property could be used for clock associated to device > >> that will be passthrough-ed to a guest. In this case, the clock would be > >> enabled even if the device is not in use which will result more power > >> consumption. > > > > > > I took the description directly from Michael's proposal > > > > http://www.spinics.net/lists/arm-kernel/msg516576.html > > > > Would it be possible that you two experts agree on the exact wording you > > like to see? > > I think the wording suggested by Mark in [1] represents better what we > would like to achieve with this property. Mark's copy is fine by me. The main thing is to avoid talking about "unused" clocks being disabled by the OS. If you have need clocks to be enabled, then you simply claim them and enable them. All of the other stuff is just noise. Regards, Mike > > Regards, > > [1] https://www.spinics.net/lists/arm-kernel/msg516158.html > > -- > Julien Grall
Quoting Julien Grall (2016-07-08 02:34:43) > Hi Dirk, > > On 08/07/16 08:44, Dirk Behme wrote: > > Xen hypervisor drivers might replace native OS drivers. The result is > > that some important clocks that are enabled by the OS in the non-Xen > > case are not properly enabled in the presence of Xen. The clocks > > property enumerates the clocks that must be enabled by the Xen clock > > consumer. > > > > An example is a serial driver enabled by the hypervisor. Xen must > > I would say "An example is the UART used by the hypervisor." > > > consume and enable these clocks in the OS to ensure behavior continues > > after firmware configures the UART hardware and corresponding clock > > harder. > > What do you mean by "harder"? > > Also, relying on DOM0 to enable the clock looks very wrong to me and you > give an example which prove that. The UART will be used before hand by > Xen, however it will not be possible to use it if you expect DOM0 to > enable the clock (or even modify the clock frequency). > > The clock should be enabled either by the firmware or Xen. But not DOM0. > DOM0 should not touch this clock at all. > > Furthermore, this property could be used for clock associated to device > that will be passthrough-ed to a guest. In this case, the clock would be > enabled even if the device is not in use which will result more power > consumption. Is there a need to pass clock references through to guests? If so the unmerged CLK_ENABLE_HAND_OFF[0] feature might be useful to you? If this flag is set on a given clk, it will be enabled at the time it is registered by the clk provider driver, and it's enable_count reference counter will be "handed off" to the first consumer that calls clk_get() and clk_prepare_enable() on it. This means the clock CAN be gated by it's proper driver, but it will be enabled at boot time as well. This is useful for use cases like splash screens where the bootloader configures the display and plays some animation, and we want the linux kernel to take over the display controller hardware without cutting clocks, blanking or reseting it. Handing off the clock reference count helps achieve this. [0] lkml.kernel.org/g/1455225554-13267-1-git-send-email-mturquette@baylibre.com Regards, Mike > > > > > Up to now, the workaround for this has been to use the Linux kernel > > command line parameter 'clk_ignore_unused'. See Xen bug > > > > http://bugs.xenproject.org/xen/bug/45 > > > > too. > > > > To fix this, we will add the "unused" clocks in Xen to the hypervisor > > node. The OS has to consume and enable the clocks from the hypervisor > > node, then. > > > > Therefore, check if there is a "clocks" entry in the hypervisor node > > and if so consume and enable the given clocks. This prevents the clocks > > from being disabled by the OS. > > > > Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com> > > --- > > Changes in v3: Use the xen.txt description proposed by Michael. Thanks! > > > > Changes in v2: Drop the Linux implementation details like clk_disable_unused > > in xen.txt. > > > > Documentation/devicetree/bindings/arm/xen.txt | 13 ++++++++ > > arch/arm/xen/enlighten.c | 47 +++++++++++++++++++++++++++ > > 2 files changed, 60 insertions(+) > > > > diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt > > index c9b9321..00f2165 100644 > > --- a/Documentation/devicetree/bindings/arm/xen.txt > > +++ b/Documentation/devicetree/bindings/arm/xen.txt > > @@ -17,6 +17,19 @@ the following properties: > > A GIC node is also required. > > This property is unnecessary when booting Dom0 using ACPI. > > > > +Optional properties: > > + > > +clocks: one or more clocks to be enabled > > + Xen hypervisor drivers might replace native OS drivers. The result is > > "native OS" has no meaning on Xen. This seems to be a cumbersome way to > say that the device will be used by the hypervisor and hidden to DOM0 > (aka hardware domain). > > > + that some important clocks that are enabled by the OS in the non-Xen > > + case are not properly enabled in the presence of Xen. The clocks > > + property enumerates the clocks that must be enabled by the Xen clock > > + consumer. > > + An example is a serial driver enabled by the hypervisor. Xen must > > + consume and enable these clocks in the OS to ensure behavior continues > > + after firmware configures the UART hardware and corresponding clock > > + harder. > > + > > To support UEFI on Xen ARM virtual platforms, Xen populates the FDT "uefi" node > > under /hypervisor with following parameters: > > Regards, > > -- > Julien Grall
Hi Mike, On 08/07/16 18:06, Michael Turquette wrote: > Quoting Julien Grall (2016-07-08 02:34:43) >> Hi Dirk, >> >> On 08/07/16 08:44, Dirk Behme wrote: >>> Xen hypervisor drivers might replace native OS drivers. The result is >>> that some important clocks that are enabled by the OS in the non-Xen >>> case are not properly enabled in the presence of Xen. The clocks >>> property enumerates the clocks that must be enabled by the Xen clock >>> consumer. >>> >>> An example is a serial driver enabled by the hypervisor. Xen must >> >> I would say "An example is the UART used by the hypervisor." >> >>> consume and enable these clocks in the OS to ensure behavior continues >>> after firmware configures the UART hardware and corresponding clock >>> harder. >> >> What do you mean by "harder"? >> >> Also, relying on DOM0 to enable the clock looks very wrong to me and you >> give an example which prove that. The UART will be used before hand by >> Xen, however it will not be possible to use it if you expect DOM0 to >> enable the clock (or even modify the clock frequency). >> >> The clock should be enabled either by the firmware or Xen. But not DOM0. >> DOM0 should not touch this clock at all. >> >> Furthermore, this property could be used for clock associated to device >> that will be passthrough-ed to a guest. In this case, the clock would be >> enabled even if the device is not in use which will result more power >> consumption. > > Is there a need to pass clock references through to guests? If so the > unmerged CLK_ENABLE_HAND_OFF[0] feature might be useful to you? If this > flag is set on a given clk, it will be enabled at the time it is > registered by the clk provider driver, and it's enable_count reference > counter will be "handed off" to the first consumer that calls clk_get() > and clk_prepare_enable() on it. This means the clock CAN be gated by > it's proper driver, but it will be enabled at boot time as well. Some driver requires to have the clock in hand to be able to configure the clock (such as setting the rate). So we would have to find a way to let the guest using the clock either by assigning the clock or some PV clock driver. However, platform device passthrough (i.e non-pci device) cannot be done generically. The user has to provide a lots of information manually (such as MMIO, IRQ, device tree node...). So I am not sure if we want to have a generic solution here. I though it would be worth to mention it because we may (or not) use this clock to tell DOM0 (don't touch it). > This is useful for use cases like splash screens where the bootloader > configures the display and plays some animation, and we want the linux > kernel to take over the display controller hardware without cutting > clocks, blanking or reseting it. Handing off the clock reference count > helps achieve this. From my understanding, any device used by Xen would be in a similar situation, although there will be no driver in Linux. The current patch (as well as the v4) is calling clk_prepare_enable for each clock used by Xen. Could enabling the clock create unexpected behavior such as the UART loosing/dropping characters? Regards,
Quoting Julien Grall (2016-07-12 02:21:12) > Hi Mike, > > On 08/07/16 18:06, Michael Turquette wrote: > > Quoting Julien Grall (2016-07-08 02:34:43) > >> Hi Dirk, > >> > >> On 08/07/16 08:44, Dirk Behme wrote: > >>> Xen hypervisor drivers might replace native OS drivers. The result is > >>> that some important clocks that are enabled by the OS in the non-Xen > >>> case are not properly enabled in the presence of Xen. The clocks > >>> property enumerates the clocks that must be enabled by the Xen clock > >>> consumer. > >>> > >>> An example is a serial driver enabled by the hypervisor. Xen must > >> > >> I would say "An example is the UART used by the hypervisor." > >> > >>> consume and enable these clocks in the OS to ensure behavior continues > >>> after firmware configures the UART hardware and corresponding clock > >>> harder. > >> > >> What do you mean by "harder"? > >> > >> Also, relying on DOM0 to enable the clock looks very wrong to me and you > >> give an example which prove that. The UART will be used before hand by > >> Xen, however it will not be possible to use it if you expect DOM0 to > >> enable the clock (or even modify the clock frequency). > >> > >> The clock should be enabled either by the firmware or Xen. But not DOM0. > >> DOM0 should not touch this clock at all. > >> > >> Furthermore, this property could be used for clock associated to device > >> that will be passthrough-ed to a guest. In this case, the clock would be > >> enabled even if the device is not in use which will result more power > >> consumption. > > > > Is there a need to pass clock references through to guests? If so the > > unmerged CLK_ENABLE_HAND_OFF[0] feature might be useful to you? If this > > flag is set on a given clk, it will be enabled at the time it is > > registered by the clk provider driver, and it's enable_count reference > > counter will be "handed off" to the first consumer that calls clk_get() > > and clk_prepare_enable() on it. This means the clock CAN be gated by > > it's proper driver, but it will be enabled at boot time as well. > > Some driver requires to have the clock in hand to be able to configure > the clock (such as setting the rate). So we would have to find a way to > let the guest using the clock either by assigning the clock or some PV > clock driver. > > However, platform device passthrough (i.e non-pci device) cannot be done > generically. The user has to provide a lots of information manually > (such as MMIO, IRQ, device tree node...). So I am not sure if we want to > have a generic solution here. > > I though it would be worth to mention it because we may (or not) use > this clock to tell DOM0 (don't touch it). > > > This is useful for use cases like splash screens where the bootloader > > configures the display and plays some animation, and we want the linux > > kernel to take over the display controller hardware without cutting > > clocks, blanking or reseting it. Handing off the clock reference count > > helps achieve this. > > From my understanding, any device used by Xen would be in a similar > situation, although there will be no driver in Linux. The current patch > (as well as the v4) is calling clk_prepare_enable for each clock used by > Xen. Could enabling the clock create unexpected behavior such as the > UART loosing/dropping characters? In general, no, it will not cause any bad side effect. Regards, Mike > > Regards, > > -- > Julien Grall
diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt index c9b9321..00f2165 100644 --- a/Documentation/devicetree/bindings/arm/xen.txt +++ b/Documentation/devicetree/bindings/arm/xen.txt @@ -17,6 +17,19 @@ the following properties: A GIC node is also required. This property is unnecessary when booting Dom0 using ACPI. +Optional properties: + +clocks: one or more clocks to be enabled + Xen hypervisor drivers might replace native OS drivers. The result is + that some important clocks that are enabled by the OS in the non-Xen + case are not properly enabled in the presence of Xen. The clocks + property enumerates the clocks that must be enabled by the Xen clock + consumer. + An example is a serial driver enabled by the hypervisor. Xen must + consume and enable these clocks in the OS to ensure behavior continues + after firmware configures the UART hardware and corresponding clock + harder. + To support UEFI on Xen ARM virtual platforms, Xen populates the FDT "uefi" node under /hypervisor with following parameters: diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c index 47acb36..5c546d0 100644 --- a/arch/arm/xen/enlighten.c +++ b/arch/arm/xen/enlighten.c @@ -24,6 +24,7 @@ #include <linux/of_fdt.h> #include <linux/of_irq.h> #include <linux/of_address.h> +#include <linux/clk-provider.h> #include <linux/cpuidle.h> #include <linux/cpufreq.h> #include <linux/cpu.h> @@ -444,6 +445,52 @@ static int __init xen_pm_init(void) } late_initcall(xen_pm_init); +/* + * Check if we want to register some clocks, that they + * are not freed because unused by clk_disable_unused(). + * E.g. the serial console clock. + */ +static int __init xen_arm_register_clks(void) +{ + struct clk *clk; + struct device_node *xen_node; + unsigned int i, count; + int ret = 0; + + xen_node = of_find_compatible_node(NULL, NULL, "xen,xen"); + if (!xen_node) { + pr_err("Xen support was detected before, but it has disappeared\n"); + return -EINVAL; + } + + count = of_clk_get_parent_count(xen_node); + if (!count) + goto out; + + for (i = 0; i < count; i++) { + clk = of_clk_get(xen_node, i); + if (IS_ERR(clk)) { + pr_err("Xen failed to register clock %i. Error: %li\n", + i, PTR_ERR(clk)); + ret = PTR_ERR(clk); + goto out; + } + + ret = clk_prepare_enable(clk); + if (ret < 0) { + pr_err("Xen failed to enable clock %i. Error: %i\n", + i, ret); + goto out; + } + } + + ret = 0; + +out: + of_node_put(xen_node); + return ret; +} +late_initcall(xen_arm_register_clks); /* empty stubs */ void xen_arch_pre_suspend(void) { }
Xen hypervisor drivers might replace native OS drivers. The result is that some important clocks that are enabled by the OS in the non-Xen case are not properly enabled in the presence of Xen. The clocks property enumerates the clocks that must be enabled by the Xen clock consumer. An example is a serial driver enabled by the hypervisor. Xen must consume and enable these clocks in the OS to ensure behavior continues after firmware configures the UART hardware and corresponding clock harder. Up to now, the workaround for this has been to use the Linux kernel command line parameter 'clk_ignore_unused'. See Xen bug http://bugs.xenproject.org/xen/bug/45 too. To fix this, we will add the "unused" clocks in Xen to the hypervisor node. The OS has to consume and enable the clocks from the hypervisor node, then. Therefore, check if there is a "clocks" entry in the hypervisor node and if so consume and enable the given clocks. This prevents the clocks from being disabled by the OS. Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com> --- Changes in v3: Use the xen.txt description proposed by Michael. Thanks! Changes in v2: Drop the Linux implementation details like clk_disable_unused in xen.txt. Documentation/devicetree/bindings/arm/xen.txt | 13 ++++++++ arch/arm/xen/enlighten.c | 47 +++++++++++++++++++++++++++ 2 files changed, 60 insertions(+)