| Message ID | 1450285686-844-3-git-send-email-daniel.thompson@linaro.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi Daniel, On 16/12/15 17:08, Daniel Thompson wrote: > It is possible for the secure world to reserve certain SGI IDs for itself. > Currently we have limited visibility of which IDs are safe to use for IPIs. > > Modify the GIC initialization code to actively search for reserved SGI IDs > and report if any are found. Warn even more loudly if the reserved SGIs > overlap with the normal IPI range. > > When run on an Inforce IFC6410 (Snapdragon 600) this code produces the > following messages: > ~~~ cut here ~~~ > CPU0: Detected reserved SGI IDs: 14-15 > CPU1: Detected reserved SGI IDs: 15 > CPU2: Detected reserved SGI IDs: 15 > CPU3: Detected reserved SGI IDs: 15 > ~~~ cut here ~~~ > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> > --- > drivers/irqchip/irq-gic.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 53 insertions(+) > > diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c > index abf2ffaed392..541622da7049 100644 > --- a/drivers/irqchip/irq-gic.c > +++ b/drivers/irqchip/irq-gic.c > @@ -490,6 +490,7 @@ static void gic_cpu_init(struct gic_chip_data *gic) > void __iomem *base = gic_data_cpu_base(gic); > unsigned int cpu_mask, cpu = smp_processor_id(); > int i; > + DECLARE_BITMAP(sgi_mask, 16); > > /* > * Setting up the CPU map is only relevant for the primary GIC > @@ -511,6 +512,58 @@ static void gic_cpu_init(struct gic_chip_data *gic) > for (i = 0; i < NR_GIC_CPU_IF; i++) > if (i != cpu) > gic_cpu_map[i] &= ~cpu_mask; > + > + /* > + * Fiddle with the SGI set/clear registers to try identify > + * any IPIs that are reserved for secure world. > + */ > + bitmap_fill(sgi_mask, 16); > + > + for (i = 0; i < 16; i++) { > + void __iomem *set_reg = > + dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3); > + void __iomem *clear_reg = > + dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3); > + unsigned long mask = cpu_mask << (8*(i%4)); > + unsigned long flags, pending, after_clear, after_set; Please make these u32, as unsigned long is 64bit on arm64. Another thing to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can save yourself some this hassle shifting things around and just write a single byte. You're already writing 16 times anyway... Another thing to consider is that these locations are only defined on GICv2 and not GICv1, so this patch is likely to cause trouble on older HW. > + > + local_irq_save(flags); Why do you need to do this? The CPU interface is not enabled yet, so I can't see how you could get an interrupt on this CPU. > + > + /* record original value */ > + pending = readl_relaxed(set_reg); > + > + /* clear, test, set, and test again */ > + writel_relaxed(mask, clear_reg); > + after_clear = readl_relaxed(set_reg); > + writel_relaxed(mask, set_reg); > + after_set = readl_relaxed(set_reg); It should be enough to write to the SET register, and read back, as the bit is RAZ/WI when the interrupt is Group-0. > + > + /* restore original value */ > + writel_relaxed(mask & ~pending, clear_reg); > + > + local_irq_restore(flags); > + > + if (mask & ~after_clear && mask & after_set) > + clear_bit(i, sgi_mask); > + } > + > + /* > + * Show the SGI mask if it is "interesting". Here interesting > + * means that the set/clear register is implemented > + * (mask is not full) and it tells us that the secure world > + * has reserved some SGIs (mask is not empty). > + */ > + if (!bitmap_full(sgi_mask, 16) && !bitmap_empty(sgi_mask, 16)) > + pr_info("CPU%d: Detected reserved SGI IDs: %*pbl\n", > + cpu, 16, sgi_mask); > + > + /* > + * Yell if the reserved IDs make the system unviable. > + */ > + if (!bitmap_full(sgi_mask, 16) && > + find_first_bit(sgi_mask, 16) < NR_IPI) > + pr_crit("CPU%d: Not enough SGI IDs; expect failure\n", > + cpu); > } > > gic_cpu_config(dist_base, NULL); > Thanks, M.
On Wed, Dec 16, 2015 at 05:47:09PM +0000, Marc Zyngier wrote: > Hi Daniel, Hi Marc Thanks for the review. > On 16/12/15 17:08, Daniel Thompson wrote: > > It is possible for the secure world to reserve certain SGI IDs for itself. > > Currently we have limited visibility of which IDs are safe to use for IPIs. > > > > Modify the GIC initialization code to actively search for reserved SGI IDs > > and report if any are found. Warn even more loudly if the reserved SGIs > > overlap with the normal IPI range. > > > > When run on an Inforce IFC6410 (Snapdragon 600) this code produces the > > following messages: > > ~~~ cut here ~~~ > > CPU0: Detected reserved SGI IDs: 14-15 > > CPU1: Detected reserved SGI IDs: 15 > > CPU2: Detected reserved SGI IDs: 15 > > CPU3: Detected reserved SGI IDs: 15 > > ~~~ cut here ~~~ > > > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> BTW you *didn't* say "this code is pointless and I hate it"... Does that mean I should be looking at adding similar code for GICv3+? I wanted to guage reactions to this sort of diagnostics before getting carried away! > > + > > + /* > > + * Fiddle with the SGI set/clear registers to try identify > > + * any IPIs that are reserved for secure world. > > + */ > > + bitmap_fill(sgi_mask, 16); > > + > > + for (i = 0; i < 16; i++) { > > + void __iomem *set_reg = > > + dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3); > > + void __iomem *clear_reg = > > + dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3); > > + unsigned long mask = cpu_mask << (8*(i%4)); > > + unsigned long flags, pending, after_clear, after_set; > > Please make these u32, as unsigned long is 64bit on arm64. Another thing > to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can > save yourself some this hassle shifting things around and just write a > single byte. You're already writing 16 times anyway... Will do both. > Another thing to consider is that these locations are only defined on > GICv2 and not GICv1, so this patch is likely to cause trouble on older HW. As presented the code relies on the RAZ/WI property of reserved registers to avoid issues on GICv1; it does not report anything if there appear to be know working SGIs on the assumption we are actually running on a GICv1. You'd prefer an explicit version check? > > + > > + local_irq_save(flags); > > Why do you need to do this? The CPU interface is not enabled yet, so I > can't see how you could get an interrupt on this CPU. Agreed. Can get rid of these. > > + > > + /* record original value */ > > + pending = readl_relaxed(set_reg); > > + > > + /* clear, test, set, and test again */ > > + writel_relaxed(mask, clear_reg); > > + after_clear = readl_relaxed(set_reg); > > + writel_relaxed(mask, set_reg); > > + after_set = readl_relaxed(set_reg); > > It should be enough to write to the SET register, and read back, as the > bit is RAZ/WI when the interrupt is Group-0. Good point. Will simplify. Daniel.
On Thu, 17 Dec 2015 19:26:10 +0000 Daniel Thompson <daniel.thompson@linaro.org> wrote: > On Wed, Dec 16, 2015 at 05:47:09PM +0000, Marc Zyngier wrote: > > Hi Daniel, > > Hi Marc > > Thanks for the review. > > > > On 16/12/15 17:08, Daniel Thompson wrote: > > > It is possible for the secure world to reserve certain SGI IDs for itself. > > > Currently we have limited visibility of which IDs are safe to use for IPIs. > > > > > > Modify the GIC initialization code to actively search for reserved SGI IDs > > > and report if any are found. Warn even more loudly if the reserved SGIs > > > overlap with the normal IPI range. > > > > > > When run on an Inforce IFC6410 (Snapdragon 600) this code produces the > > > following messages: > > > ~~~ cut here ~~~ > > > CPU0: Detected reserved SGI IDs: 14-15 > > > CPU1: Detected reserved SGI IDs: 15 > > > CPU2: Detected reserved SGI IDs: 15 > > > CPU3: Detected reserved SGI IDs: 15 > > > ~~~ cut here ~~~ > > > > > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> > > BTW you *didn't* say "this code is pointless and I hate it"... Well, this was obviously used to detect another issue (patch #1), so I can't see any harm in trying to sanitize things, as long as it doesn't break anything else. I imagine this patch will also trigger on my sunxi platforms, which use a SGI to implement PSCI in secure mode. > Does that mean I should be looking at adding similar code for GICv3+? I > wanted to guage reactions to this sort of diagnostics before getting > carried away! Could be useful as well. > > > > > + > > > + /* > > > + * Fiddle with the SGI set/clear registers to try identify > > > + * any IPIs that are reserved for secure world. > > > + */ > > > + bitmap_fill(sgi_mask, 16); > > > + > > > + for (i = 0; i < 16; i++) { > > > + void __iomem *set_reg = > > > + dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3); > > > + void __iomem *clear_reg = > > > + dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3); > > > + unsigned long mask = cpu_mask << (8*(i%4)); > > > + unsigned long flags, pending, after_clear, after_set; > > > > Please make these u32, as unsigned long is 64bit on arm64. Another thing > > to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can > > save yourself some this hassle shifting things around and just write a > > single byte. You're already writing 16 times anyway... > > Will do both. > > > > Another thing to consider is that these locations are only defined on > > GICv2 and not GICv1, so this patch is likely to cause trouble on older HW. > > As presented the code relies on the RAZ/WI property of reserved > registers to avoid issues on GICv1; it does not report anything if there > appear to be know working SGIs on the assumption we are actually running > on a GICv1. > > You'd prefer an explicit version check? I'd rather be cautious and check for the architecture version, specially if you settle for the byte access mentioned above (a GICv1 may not support byte access and explode unexpectedly). ICPIDR2.ArchRev should be the right thing to check. > > > > > + > > > + local_irq_save(flags); > > > > Why do you need to do this? The CPU interface is not enabled yet, so I > > can't see how you could get an interrupt on this CPU. > > Agreed. Can get rid of these. > > > > > + > > > + /* record original value */ > > > + pending = readl_relaxed(set_reg); > > > + > > > + /* clear, test, set, and test again */ > > > + writel_relaxed(mask, clear_reg); > > > + after_clear = readl_relaxed(set_reg); > > > + writel_relaxed(mask, set_reg); > > > + after_set = readl_relaxed(set_reg); > > > > It should be enough to write to the SET register, and read back, as the > > bit is RAZ/WI when the interrupt is Group-0. > > Good point. Will simplify. I'd also suggest moving the whole thing to a separate function that'd get called from gic_cpu_init(). Thanks, M.
On 18/12/15 07:39, Marc Zyngier wrote: >>> On 16/12/15 17:08, Daniel Thompson wrote: >>>> It is possible for the secure world to reserve certain SGI IDs for itself. >>>> Currently we have limited visibility of which IDs are safe to use for IPIs. >>>> >>>> Modify the GIC initialization code to actively search for reserved SGI IDs >>>> and report if any are found. Warn even more loudly if the reserved SGIs >>>> overlap with the normal IPI range. >>>> >>>> When run on an Inforce IFC6410 (Snapdragon 600) this code produces the >>>> following messages: >>>> ~~~ cut here ~~~ >>>> CPU0: Detected reserved SGI IDs: 14-15 >>>> CPU1: Detected reserved SGI IDs: 15 >>>> CPU2: Detected reserved SGI IDs: 15 >>>> CPU3: Detected reserved SGI IDs: 15 >>>> ~~~ cut here ~~~ >>>> >>>> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> ... >>> Another thing to consider is that these locations are only defined on >>> GICv2 and not GICv1, so this patch is likely to cause trouble on older HW. >> >> As presented the code relies on the RAZ/WI property of reserved >> registers to avoid issues on GICv1; it does not report anything if there >> appear to be know working SGIs on the assumption we are actually running >> on a GICv1. >> >> You'd prefer an explicit version check? > > I'd rather be cautious and check for the architecture version, > specially if you settle for the byte access mentioned above (a GICv1 > may not support byte access and explode unexpectedly). ICPIDR2.ArchRev > should be the right thing to check. Will do. >>>> + >>>> + /* record original value */ >>>> + pending = readl_relaxed(set_reg); >>>> + >>>> + /* clear, test, set, and test again */ >>>> + writel_relaxed(mask, clear_reg); >>>> + after_clear = readl_relaxed(set_reg); >>>> + writel_relaxed(mask, set_reg); >>>> + after_set = readl_relaxed(set_reg); >>> >>> It should be enough to write to the SET register, and read back, as the >>> bit is RAZ/WI when the interrupt is Group-0. >> >> Good point. Will simplify. > > I'd also suggest moving the whole thing to a separate function that'd > get called from gic_cpu_init(). Will do. I think I will also upgrade the pr_crit() to a WARN_ON() and hard code the check to 8 rather than NR_IPI. Currently NR_IPI is small on arm64 so it would be good for us to shout loudly about latent firmware/secure-zone misconfiguration. Daniel.
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c index abf2ffaed392..541622da7049 100644 --- a/drivers/irqchip/irq-gic.c +++ b/drivers/irqchip/irq-gic.c @@ -490,6 +490,7 @@ static void gic_cpu_init(struct gic_chip_data *gic) void __iomem *base = gic_data_cpu_base(gic); unsigned int cpu_mask, cpu = smp_processor_id(); int i; + DECLARE_BITMAP(sgi_mask, 16); /* * Setting up the CPU map is only relevant for the primary GIC @@ -511,6 +512,58 @@ static void gic_cpu_init(struct gic_chip_data *gic) for (i = 0; i < NR_GIC_CPU_IF; i++) if (i != cpu) gic_cpu_map[i] &= ~cpu_mask; + + /* + * Fiddle with the SGI set/clear registers to try identify + * any IPIs that are reserved for secure world. + */ + bitmap_fill(sgi_mask, 16); + + for (i = 0; i < 16; i++) { + void __iomem *set_reg = + dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3); + void __iomem *clear_reg = + dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3); + unsigned long mask = cpu_mask << (8*(i%4)); + unsigned long flags, pending, after_clear, after_set; + + local_irq_save(flags); + + /* record original value */ + pending = readl_relaxed(set_reg); + + /* clear, test, set, and test again */ + writel_relaxed(mask, clear_reg); + after_clear = readl_relaxed(set_reg); + writel_relaxed(mask, set_reg); + after_set = readl_relaxed(set_reg); + + /* restore original value */ + writel_relaxed(mask & ~pending, clear_reg); + + local_irq_restore(flags); + + if (mask & ~after_clear && mask & after_set) + clear_bit(i, sgi_mask); + } + + /* + * Show the SGI mask if it is "interesting". Here interesting + * means that the set/clear register is implemented + * (mask is not full) and it tells us that the secure world + * has reserved some SGIs (mask is not empty). + */ + if (!bitmap_full(sgi_mask, 16) && !bitmap_empty(sgi_mask, 16)) + pr_info("CPU%d: Detected reserved SGI IDs: %*pbl\n", + cpu, 16, sgi_mask); + + /* + * Yell if the reserved IDs make the system unviable. + */ + if (!bitmap_full(sgi_mask, 16) && + find_first_bit(sgi_mask, 16) < NR_IPI) + pr_crit("CPU%d: Not enough SGI IDs; expect failure\n", + cpu); } gic_cpu_config(dist_base, NULL);
It is possible for the secure world to reserve certain SGI IDs for itself. Currently we have limited visibility of which IDs are safe to use for IPIs. Modify the GIC initialization code to actively search for reserved SGI IDs and report if any are found. Warn even more loudly if the reserved SGIs overlap with the normal IPI range. When run on an Inforce IFC6410 (Snapdragon 600) this code produces the following messages: ~~~ cut here ~~~ CPU0: Detected reserved SGI IDs: 14-15 CPU1: Detected reserved SGI IDs: 15 CPU2: Detected reserved SGI IDs: 15 CPU3: Detected reserved SGI IDs: 15 ~~~ cut here ~~~ Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> --- drivers/irqchip/irq-gic.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)