| Message ID | 20221115143824.2798908-3-ardb@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | arm64: mm: Model LVA support as a CPU feature | expand |
On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > extremely early, before creating the kernel page tables or enabling the > MMU. We cannot override the feature this early, and so large virtual > addressing is always enabled on CPUs that implement support for it if > the software support for it was enabled at build time. It also means we > rely on non-trivial code in asm to deal with this feature. > > Given that both the ID map and the TTBR1 mapping of the kernel image are > guaranteed to be 48-bit addressable, it is not actually necessary to > enable support this early, and instead, we can model it as a CPU > feature. That way, we can rely on code patching to get the correct > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > On the primary boot path, we simply enable the MMU with 48-bit virtual > addressing initially, and update TCR.T1SZ from C code if LVA is > supported, right before creating the kernel mapping. Given that TTBR1 > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > be safe without the need for explicit TLB maintenance. I'm not sure we can skip the TLBI here. There's some weird rule in the ARM ARM that if you change any of fields that may be cached in the TLB, maintenance is needed even if the MMU is off. From the latest version (I.a, I didn't dig into H.a), R_VNRFW: When a System register field is modified and that field is permitted to be cached in a TLB, software is required to invalidate all TLB entries that might be affected by the field, at any address translation stage in the translation regime even if the translation stage is disabled, using the appropriate VMID and ASID, after any required System register synchronization.
On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > extremely early, before creating the kernel page tables or enabling the > > MMU. We cannot override the feature this early, and so large virtual > > addressing is always enabled on CPUs that implement support for it if > > the software support for it was enabled at build time. It also means we > > rely on non-trivial code in asm to deal with this feature. > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > guaranteed to be 48-bit addressable, it is not actually necessary to > > enable support this early, and instead, we can model it as a CPU > > feature. That way, we can rely on code patching to get the correct > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > addressing initially, and update TCR.T1SZ from C code if LVA is > > supported, right before creating the kernel mapping. Given that TTBR1 > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > be safe without the need for explicit TLB maintenance. > > I'm not sure we can skip the TLBI here. There's some weird rule in the > ARM ARM that if you change any of fields that may be cached in the TLB, > maintenance is needed even if the MMU is off. From the latest version > (I.a, I didn't dig into H.a), > > R_VNRFW: > When a System register field is modified and that field is permitted > to be cached in a TLB, software is required to invalidate all TLB > entries that might be affected by the field, at any address > translation stage in the translation regime even if the translation > stage is disabled, using the appropriate VMID and ASID, after any > required System register synchronization. > Don't we already rely on this in cpu_set_default_tcr_t0sz() / cpu_set_idmap_tcr_t0sz() ?
On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > extremely early, before creating the kernel page tables or enabling the > > > MMU. We cannot override the feature this early, and so large virtual > > > addressing is always enabled on CPUs that implement support for it if > > > the software support for it was enabled at build time. It also means we > > > rely on non-trivial code in asm to deal with this feature. > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > enable support this early, and instead, we can model it as a CPU > > > feature. That way, we can rely on code patching to get the correct > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > be safe without the need for explicit TLB maintenance. > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > ARM ARM that if you change any of fields that may be cached in the TLB, > > maintenance is needed even if the MMU is off. From the latest version > > (I.a, I didn't dig into H.a), > > > > R_VNRFW: > > When a System register field is modified and that field is permitted > > to be cached in a TLB, software is required to invalidate all TLB > > entries that might be affected by the field, at any address > > translation stage in the translation regime even if the translation > > stage is disabled, using the appropriate VMID and ASID, after any > > required System register synchronization. > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > cpu_set_idmap_tcr_t0sz() ? Yeah, we do this and depending on how you read the above rule, we may need to move the local_flush_tlb_all() line after T0SZ setting.
On Wed, 30 Nov 2022 at 17:28, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > > extremely early, before creating the kernel page tables or enabling the > > > > MMU. We cannot override the feature this early, and so large virtual > > > > addressing is always enabled on CPUs that implement support for it if > > > > the software support for it was enabled at build time. It also means we > > > > rely on non-trivial code in asm to deal with this feature. > > > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > > enable support this early, and instead, we can model it as a CPU > > > > feature. That way, we can rely on code patching to get the correct > > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > > be safe without the need for explicit TLB maintenance. > > > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > > ARM ARM that if you change any of fields that may be cached in the TLB, > > > maintenance is needed even if the MMU is off. From the latest version > > > (I.a, I didn't dig into H.a), > > > > > > R_VNRFW: > > > When a System register field is modified and that field is permitted > > > to be cached in a TLB, software is required to invalidate all TLB > > > entries that might be affected by the field, at any address > > > translation stage in the translation regime even if the translation > > > stage is disabled, using the appropriate VMID and ASID, after any > > > required System register synchronization. > > > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > > cpu_set_idmap_tcr_t0sz() ? > > Yeah, we do this and depending on how you read the above rule, we may > need to move the local_flush_tlb_all() line after T0SZ setting. > OK, so wouldn't this mean that we cannot change TxSZ at all without going through a MMU off/on cycle?
On Wed, Nov 30, 2022 at 05:29:55PM +0100, Ard Biesheuvel wrote: > On Wed, 30 Nov 2022 at 17:28, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > > > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > > > extremely early, before creating the kernel page tables or enabling the > > > > > MMU. We cannot override the feature this early, and so large virtual > > > > > addressing is always enabled on CPUs that implement support for it if > > > > > the software support for it was enabled at build time. It also means we > > > > > rely on non-trivial code in asm to deal with this feature. > > > > > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > > > enable support this early, and instead, we can model it as a CPU > > > > > feature. That way, we can rely on code patching to get the correct > > > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > > > be safe without the need for explicit TLB maintenance. > > > > > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > > > ARM ARM that if you change any of fields that may be cached in the TLB, > > > > maintenance is needed even if the MMU is off. From the latest version > > > > (I.a, I didn't dig into H.a), > > > > > > > > R_VNRFW: > > > > When a System register field is modified and that field is permitted > > > > to be cached in a TLB, software is required to invalidate all TLB > > > > entries that might be affected by the field, at any address > > > > translation stage in the translation regime even if the translation > > > > stage is disabled, using the appropriate VMID and ASID, after any > > > > required System register synchronization. > > > > > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > > > cpu_set_idmap_tcr_t0sz() ? > > > > Yeah, we do this and depending on how you read the above rule, we may > > need to move the local_flush_tlb_all() line after T0SZ setting. > > OK, so wouldn't this mean that we cannot change TxSZ at all without > going through a MMU off/on cycle? I don't think so. The way I see it is that the change is not guaranteed to take effect until we invalidate the TLBs. We don't risk fetching random stuff in the TLB since we have the reserved TTBR0 at that point. If the subsequent cpu_switch_mm() changed the ASID, in some interpretation of the ARM ARM we could have skipped the TLBI but that's not the case anyway. Looking for Mark R's opinion as I recall he talked to the architects in the past about this (though I think it was in the context of SCTLR_EL1).
On Wed, Nov 30, 2022 at 04:40:11PM +0000, Catalin Marinas wrote: > On Wed, Nov 30, 2022 at 05:29:55PM +0100, Ard Biesheuvel wrote: > > On Wed, 30 Nov 2022 at 17:28, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > > > > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > > > > extremely early, before creating the kernel page tables or enabling the > > > > > > MMU. We cannot override the feature this early, and so large virtual > > > > > > addressing is always enabled on CPUs that implement support for it if > > > > > > the software support for it was enabled at build time. It also means we > > > > > > rely on non-trivial code in asm to deal with this feature. > > > > > > > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > > > > enable support this early, and instead, we can model it as a CPU > > > > > > feature. That way, we can rely on code patching to get the correct > > > > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > > > > be safe without the need for explicit TLB maintenance. > > > > > > > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > > > > ARM ARM that if you change any of fields that may be cached in the TLB, > > > > > maintenance is needed even if the MMU is off. From the latest version > > > > > (I.a, I didn't dig into H.a), > > > > > > > > > > R_VNRFW: > > > > > When a System register field is modified and that field is permitted > > > > > to be cached in a TLB, software is required to invalidate all TLB > > > > > entries that might be affected by the field, at any address > > > > > translation stage in the translation regime even if the translation > > > > > stage is disabled, using the appropriate VMID and ASID, after any > > > > > required System register synchronization. > > > > > > > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > > > > cpu_set_idmap_tcr_t0sz() ? > > > > > > Yeah, we do this and depending on how you read the above rule, we may > > > need to move the local_flush_tlb_all() line after T0SZ setting. > > > > OK, so wouldn't this mean that we cannot change TxSZ at all without > > going through a MMU off/on cycle? > > I don't think so. The way I see it is that the change is not guaranteed > to take effect until we invalidate the TLBs. We don't risk fetching > random stuff in the TLB since we have the reserved TTBR0 at that point. > If the subsequent cpu_switch_mm() changed the ASID, in some > interpretation of the ARM ARM we could have skipped the TLBI but that's > not the case anyway. > > Looking for Mark R's opinion as I recall he talked to the architects in > the past about this (though I think it was in the context of SCTLR_EL1). The architecture is unfortunately vague here. From prior discussions with architects, the general rule was "if it's permitted to be cached in a TLB, an update must be followed by a TLBI for that update to take effect, regardless of SCTLR_ELx.M". The architecture isn't very specific about what scope fo maintenance is required (e.g. if certain fields are tagged by ASID/VMID), but I beleive it's sufficient to use a (VM)ALL for the current translation regime (which might be stronger than necessary). So for this case, my understanding is: 1) When changing TxSZ, we need a subsequent invalidate before the change is guaranteed to take effect. So I believe that what we do today isn't quite right. 2) During the window between writing to TxSZ and completing the invalidation, I'm not sure how the MMU is permitted to behave w.r.t. intermediate walk entries. I could imagine that those become (CONSTRAINED) UNPREDICTABLE , and that we might need to ensure those are invalidated (or inactive and prior walks completed) before we write to TCR_EL1. I can go chase this up with the architects; in the mean time my thinking would be that we should retain the existing maintenance. There's almost certainly more stuff that we'd need to fix in this area. Thanks, Mark.
On Thu, 1 Dec 2022 at 12:14, Mark Rutland <mark.rutland@arm.com> wrote: > > On Wed, Nov 30, 2022 at 04:40:11PM +0000, Catalin Marinas wrote: > > On Wed, Nov 30, 2022 at 05:29:55PM +0100, Ard Biesheuvel wrote: > > > On Wed, 30 Nov 2022 at 17:28, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > > > > > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > > > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > > > > > extremely early, before creating the kernel page tables or enabling the > > > > > > > MMU. We cannot override the feature this early, and so large virtual > > > > > > > addressing is always enabled on CPUs that implement support for it if > > > > > > > the software support for it was enabled at build time. It also means we > > > > > > > rely on non-trivial code in asm to deal with this feature. > > > > > > > > > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > > > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > > > > > enable support this early, and instead, we can model it as a CPU > > > > > > > feature. That way, we can rely on code patching to get the correct > > > > > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > > > > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > > > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > > > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > > > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > > > > > be safe without the need for explicit TLB maintenance. > > > > > > > > > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > > > > > ARM ARM that if you change any of fields that may be cached in the TLB, > > > > > > maintenance is needed even if the MMU is off. From the latest version > > > > > > (I.a, I didn't dig into H.a), > > > > > > > > > > > > R_VNRFW: > > > > > > When a System register field is modified and that field is permitted > > > > > > to be cached in a TLB, software is required to invalidate all TLB > > > > > > entries that might be affected by the field, at any address > > > > > > translation stage in the translation regime even if the translation > > > > > > stage is disabled, using the appropriate VMID and ASID, after any > > > > > > required System register synchronization. > > > > > > > > > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > > > > > cpu_set_idmap_tcr_t0sz() ? > > > > > > > > Yeah, we do this and depending on how you read the above rule, we may > > > > need to move the local_flush_tlb_all() line after T0SZ setting. > > > > > > OK, so wouldn't this mean that we cannot change TxSZ at all without > > > going through a MMU off/on cycle? > > > > I don't think so. The way I see it is that the change is not guaranteed > > to take effect until we invalidate the TLBs. We don't risk fetching > > random stuff in the TLB since we have the reserved TTBR0 at that point. > > If the subsequent cpu_switch_mm() changed the ASID, in some > > interpretation of the ARM ARM we could have skipped the TLBI but that's > > not the case anyway. > > > > Looking for Mark R's opinion as I recall he talked to the architects in > > the past about this (though I think it was in the context of SCTLR_EL1). > > The architecture is unfortunately vague here. > > From prior discussions with architects, the general rule was "if it's permitted > to be cached in a TLB, an update must be followed by a TLBI for that update to > take effect, regardless of SCTLR_ELx.M". The architecture isn't very specific > about what scope fo maintenance is required (e.g. if certain fields are tagged > by ASID/VMID), but I beleive it's sufficient to use a (VM)ALL for the current > translation regime (which might be stronger than necessary). > > So for this case, my understanding is: > > 1) When changing TxSZ, we need a subsequent invalidate before the change is > guaranteed to take effect. So I believe that what we do today isn't quite > right. > > 2) During the window between writing to TxSZ and completing the invalidation, > I'm not sure how the MMU is permitted to behave w.r.t. intermediate walk > entries. I could imagine that those become (CONSTRAINED) UNPREDICTABLE , and > that we might need to ensure those are invalidated (or inactive and prior > walks completed) before we write to TCR_EL1. > OK, so that would at least mean that we can modify T1SZ while TTBR1 is pointing to reserved_pg_dir without running the risk of TLB conflicts, right? (and similarly for TTBR0). Given that we never modify TxSZ on a hot path, sprinkling some bonus TLBIs (and pivoting via reserved_pg_dir if needed) is hardly an issue here. > I can go chase this up with the architects; in the mean time my thinking would > be that we should retain the existing maintenance. > Yes, that would be helpful, thanks. > There's almost certainly more stuff that we'd need to fix in this area. > > Thanks, > Mark.
On Thu, Dec 01, 2022 at 12:22:43PM +0100, Ard Biesheuvel wrote: > On Thu, 1 Dec 2022 at 12:14, Mark Rutland <mark.rutland@arm.com> wrote: > > > > On Wed, Nov 30, 2022 at 04:40:11PM +0000, Catalin Marinas wrote: > > > On Wed, Nov 30, 2022 at 05:29:55PM +0100, Ard Biesheuvel wrote: > > > > On Wed, 30 Nov 2022 at 17:28, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > > On Wed, Nov 30, 2022 at 03:56:26PM +0100, Ard Biesheuvel wrote: > > > > > > On Wed, 30 Nov 2022 at 15:50, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > > > > > On Tue, Nov 15, 2022 at 03:38:23PM +0100, Ard Biesheuvel wrote: > > > > > > > > Currently, we detect CPU support for 52-bit virtual addressing (LVA) > > > > > > > > extremely early, before creating the kernel page tables or enabling the > > > > > > > > MMU. We cannot override the feature this early, and so large virtual > > > > > > > > addressing is always enabled on CPUs that implement support for it if > > > > > > > > the software support for it was enabled at build time. It also means we > > > > > > > > rely on non-trivial code in asm to deal with this feature. > > > > > > > > > > > > > > > > Given that both the ID map and the TTBR1 mapping of the kernel image are > > > > > > > > guaranteed to be 48-bit addressable, it is not actually necessary to > > > > > > > > enable support this early, and instead, we can model it as a CPU > > > > > > > > feature. That way, we can rely on code patching to get the correct > > > > > > > > TCR.T1SZ values programmed on secondary boot and suspend from resume. > > > > > > > > > > > > > > > > On the primary boot path, we simply enable the MMU with 48-bit virtual > > > > > > > > addressing initially, and update TCR.T1SZ from C code if LVA is > > > > > > > > supported, right before creating the kernel mapping. Given that TTBR1 > > > > > > > > still points to reserved_pg_dir at this point, updating TCR.T1SZ should > > > > > > > > be safe without the need for explicit TLB maintenance. > > > > > > > > > > > > > > I'm not sure we can skip the TLBI here. There's some weird rule in the > > > > > > > ARM ARM that if you change any of fields that may be cached in the TLB, > > > > > > > maintenance is needed even if the MMU is off. From the latest version > > > > > > > (I.a, I didn't dig into H.a), > > > > > > > > > > > > > > R_VNRFW: > > > > > > > When a System register field is modified and that field is permitted > > > > > > > to be cached in a TLB, software is required to invalidate all TLB > > > > > > > entries that might be affected by the field, at any address > > > > > > > translation stage in the translation regime even if the translation > > > > > > > stage is disabled, using the appropriate VMID and ASID, after any > > > > > > > required System register synchronization. > > > > > > > > > > > > Don't we already rely on this in cpu_set_default_tcr_t0sz() / > > > > > > cpu_set_idmap_tcr_t0sz() ? > > > > > > > > > > Yeah, we do this and depending on how you read the above rule, we may > > > > > need to move the local_flush_tlb_all() line after T0SZ setting. > > > > > > > > OK, so wouldn't this mean that we cannot change TxSZ at all without > > > > going through a MMU off/on cycle? > > > > > > I don't think so. The way I see it is that the change is not guaranteed > > > to take effect until we invalidate the TLBs. We don't risk fetching > > > random stuff in the TLB since we have the reserved TTBR0 at that point. > > > If the subsequent cpu_switch_mm() changed the ASID, in some > > > interpretation of the ARM ARM we could have skipped the TLBI but that's > > > not the case anyway. > > > > > > Looking for Mark R's opinion as I recall he talked to the architects in > > > the past about this (though I think it was in the context of SCTLR_EL1). > > > > The architecture is unfortunately vague here. > > > > From prior discussions with architects, the general rule was "if it's permitted > > to be cached in a TLB, an update must be followed by a TLBI for that update to > > take effect, regardless of SCTLR_ELx.M". The architecture isn't very specific > > about what scope fo maintenance is required (e.g. if certain fields are tagged > > by ASID/VMID), but I beleive it's sufficient to use a (VM)ALL for the current > > translation regime (which might be stronger than necessary). > > > > So for this case, my understanding is: > > > > 1) When changing TxSZ, we need a subsequent invalidate before the change is > > guaranteed to take effect. So I believe that what we do today isn't quite > > right. > > > > 2) During the window between writing to TxSZ and completing the invalidation, > > I'm not sure how the MMU is permitted to behave w.r.t. intermediate walk > > entries. I could imagine that those become (CONSTRAINED) UNPREDICTABLE , and > > that we might need to ensure those are invalidated (or inactive and prior > > walks completed) before we write to TCR_EL1. > > > > OK, so that would at least mean that we can modify T1SZ while TTBR1 is > pointing to reserved_pg_dir without running the risk of TLB conflicts, > right? (and similarly for TTBR0). That is my understanding, yes. > Given that we never modify TxSZ on a hot path, sprinkling some bonus TLBIs > (and pivoting via reserved_pg_dir if needed) is hardly an issue here. Agreed. > > I can go chase this up with the architects; in the mean time my thinking would > > be that we should retain the existing maintenance. > > Yes, that would be helpful, thanks. Cool; I\ll try to do that shortly, and will keep you updated. Thanks, Mark.
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index a4e1d832a15a2d7a..20e15c3f4589bd38 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -183,9 +183,11 @@ #include <asm/boot.h> #include <asm/bug.h> #include <asm/sections.h> +#include <asm/sysreg.h> #if VA_BITS > 48 -extern u64 vabits_actual; +// For reasons of #include hell, we can't use TCR_T1SZ_OFFSET/TCR_T1SZ_MASK here +#define vabits_actual (64 - ((read_sysreg(tcr_el1) >> 16) & 63)) #else #define vabits_actual ((u64)VA_BITS) #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index eca9df123a8b354b..b44aece5024c3e2d 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2654,6 +2654,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .cpu_enable = cpu_trap_el0_impdef, }, +#ifdef CONFIG_ARM64_VA_BITS_52 + { + .desc = "52-bit Virtual Addressing (LVA)", + .capability = ARM64_HAS_LVA, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_width = 4, + .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64MMFR2_EL1_VARange_52, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 3b3c5e8e84af890e..6abf513189c7ebc9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) @@ -95,14 +94,6 @@ SYM_CODE_START(primary_entry) * On return, the CPU will be ready for the MMU to be turned on and * the TCR will have been set. */ -#if VA_BITS > 48 - mrs_s x0, SYS_ID_AA64MMFR2_EL1 - tst x0, #0xf << ID_AA64MMFR2_EL1_VARange_SHIFT - mov x0, #VA_BITS - mov x25, #VA_BITS_MIN - csel x25, x25, x0, eq - mov x0, x25 -#endif bl __cpu_setup // initialise processor b __primary_switch SYM_CODE_END(primary_entry) @@ -406,11 +397,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag -#if VA_BITS > 48 - adr_l x8, vabits_actual // Set this early so KASAN early init - str x25, [x8] // ... observes the correct value - dc civac, x8 // Make visible to booting secondaries -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -525,9 +511,6 @@ SYM_FUNC_START_LOCAL(secondary_startup) mov x20, x0 // preserve boot mode bl finalise_el2 bl __cpu_secondary_check52bitva -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir adrp x2, idmap_pg_dir @@ -628,10 +611,9 @@ SYM_FUNC_END(__enable_mmu) SYM_FUNC_START(__cpu_secondary_check52bitva) #if VA_BITS > 48 - ldr_l x0, vabits_actual - cmp x0, #52 - b.ne 2f - +alternative_if_not ARM64_HAS_LVA + ret +alternative_else_nop_endif mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) cbnz x0, 2f diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 2bbf017147830bbe..3504e3266b02f636 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -122,6 +122,15 @@ static bool __init arm64_early_this_cpu_has_e0pd(void) ID_AA64MMFR2_EL1_E0PD_SHIFT); } +static bool __init arm64_early_this_cpu_has_lva(void) +{ + u64 mmfr2; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_VARange_SHIFT); +} + static bool __init arm64_early_this_cpu_has_pac(void) { u64 isar1, isar2; @@ -274,6 +283,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) /* Parse the command line for CPU feature overrides */ init_feature_override(boot_status, fdt, chosen); + if (VA_BITS > VA_BITS_MIN && arm64_early_this_cpu_has_lva()) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + if (IS_ENABLED(CONFIG_ARM64_WXN) && cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, ARM64_SW_FEATURE_OVERRIDE_NOWXN)) diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 97c9de57725dfddb..617f78ad43a185c2 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -101,9 +101,6 @@ SYM_FUNC_END(__cpu_suspend_enter) SYM_CODE_START(cpu_resume) bl init_kernel_el bl finalise_el2 -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a9714b00f5410d7d..63fb62e16a1f8873 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,11 +45,6 @@ int idmap_t0sz __ro_after_init; -#if VA_BITS > 48 -u64 vabits_actual __ro_after_init = VA_BITS_MIN; -EXPORT_SYMBOL(vabits_actual); -#endif - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 98531775ff529dc8..02818fa6aded3218 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -400,8 +400,6 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) * * Initialise the processor for turning the MMU on. * - * Input: - * x0 - actual number of VA bits (ignored unless VA_BITS > 48) * Output: * Return in x0 the value of the SCTLR_EL1 register. */ @@ -426,20 +424,20 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ + mov_q tcr, TCR_TxSZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 -#ifdef CONFIG_ARM64_VA_BITS_52 - sub x9, xzr, x0 - add x9, x9, #64 - tcr_set_t1sz tcr, x9 -#else +#if VA_BITS > VA_BITS_MIN +alternative_if ARM64_HAS_LVA + eor tcr, tcr, #TCR_T1SZ(VA_BITS) ^ TCR_T1SZ(VA_BITS_MIN) +alternative_else_nop_endif +#elif VA_BITS < 48 idmap_get_t0sz x9 -#endif tcr_set_t0sz tcr, x9 +#endif /* * Set the IPS bits in TCR_EL1. diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index f1c0347ec31a85c7..ec650a2cf4330179 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -30,6 +30,7 @@ HAS_GENERIC_AUTH_IMP_DEF HAS_IRQ_PRIO_MASKING HAS_LDAPR HAS_LSE_ATOMICS +HAS_LVA HAS_NO_FPSIMD HAS_NO_HW_PREFETCH HAS_PAN
Currently, we detect CPU support for 52-bit virtual addressing (LVA) extremely early, before creating the kernel page tables or enabling the MMU. We cannot override the feature this early, and so large virtual addressing is always enabled on CPUs that implement support for it if the software support for it was enabled at build time. It also means we rely on non-trivial code in asm to deal with this feature. Given that both the ID map and the TTBR1 mapping of the kernel image are guaranteed to be 48-bit addressable, it is not actually necessary to enable support this early, and instead, we can model it as a CPU feature. That way, we can rely on code patching to get the correct TCR.T1SZ values programmed on secondary boot and suspend from resume. On the primary boot path, we simply enable the MMU with 48-bit virtual addressing initially, and update TCR.T1SZ from C code if LVA is supported, right before creating the kernel mapping. Given that TTBR1 still points to reserved_pg_dir at this point, updating TCR.T1SZ should be safe without the need for explicit TLB maintenance. Since this gets rid of all accesses to the vabits_actual variable from asm code that occurred before TCR.T1SZ had been programmed, we no longer have a need for this variable, and we can replace it with a C expression that produces the correct value directly, based on the value of TCR.T1SZ. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm64/include/asm/memory.h | 4 +++- arch/arm64/kernel/cpufeature.c | 13 +++++++++++ arch/arm64/kernel/head.S | 24 +++----------------- arch/arm64/kernel/pi/map_kernel.c | 12 ++++++++++ arch/arm64/kernel/sleep.S | 3 --- arch/arm64/mm/mmu.c | 5 ---- arch/arm64/mm/proc.S | 16 ++++++------- arch/arm64/tools/cpucaps | 1 + 8 files changed, 39 insertions(+), 39 deletions(-)