| Message ID | 20250227144150.1667735-1-suzuki.poulose@arm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | arm64: realm: Fix DMA address for devices | expand |
Hi Marek, On 27/02/2025 14:41, Suzuki K Poulose wrote: > Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. The address > space (GPA or IPA) of a Realm VM is split into two halves, with private bottom > half and shared top half. In Linux we treat the "top" bit of the IPA space as > an attribute, to indicate whether it is shared or not (MSB == 1 implies shared). > Stage2 (GPA to PA) translations used by the CPU accesses, cover the full IPA space, > and are managed by RMM. The "top" bit as attribute is only a software construct. > > At present any device passed through to a Realm is treated as untrusted and the > Realm uses bounce buffering for any DMA, using the "decrypted" (shared) DMA > buffers (i.e., IPA with top bit set). In Linux, we only send the "DMA" address > masking the "top" bit. In Arm CCA, SMMU for untrusted devices are managed by the > non-secure Host and thus it can be confusing for the host/device when an unmasked > address is provided. Given there could be other hypervisors than Linux/KVM > running Arm CCA guests, the Realm Guest must adhere to a single convention for > the DMA address. This gets further complicated when we add support for trusted > devices, which can DMA into the full Realm memory space, once accepted. Thus, > a DMA masked address (with "top" bit lost) will prevent a trusted device from > accessing a shared buffer. > > To resolve this Arm has decided to standardise the DMA address used by the Realm > to include the full IPA address bits (including the "top" bit, which Linux uses > as an attribute). This implies, any DMA to a shared buffer must have the top bit > of the IPA space set. > > There is already a provision to do this in phys_to_dma* and dma_to_phys(), but > that is specific to AMD SME and is quite the opposite of what we need for Arm CCA. > i.e., For Arm CCA we need to set the bit for "decrypted" DMA and clear the bit > for "encrypted". > > This series converts the existing __sme_* helpers to a bit more generalised versions : > dma_addr_decrypted() and dma_encrypted(). Also, while converting a DMA address back > to CPU physical address requires clearing off any "encryption/decryption" bits. > I have named this "dma_addr_canonical()". (The other options are : > * dma_addr_clear_encryption - Well, not just for encryption, but we clear decryption > too, so not ideal. > * dma_addr_normal > * dma_addr_clear > * dma_addr_default > > This also implies that the VMMs must take care to : > > 1. Create the S2-SMMU mappings for VFIO at the "unprotected" alias. > 2. Always mask the "top" bit off any IPA it receives from the Realm for DMA. > KVM already does that today and no changes are required. > > A kvmtool branch with the changes above is available here [1]. There are two > patches [2] & [3], that are really required on top of the Arm CCA support. > > Ideally it would be good to get this backported to v6.13 stable kernel releases > to make sure that they are compliant with this change. > Please could you take a look at this series and let us know your thoughts ? If you are happy with the changes, are you happy to pull this through the DMA MAP tree ? The relevant bits have been reviewed/ acked by people (arm64 and AMD bits). Kind regards Suzuki > Changes since v2: > Link: https://lkml.kernel.org/r/20250219220751.1276854-1-suzuki.poulose@arm.com > - Collect Acks & Reviews for Patch 1 > - Rename helpers > dma_encrypted => dma_addr_encrypted > dma_decrypted => dma_addr_unencrypted > dma_clear_encryption => dma_addr_canonical > - For Arm CCA, use PROT_NS_SHARED, set/clear only the top IPA bit. > - Drop dma_addr_encrypted() helper for Arm CCA as it is a NOP > > Changes since v1 > Link: https://lkml.kernel.org/r/20250212171411.951874-1-suzuki.poulose@arm.com > - Follow Robin's suggestion to generalise the DMA address conversion helpers > to provide dma_{encrypte,decrypted,clear_encryption}. See PATCH 2 for more > details. > - Add a fix to the ordering of "__sme_clr" for dma_to_phys (PATCH 1) > > [1] git@git.gitlab.arm.com:linux-arm/kvmtool-cca.git cca/guest-dma-alias/v1 > [2] https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/ea37a6eb968abe4c75be4a8a90808714657c2ef7 > [3] https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/8afd0d5e6a7ee444dd0c1565fe94ecd831054a29 > > Cc: Will Deacon <will@kernel.org> > Cc: Jean-Philippe Brucker <jean-philippe@linaro.org> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Robin Murphy <robin.murphy@arm.com> > Cc: Steven Price <steven.price@arm.com> > Cc: Christoph Hellwig <hch@lst.de> > Cc: Marek Szyprowski <m.szyprowski@samsung.com> > Cc: Tom Lendacky <thomas.lendacky@amd.com> > Cc: Gavin Shan <gshan@redhat.com> > > Suzuki K Poulose (3): > dma: Fix encryption bit clearing for dma_to_phys > dma: Introduce generic dma_addr_*crypted helpers > arm64: realm: Use aliased addresses for device DMA to shared buffers > > arch/arm64/include/asm/mem_encrypt.h | 11 +++++++++++ > include/linux/dma-direct.h | 13 +++++++++---- > include/linux/mem_encrypt.h | 23 +++++++++++++++++++++++ > 3 files changed, 43 insertions(+), 4 deletions(-) >
Hi, On 03.03.2025 12:35, Suzuki K Poulose wrote: > On 27/02/2025 14:41, Suzuki K Poulose wrote: >> Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. >> The address >> space (GPA or IPA) of a Realm VM is split into two halves, with >> private bottom >> half and shared top half. In Linux we treat the "top" bit of the IPA >> space as >> an attribute, to indicate whether it is shared or not (MSB == 1 >> implies shared). >> Stage2 (GPA to PA) translations used by the CPU accesses, cover the >> full IPA space, >> and are managed by RMM. The "top" bit as attribute is only a software >> construct. >> >> At present any device passed through to a Realm is treated as >> untrusted and the >> Realm uses bounce buffering for any DMA, using the "decrypted" >> (shared) DMA >> buffers (i.e., IPA with top bit set). In Linux, we only send the >> "DMA" address >> masking the "top" bit. In Arm CCA, SMMU for untrusted devices are >> managed by the >> non-secure Host and thus it can be confusing for the host/device when >> an unmasked >> address is provided. Given there could be other hypervisors than >> Linux/KVM >> running Arm CCA guests, the Realm Guest must adhere to a single >> convention for >> the DMA address. This gets further complicated when we add support >> for trusted >> devices, which can DMA into the full Realm memory space, once >> accepted. Thus, >> a DMA masked address (with "top" bit lost) will prevent a trusted >> device from >> accessing a shared buffer. >> >> To resolve this Arm has decided to standardise the DMA address used >> by the Realm >> to include the full IPA address bits (including the "top" bit, which >> Linux uses >> as an attribute). This implies, any DMA to a shared buffer must have >> the top bit >> of the IPA space set. >> >> There is already a provision to do this in phys_to_dma* and >> dma_to_phys(), but >> that is specific to AMD SME and is quite the opposite of what we need >> for Arm CCA. >> i.e., For Arm CCA we need to set the bit for "decrypted" DMA and >> clear the bit >> for "encrypted". >> >> This series converts the existing __sme_* helpers to a bit more >> generalised versions : >> dma_addr_decrypted() and dma_encrypted(). Also, while converting a >> DMA address back >> to CPU physical address requires clearing off any >> "encryption/decryption" bits. >> I have named this "dma_addr_canonical()". (The other options are : >> * dma_addr_clear_encryption - Well, not just for encryption, but >> we clear decryption >> too, so not ideal. >> * dma_addr_normal >> * dma_addr_clear >> * dma_addr_default >> >> This also implies that the VMMs must take care to : >> >> 1. Create the S2-SMMU mappings for VFIO at the "unprotected" alias. >> 2. Always mask the "top" bit off any IPA it receives from the Realm >> for DMA. >> KVM already does that today and no changes are required. >> >> A kvmtool branch with the changes above is available here [1]. There >> are two >> patches [2] & [3], that are really required on top of the Arm CCA >> support. >> >> Ideally it would be good to get this backported to v6.13 stable >> kernel releases >> to make sure that they are compliant with this change. >> > > Please could you take a look at this series and let us know your > thoughts ? If you are happy with the changes, are you happy to pull > this through the DMA MAP tree ? The relevant bits have been reviewed/ > acked by people (arm64 and AMD bits). The changes look fine. However I won't be able to setup new dma-mapping git tree this week because I got really sick has to stay in bed. :/ If You don't want such delay, please merge it via ARM64 tree. Here is my: Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> Best regards
On 04/03/2025 13:40, Marek Szyprowski wrote: > Hi, > > On 03.03.2025 12:35, Suzuki K Poulose wrote: >> On 27/02/2025 14:41, Suzuki K Poulose wrote: >>> Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. >>> The address >>> space (GPA or IPA) of a Realm VM is split into two halves, with >>> private bottom >>> half and shared top half. In Linux we treat the "top" bit of the IPA >>> space as >>> an attribute, to indicate whether it is shared or not (MSB == 1 >>> implies shared). >>> Stage2 (GPA to PA) translations used by the CPU accesses, cover the >>> full IPA space, >>> and are managed by RMM. The "top" bit as attribute is only a software >>> construct. >>> >>> At present any device passed through to a Realm is treated as >>> untrusted and the >>> Realm uses bounce buffering for any DMA, using the "decrypted" >>> (shared) DMA >>> buffers (i.e., IPA with top bit set). In Linux, we only send the >>> "DMA" address >>> masking the "top" bit. In Arm CCA, SMMU for untrusted devices are >>> managed by the >>> non-secure Host and thus it can be confusing for the host/device when >>> an unmasked >>> address is provided. Given there could be other hypervisors than >>> Linux/KVM >>> running Arm CCA guests, the Realm Guest must adhere to a single >>> convention for >>> the DMA address. This gets further complicated when we add support >>> for trusted >>> devices, which can DMA into the full Realm memory space, once >>> accepted. Thus, >>> a DMA masked address (with "top" bit lost) will prevent a trusted >>> device from >>> accessing a shared buffer. >>> >>> To resolve this Arm has decided to standardise the DMA address used >>> by the Realm >>> to include the full IPA address bits (including the "top" bit, which >>> Linux uses >>> as an attribute). This implies, any DMA to a shared buffer must have >>> the top bit >>> of the IPA space set. >>> >>> There is already a provision to do this in phys_to_dma* and >>> dma_to_phys(), but >>> that is specific to AMD SME and is quite the opposite of what we need >>> for Arm CCA. >>> i.e., For Arm CCA we need to set the bit for "decrypted" DMA and >>> clear the bit >>> for "encrypted". >>> >>> This series converts the existing __sme_* helpers to a bit more >>> generalised versions : >>> dma_addr_decrypted() and dma_encrypted(). Also, while converting a >>> DMA address back >>> to CPU physical address requires clearing off any >>> "encryption/decryption" bits. >>> I have named this "dma_addr_canonical()". (The other options are : >>> * dma_addr_clear_encryption - Well, not just for encryption, but >>> we clear decryption >>> too, so not ideal. >>> * dma_addr_normal >>> * dma_addr_clear >>> * dma_addr_default >>> >>> This also implies that the VMMs must take care to : >>> >>> 1. Create the S2-SMMU mappings for VFIO at the "unprotected" alias. >>> 2. Always mask the "top" bit off any IPA it receives from the Realm >>> for DMA. >>> KVM already does that today and no changes are required. >>> >>> A kvmtool branch with the changes above is available here [1]. There >>> are two >>> patches [2] & [3], that are really required on top of the Arm CCA >>> support. >>> >>> Ideally it would be good to get this backported to v6.13 stable >>> kernel releases >>> to make sure that they are compliant with this change. >>> >> >> Please could you take a look at this series and let us know your >> thoughts ? If you are happy with the changes, are you happy to pull >> this through the DMA MAP tree ? The relevant bits have been reviewed/ >> acked by people (arm64 and AMD bits). > > The changes look fine. However I won't be able to setup new dma-mapping > git tree this week because I got really sick has to stay in bed. :/ If > You don't want such delay, please merge it via ARM64 tree. Here is my: Sorry to hear that. Hope you feel better soon. > > Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> Thanks Marek. Btw, this series fixes the "Realm Guest" support for Linux, which was merged in v6.13. To be precise, this should have : Fixes: 42be24a4178f ("arm64: Enable memory encrypt for Realms") Will/Catalin, Please let me know if you would like me to send the series with all the Acks, Reviews and mainly the Fixes tag added ? Kind regards Suzuki > > > Best regards
On Thu, Mar 06, 2025 at 11:39:14AM +0000, Suzuki K Poulose wrote: > On 04/03/2025 13:40, Marek Szyprowski wrote: > > > On 27/02/2025 14:41, Suzuki K Poulose wrote: > > > > Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. The address > > > > space (GPA or IPA) of a Realm VM is split into two halves, with private bottom > > > > half and shared top half. In Linux we treat the "top" bit of the IPA space as > > > > an attribute, to indicate whether it is shared or not (MSB == 1 implies shared). > > > > Stage2 (GPA to PA) translations used by the CPU accesses, cover the full IPA space, > > > > and are managed by RMM. The "top" bit as attribute is only a software construct. [...] > > The changes look fine. However I won't be able to setup new dma-mapping > > git tree this week because I got really sick has to stay in bed. :/ If > > You don't want such delay, please merge it via ARM64 tree. Here is my: > > Sorry to hear that. Hope you feel better soon. > > > Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> > > Thanks Marek. > > Btw, this series fixes the "Realm Guest" support for Linux, which was merged > in v6.13. To be precise, this should have : > > Fixes: 42be24a4178f ("arm64: Enable memory encrypt for Realms") > > Will/Catalin, > > Please let me know if you would like me to send the series with > all the Acks, Reviews and mainly the Fixes tag added ? We can add the acks and fixes tag, no worries. I can queue them for 6.15 unless Will takes them as fixes for 6.14 (I'll wait for a bit).
On Thu, Mar 06, 2025 at 06:06:44PM +0000, Catalin Marinas wrote: > On Thu, Mar 06, 2025 at 11:39:14AM +0000, Suzuki K Poulose wrote: > > On 04/03/2025 13:40, Marek Szyprowski wrote: > > > > On 27/02/2025 14:41, Suzuki K Poulose wrote: > > > > > Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. The address > > > > > space (GPA or IPA) of a Realm VM is split into two halves, with private bottom > > > > > half and shared top half. In Linux we treat the "top" bit of the IPA space as > > > > > an attribute, to indicate whether it is shared or not (MSB == 1 implies shared). > > > > > Stage2 (GPA to PA) translations used by the CPU accesses, cover the full IPA space, > > > > > and are managed by RMM. The "top" bit as attribute is only a software construct. > [...] > > > The changes look fine. However I won't be able to setup new dma-mapping > > > git tree this week because I got really sick has to stay in bed. :/ If > > > You don't want such delay, please merge it via ARM64 tree. Here is my: > > > > Sorry to hear that. Hope you feel better soon. > > > > > Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> > > > > Thanks Marek. > > > > Btw, this series fixes the "Realm Guest" support for Linux, which was merged > > in v6.13. To be precise, this should have : > > > > Fixes: 42be24a4178f ("arm64: Enable memory encrypt for Realms") > > > > Will/Catalin, > > > > Please let me know if you would like me to send the series with > > all the Acks, Reviews and mainly the Fixes tag added ? > > We can add the acks and fixes tag, no worries. I can queue them for 6.15 > unless Will takes them as fixes for 6.14 (I'll wait for a bit). For the series: Acked-by: Will Deacon <will@kernel.org> but I'd prefer this to land in 6.15 at this stage given that (a) it's always been broken and (b) we're not exactly tripping over CCA-capable hardware. Cheers, Will
On Thu, 27 Feb 2025 14:41:47 +0000, Suzuki K Poulose wrote: > Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. The address > space (GPA or IPA) of a Realm VM is split into two halves, with private bottom > half and shared top half. In Linux we treat the "top" bit of the IPA space as > an attribute, to indicate whether it is shared or not (MSB == 1 implies shared). > Stage2 (GPA to PA) translations used by the CPU accesses, cover the full IPA space, > and are managed by RMM. The "top" bit as attribute is only a software construct. > > [...] Applied to arm64 (for-next/cca-dma-address), thanks! [1/3] dma: Fix encryption bit clearing for dma_to_phys https://git.kernel.org/arm64/c/c380931712d1 [2/3] dma: Introduce generic dma_addr_*crypted helpers https://git.kernel.org/arm64/c/b66e2ee7b6c8 [3/3] arm64: realm: Use aliased addresses for device DMA to shared buffers https://git.kernel.org/arm64/c/7d953a062416
Linux can be run as a Confidential Guest in Arm CCA from Linux v6.13. The address space (GPA or IPA) of a Realm VM is split into two halves, with private bottom half and shared top half. In Linux we treat the "top" bit of the IPA space as an attribute, to indicate whether it is shared or not (MSB == 1 implies shared). Stage2 (GPA to PA) translations used by the CPU accesses, cover the full IPA space, and are managed by RMM. The "top" bit as attribute is only a software construct. At present any device passed through to a Realm is treated as untrusted and the Realm uses bounce buffering for any DMA, using the "decrypted" (shared) DMA buffers (i.e., IPA with top bit set). In Linux, we only send the "DMA" address masking the "top" bit. In Arm CCA, SMMU for untrusted devices are managed by the non-secure Host and thus it can be confusing for the host/device when an unmasked address is provided. Given there could be other hypervisors than Linux/KVM running Arm CCA guests, the Realm Guest must adhere to a single convention for the DMA address. This gets further complicated when we add support for trusted devices, which can DMA into the full Realm memory space, once accepted. Thus, a DMA masked address (with "top" bit lost) will prevent a trusted device from accessing a shared buffer. To resolve this Arm has decided to standardise the DMA address used by the Realm to include the full IPA address bits (including the "top" bit, which Linux uses as an attribute). This implies, any DMA to a shared buffer must have the top bit of the IPA space set. There is already a provision to do this in phys_to_dma* and dma_to_phys(), but that is specific to AMD SME and is quite the opposite of what we need for Arm CCA. i.e., For Arm CCA we need to set the bit for "decrypted" DMA and clear the bit for "encrypted". This series converts the existing __sme_* helpers to a bit more generalised versions : dma_addr_decrypted() and dma_encrypted(). Also, while converting a DMA address back to CPU physical address requires clearing off any "encryption/decryption" bits. I have named this "dma_addr_canonical()". (The other options are : * dma_addr_clear_encryption - Well, not just for encryption, but we clear decryption too, so not ideal. * dma_addr_normal * dma_addr_clear * dma_addr_default This also implies that the VMMs must take care to : 1. Create the S2-SMMU mappings for VFIO at the "unprotected" alias. 2. Always mask the "top" bit off any IPA it receives from the Realm for DMA. KVM already does that today and no changes are required. A kvmtool branch with the changes above is available here [1]. There are two patches [2] & [3], that are really required on top of the Arm CCA support. Ideally it would be good to get this backported to v6.13 stable kernel releases to make sure that they are compliant with this change. Changes since v2: Link: https://lkml.kernel.org/r/20250219220751.1276854-1-suzuki.poulose@arm.com - Collect Acks & Reviews for Patch 1 - Rename helpers dma_encrypted => dma_addr_encrypted dma_decrypted => dma_addr_unencrypted dma_clear_encryption => dma_addr_canonical - For Arm CCA, use PROT_NS_SHARED, set/clear only the top IPA bit. - Drop dma_addr_encrypted() helper for Arm CCA as it is a NOP Changes since v1 Link: https://lkml.kernel.org/r/20250212171411.951874-1-suzuki.poulose@arm.com - Follow Robin's suggestion to generalise the DMA address conversion helpers to provide dma_{encrypte,decrypted,clear_encryption}. See PATCH 2 for more details. - Add a fix to the ordering of "__sme_clr" for dma_to_phys (PATCH 1) [1] git@git.gitlab.arm.com:linux-arm/kvmtool-cca.git cca/guest-dma-alias/v1 [2] https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/ea37a6eb968abe4c75be4a8a90808714657c2ef7 [3] https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/8afd0d5e6a7ee444dd0c1565fe94ecd831054a29 Cc: Will Deacon <will@kernel.org> Cc: Jean-Philippe Brucker <jean-philippe@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Robin Murphy <robin.murphy@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Cc: Gavin Shan <gshan@redhat.com> Suzuki K Poulose (3): dma: Fix encryption bit clearing for dma_to_phys dma: Introduce generic dma_addr_*crypted helpers arm64: realm: Use aliased addresses for device DMA to shared buffers arch/arm64/include/asm/mem_encrypt.h | 11 +++++++++++ include/linux/dma-direct.h | 13 +++++++++---- include/linux/mem_encrypt.h | 23 +++++++++++++++++++++++ 3 files changed, 43 insertions(+), 4 deletions(-)