| Message ID | 78df4f8e2510e88f3ded59eb385f79b4442ed4f2.1566238530.git.robin.murphy@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | iommu/io-pgtable: Cleanup and prep for split tables | expand |
On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > Although it's conceptually nice for the io_pgtable_cfg to provide a > standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > looks exactly like an Arm CPU, and they all have various other TCR > controls which io-pgtable can't be expected to understand. Thus since > there is an expectation that drivers will have to add to the given TCR > value anyway, let's strip it down to just the essentials that are > directly relevant to io-pgatble's inner workings - namely the address > sizes, walk attributes, and where appropriate, format selection. > > Signed-off-by: Robin Murphy <robin.murphy@arm.com> > --- > drivers/iommu/arm-smmu-v3.c | 7 +------ > drivers/iommu/arm-smmu.c | 1 + > drivers/iommu/arm-smmu.h | 2 ++ > drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > drivers/iommu/io-pgtable-arm.c | 4 ---- > drivers/iommu/qcom_iommu.c | 2 +- > 6 files changed, 7 insertions(+), 15 deletions(-) Hmm, so I'm a bit nervous about this one since I think we really should be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying on the driver to do this worries me. See my comments on the next patch. Will
On 20/08/2019 11:31, Will Deacon wrote: > On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: >> Although it's conceptually nice for the io_pgtable_cfg to provide a >> standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU >> looks exactly like an Arm CPU, and they all have various other TCR >> controls which io-pgtable can't be expected to understand. Thus since >> there is an expectation that drivers will have to add to the given TCR >> value anyway, let's strip it down to just the essentials that are >> directly relevant to io-pgatble's inner workings - namely the address >> sizes, walk attributes, and where appropriate, format selection. >> >> Signed-off-by: Robin Murphy <robin.murphy@arm.com> >> --- >> drivers/iommu/arm-smmu-v3.c | 7 +------ >> drivers/iommu/arm-smmu.c | 1 + >> drivers/iommu/arm-smmu.h | 2 ++ >> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- >> drivers/iommu/io-pgtable-arm.c | 4 ---- >> drivers/iommu/qcom_iommu.c | 2 +- >> 6 files changed, 7 insertions(+), 15 deletions(-) > > Hmm, so I'm a bit nervous about this one since I think we really should > be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > on the driver to do this worries me. See my comments on the next patch. The whole idea is that we already know we can't provide a *complete* TCR value (not least because anything above bit 31 is the wild west), thus there's really no point in io-pgtable trying to provide anything other than the parts it definitely controls. It makes sense to provide this partial TCR value "as if" for TTBR0, since that's the most common case, but ultimately io-pgatble doesn't know (or need to) which TTBR the caller intends to actually use for this table. Even if the caller *is* allocating it for TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 already, so it still wouldn't be in a position to make the EPD1 call either way. Ultimately, it's the IOMMU drivers who decide what they put in which TTBR, so it's the IOMMU drivers which have to take responsibility for EPD*. Sure you can worry about it, but you can equally worry about them them misprogramming the ASID or anything else... Robin.
On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > On 20/08/2019 11:31, Will Deacon wrote: > > On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > > > Although it's conceptually nice for the io_pgtable_cfg to provide a > > > standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > > > looks exactly like an Arm CPU, and they all have various other TCR > > > controls which io-pgtable can't be expected to understand. Thus since > > > there is an expectation that drivers will have to add to the given TCR > > > value anyway, let's strip it down to just the essentials that are > > > directly relevant to io-pgatble's inner workings - namely the address > > > sizes, walk attributes, and where appropriate, format selection. > > > > > > Signed-off-by: Robin Murphy <robin.murphy@arm.com> > > > --- > > > drivers/iommu/arm-smmu-v3.c | 7 +------ > > > drivers/iommu/arm-smmu.c | 1 + > > > drivers/iommu/arm-smmu.h | 2 ++ > > > drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > > > drivers/iommu/io-pgtable-arm.c | 4 ---- > > > drivers/iommu/qcom_iommu.c | 2 +- > > > 6 files changed, 7 insertions(+), 15 deletions(-) > > > > Hmm, so I'm a bit nervous about this one since I think we really should > > be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > > on the driver to do this worries me. See my comments on the next patch. > > The whole idea is that we already know we can't provide a *complete* TCR > value (not least because anything above bit 31 is the wild west), thus > there's really no point in io-pgtable trying to provide anything other than > the parts it definitely controls. It makes sense to provide this partial TCR > value "as if" for TTBR0, since that's the most common case, but ultimately > io-pgatble doesn't know (or need to) which TTBR the caller intends to > actually use for this table. Even if the caller *is* allocating it for > TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > already, so it still wouldn't be in a position to make the EPD1 call either > way. Ok, but the driver can happily rewrite/ignore what it gets back. I suppose an alternative would be scrapped the 'u64 tcr' and instead having a bunch of named bitfields for the stuff we're actually providing, although I'd still like EPDx to be in there. > Ultimately, it's the IOMMU drivers who decide what they put in which TTBR, > so it's the IOMMU drivers which have to take responsibility for EPD*. Sure > you can worry about it, but you can equally worry about them them > misprogramming the ASID or anything else... I find the EPDx bits particularly dangerous because: - They're easily overlooked - Clobbering TTBR1 with 0x0 doesn't disable walks via TTBR1 as you might reasonably expect - If you do the above without EPD, the breakage will be subtle and given that I don't see any real downsides to us providing a default TCR value with EPD set appropriately, then I think we should do that. I'd be happy to revisit the decision later on if it's getting the way of a real use-case, but it feels like we're throwing the baby out with the bathwater at the moment and I'd rather do this incrementally based on actual need. Will
On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > On 20/08/2019 11:31, Will Deacon wrote: > >On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > >>Although it's conceptually nice for the io_pgtable_cfg to provide a > >>standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > >>looks exactly like an Arm CPU, and they all have various other TCR > >>controls which io-pgtable can't be expected to understand. Thus since > >>there is an expectation that drivers will have to add to the given TCR > >>value anyway, let's strip it down to just the essentials that are > >>directly relevant to io-pgatble's inner workings - namely the address > >>sizes, walk attributes, and where appropriate, format selection. > >> > >>Signed-off-by: Robin Murphy <robin.murphy@arm.com> > >>--- > >> drivers/iommu/arm-smmu-v3.c | 7 +------ > >> drivers/iommu/arm-smmu.c | 1 + > >> drivers/iommu/arm-smmu.h | 2 ++ > >> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > >> drivers/iommu/io-pgtable-arm.c | 4 ---- > >> drivers/iommu/qcom_iommu.c | 2 +- > >> 6 files changed, 7 insertions(+), 15 deletions(-) > > > >Hmm, so I'm a bit nervous about this one since I think we really should > >be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > >on the driver to do this worries me. See my comments on the next patch. > > The whole idea is that we already know we can't provide a *complete* TCR > value (not least because anything above bit 31 is the wild west), thus > there's really no point in io-pgtable trying to provide anything other than > the parts it definitely controls. It makes sense to provide this partial TCR > value "as if" for TTBR0, since that's the most common case, but ultimately > io-pgatble doesn't know (or need to) which TTBR the caller intends to > actually use for this table. Even if the caller *is* allocating it for > TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > already, so it still wouldn't be in a position to make the EPD1 call either > way. > > Ultimately, it's the IOMMU drivers who decide what they put in which TTBR, > so it's the IOMMU drivers which have to take responsibility for EPD*. Sure > you can worry about it, but you can equally worry about them them > misprogramming the ASID or anything else... I agree. If the driver *does* want to use TTBR1 then it gets a bit uglier to realize that the io-pgtable set the bit and mask it off. To me it is a lot clearer if this is done explicitly in the driver especially if the driver is where we are making the choice to use either ttbr0 or ttbr1. It would probably be easier to see in actual code, and I'll have a patch shortly on top of this series. Jordan
On 20/08/2019 17:07, Will Deacon wrote: > On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: >> On 20/08/2019 11:31, Will Deacon wrote: >>> On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: >>>> Although it's conceptually nice for the io_pgtable_cfg to provide a >>>> standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU >>>> looks exactly like an Arm CPU, and they all have various other TCR >>>> controls which io-pgtable can't be expected to understand. Thus since >>>> there is an expectation that drivers will have to add to the given TCR >>>> value anyway, let's strip it down to just the essentials that are >>>> directly relevant to io-pgatble's inner workings - namely the address >>>> sizes, walk attributes, and where appropriate, format selection. >>>> >>>> Signed-off-by: Robin Murphy <robin.murphy@arm.com> >>>> --- >>>> drivers/iommu/arm-smmu-v3.c | 7 +------ >>>> drivers/iommu/arm-smmu.c | 1 + >>>> drivers/iommu/arm-smmu.h | 2 ++ >>>> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- >>>> drivers/iommu/io-pgtable-arm.c | 4 ---- >>>> drivers/iommu/qcom_iommu.c | 2 +- >>>> 6 files changed, 7 insertions(+), 15 deletions(-) >>> >>> Hmm, so I'm a bit nervous about this one since I think we really should >>> be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying >>> on the driver to do this worries me. See my comments on the next patch. >> >> The whole idea is that we already know we can't provide a *complete* TCR >> value (not least because anything above bit 31 is the wild west), thus >> there's really no point in io-pgtable trying to provide anything other than >> the parts it definitely controls. It makes sense to provide this partial TCR >> value "as if" for TTBR0, since that's the most common case, but ultimately >> io-pgatble doesn't know (or need to) which TTBR the caller intends to >> actually use for this table. Even if the caller *is* allocating it for >> TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 >> already, so it still wouldn't be in a position to make the EPD1 call either >> way. > > Ok, but the driver can happily rewrite/ignore what it gets back. I suppose > an alternative would be scrapped the 'u64 tcr' and instead having a bunch > of named bitfields for the stuff we're actually providing, although I'd > still like EPDx to be in there. I like the bitfield idea; it would certainly emphasise the "you have to do something more with this" angle that I'm pushing towards here, but still leave things framed in TCR terms without having to go to some more general abstraction. It really doesn't play into your EPD argument though - such a config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. For a driver to understand that and do the right thing with it is even more involved than for the driver to just set EPD1 by itself anyway. >> Ultimately, it's the IOMMU drivers who decide what they put in which TTBR, >> so it's the IOMMU drivers which have to take responsibility for EPD*. Sure >> you can worry about it, but you can equally worry about them them >> misprogramming the ASID or anything else... > > I find the EPDx bits particularly dangerous because: > > - They're easily overlooked > - Clobbering TTBR1 with 0x0 doesn't disable walks via TTBR1 as you might > reasonably expect (FWIW I'm not sure that that is a reasonable expectation, at least for anyone savvy enough to be programming an MMU in the first place. There are plenty of systems with RAM at 0x0) > - If you do the above without EPD, the breakage will be subtle > > and given that I don't see any real downsides to us providing a default TCR > value with EPD set appropriately, then I think we should do that. I'd be > happy to revisit the decision later on if it's getting the way of a real > use-case, but it feels like we're throwing the baby out with the bathwater > at the moment and I'd rather do this incrementally based on actual need. The downside is maintaining extra complexity for the sake of a theoretical concern which hasn't been borne out in practice, with a promise of yet more complexity down the line. Moving the 3 babies which anyone acknowledges into their own private baths takes a whole -8 lines of code to implement. That said, if we don't go down the bitfield route, then I *can* leave the default TCR having EPD1 set if that makes you feel warm and safe, but it will still be resoundingly ignored. If only LPAE had created these bits as enables rather than disables then things would be logical and we could all be happy, but here we are... Robin.
On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: > On 20/08/2019 17:07, Will Deacon wrote: > > On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > > > On 20/08/2019 11:31, Will Deacon wrote: > > > > On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > > > > > Although it's conceptually nice for the io_pgtable_cfg to provide a > > > > > standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > > > > > looks exactly like an Arm CPU, and they all have various other TCR > > > > > controls which io-pgtable can't be expected to understand. Thus since > > > > > there is an expectation that drivers will have to add to the given TCR > > > > > value anyway, let's strip it down to just the essentials that are > > > > > directly relevant to io-pgatble's inner workings - namely the address > > > > > sizes, walk attributes, and where appropriate, format selection. > > > > > > > > > > Signed-off-by: Robin Murphy <robin.murphy@arm.com> > > > > > --- > > > > > drivers/iommu/arm-smmu-v3.c | 7 +------ > > > > > drivers/iommu/arm-smmu.c | 1 + > > > > > drivers/iommu/arm-smmu.h | 2 ++ > > > > > drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > > > > > drivers/iommu/io-pgtable-arm.c | 4 ---- > > > > > drivers/iommu/qcom_iommu.c | 2 +- > > > > > 6 files changed, 7 insertions(+), 15 deletions(-) > > > > > > > > Hmm, so I'm a bit nervous about this one since I think we really should > > > > be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > > > > on the driver to do this worries me. See my comments on the next patch. > > > > > > The whole idea is that we already know we can't provide a *complete* TCR > > > value (not least because anything above bit 31 is the wild west), thus > > > there's really no point in io-pgtable trying to provide anything other than > > > the parts it definitely controls. It makes sense to provide this partial TCR > > > value "as if" for TTBR0, since that's the most common case, but ultimately > > > io-pgatble doesn't know (or need to) which TTBR the caller intends to > > > actually use for this table. Even if the caller *is* allocating it for > > > TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > > > already, so it still wouldn't be in a position to make the EPD1 call either > > > way. > > > > Ok, but the driver can happily rewrite/ignore what it gets back. I suppose > > an alternative would be scrapped the 'u64 tcr' and instead having a bunch > > of named bitfields for the stuff we're actually providing, although I'd > > still like EPDx to be in there. > > I like the bitfield idea; it would certainly emphasise the "you have to do > something more with this" angle that I'm pushing towards here, but still > leave things framed in TCR terms without having to go to some more general > abstraction. It really doesn't play into your EPD argument though - such a > config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. > For a driver to understand that and do the right thing with it is even more > involved than for the driver to just set EPD1 by itself anyway. Having considered the bitfield idea some more, I'm less attached to EPDx because we simply wouldn't be making a statement about them, rather than a (dangerous) zero value and expecting it to be ignored. So I think we're in agreement on that. The only part I'm still stuck to is that I think io-pgtable should know whether it's targetting TTBR0 or TTBR1 so that it can sanitise input addresses correctly. Doing this in the driver code is possible, but I'd rather not start from that position, particularly as it would require things like sign-extension in the TLBI callbacks. > If only LPAE had created these bits as enables rather than disables then > things would be logical and we could all be happy, but here we are... I'm happy! :D:D:D Will
On 21/08/2019 13:11, Will Deacon wrote: > On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: >> On 20/08/2019 17:07, Will Deacon wrote: >>> On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: >>>> On 20/08/2019 11:31, Will Deacon wrote: >>>>> On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: >>>>>> Although it's conceptually nice for the io_pgtable_cfg to provide a >>>>>> standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU >>>>>> looks exactly like an Arm CPU, and they all have various other TCR >>>>>> controls which io-pgtable can't be expected to understand. Thus since >>>>>> there is an expectation that drivers will have to add to the given TCR >>>>>> value anyway, let's strip it down to just the essentials that are >>>>>> directly relevant to io-pgatble's inner workings - namely the address >>>>>> sizes, walk attributes, and where appropriate, format selection. >>>>>> >>>>>> Signed-off-by: Robin Murphy <robin.murphy@arm.com> >>>>>> --- >>>>>> drivers/iommu/arm-smmu-v3.c | 7 +------ >>>>>> drivers/iommu/arm-smmu.c | 1 + >>>>>> drivers/iommu/arm-smmu.h | 2 ++ >>>>>> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- >>>>>> drivers/iommu/io-pgtable-arm.c | 4 ---- >>>>>> drivers/iommu/qcom_iommu.c | 2 +- >>>>>> 6 files changed, 7 insertions(+), 15 deletions(-) >>>>> >>>>> Hmm, so I'm a bit nervous about this one since I think we really should >>>>> be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying >>>>> on the driver to do this worries me. See my comments on the next patch. >>>> >>>> The whole idea is that we already know we can't provide a *complete* TCR >>>> value (not least because anything above bit 31 is the wild west), thus >>>> there's really no point in io-pgtable trying to provide anything other than >>>> the parts it definitely controls. It makes sense to provide this partial TCR >>>> value "as if" for TTBR0, since that's the most common case, but ultimately >>>> io-pgatble doesn't know (or need to) which TTBR the caller intends to >>>> actually use for this table. Even if the caller *is* allocating it for >>>> TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 >>>> already, so it still wouldn't be in a position to make the EPD1 call either >>>> way. >>> >>> Ok, but the driver can happily rewrite/ignore what it gets back. I suppose >>> an alternative would be scrapped the 'u64 tcr' and instead having a bunch >>> of named bitfields for the stuff we're actually providing, although I'd >>> still like EPDx to be in there. >> >> I like the bitfield idea; it would certainly emphasise the "you have to do >> something more with this" angle that I'm pushing towards here, but still >> leave things framed in TCR terms without having to go to some more general >> abstraction. It really doesn't play into your EPD argument though - such a >> config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. >> For a driver to understand that and do the right thing with it is even more >> involved than for the driver to just set EPD1 by itself anyway. > > Having considered the bitfield idea some more, I'm less attached to EPDx > because we simply wouldn't be making a statement about them, rather than a > (dangerous) zero value and expecting it to be ignored. So I think we're in > agreement on that. Cool, I'll give bitfields a go for v2. > The only part I'm still stuck to is that I think io-pgtable should know > whether it's targetting TTBR0 or TTBR1 so that it can sanitise input > addresses correctly. Doing this in the driver code is possible, but I'd > rather not start from that position, particularly as it would require things > like sign-extension in the TLBI callbacks. Good point, and thanks for the prod that the way I end up propagating masked-off IOVAs through to the TLBI calls here is busted either way. I'm OK with introducing an explicit TTBR1 quirk to begin with, as it should make things a little easier to reason about, and we can always revisit later if and when we do find a need for more flexibility. >> If only LPAE had created these bits as enables rather than disables then >> things would be logical and we could all be happy, but here we are... > > I'm happy! :D:D:D Yeah, but I refuse to believe it's ever because of pagetables ;) Robin.
On Wed, Aug 21, 2019 at 01:56:20PM +0100, Robin Murphy wrote: > On 21/08/2019 13:11, Will Deacon wrote: > >On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: > >>On 20/08/2019 17:07, Will Deacon wrote: > >>>On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > >>>>On 20/08/2019 11:31, Will Deacon wrote: > >>>>>On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > >>>>>>Although it's conceptually nice for the io_pgtable_cfg to provide a > >>>>>>standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > >>>>>>looks exactly like an Arm CPU, and they all have various other TCR > >>>>>>controls which io-pgtable can't be expected to understand. Thus since > >>>>>>there is an expectation that drivers will have to add to the given TCR > >>>>>>value anyway, let's strip it down to just the essentials that are > >>>>>>directly relevant to io-pgatble's inner workings - namely the address > >>>>>>sizes, walk attributes, and where appropriate, format selection. > >>>>>> > >>>>>>Signed-off-by: Robin Murphy <robin.murphy@arm.com> > >>>>>>--- > >>>>>> drivers/iommu/arm-smmu-v3.c | 7 +------ > >>>>>> drivers/iommu/arm-smmu.c | 1 + > >>>>>> drivers/iommu/arm-smmu.h | 2 ++ > >>>>>> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > >>>>>> drivers/iommu/io-pgtable-arm.c | 4 ---- > >>>>>> drivers/iommu/qcom_iommu.c | 2 +- > >>>>>> 6 files changed, 7 insertions(+), 15 deletions(-) > >>>>> > >>>>>Hmm, so I'm a bit nervous about this one since I think we really should > >>>>>be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > >>>>>on the driver to do this worries me. See my comments on the next patch. > >>>> > >>>>The whole idea is that we already know we can't provide a *complete* TCR > >>>>value (not least because anything above bit 31 is the wild west), thus > >>>>there's really no point in io-pgtable trying to provide anything other than > >>>>the parts it definitely controls. It makes sense to provide this partial TCR > >>>>value "as if" for TTBR0, since that's the most common case, but ultimately > >>>>io-pgatble doesn't know (or need to) which TTBR the caller intends to > >>>>actually use for this table. Even if the caller *is* allocating it for > >>>>TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > >>>>already, so it still wouldn't be in a position to make the EPD1 call either > >>>>way. > >>> > >>>Ok, but the driver can happily rewrite/ignore what it gets back. I suppose > >>>an alternative would be scrapped the 'u64 tcr' and instead having a bunch > >>>of named bitfields for the stuff we're actually providing, although I'd > >>>still like EPDx to be in there. > >> > >>I like the bitfield idea; it would certainly emphasise the "you have to do > >>something more with this" angle that I'm pushing towards here, but still > >>leave things framed in TCR terms without having to go to some more general > >>abstraction. It really doesn't play into your EPD argument though - such a > >>config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. > >>For a driver to understand that and do the right thing with it is even more > >>involved than for the driver to just set EPD1 by itself anyway. > > > >Having considered the bitfield idea some more, I'm less attached to EPDx > >because we simply wouldn't be making a statement about them, rather than a > >(dangerous) zero value and expecting it to be ignored. So I think we're in > >agreement on that. > > Cool, I'll give bitfields a go for v2. > > >The only part I'm still stuck to is that I think io-pgtable should know > >whether it's targetting TTBR0 or TTBR1 so that it can sanitise input > >addresses correctly. Doing this in the driver code is possible, but I'd > >rather not start from that position, particularly as it would require things > >like sign-extension in the TLBI callbacks. Bumping this as is our tradition in the -rc1 time frame before we get all distracted with other stuff. It sounds like the last agreement was for a TTBR1 hint for the EDP and the sign extension in the functions. Let me know if you need any help. I've got a little time and more than a little motivation to keep slogging ahead toward a glorious arm-smmu-v2 per-context pagetable future. Jordan
On Thu, Oct 03, 2019 at 11:33:52AM -0600, Jordan Crouse wrote: > On Wed, Aug 21, 2019 at 01:56:20PM +0100, Robin Murphy wrote: > > On 21/08/2019 13:11, Will Deacon wrote: > > >On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: > > >>On 20/08/2019 17:07, Will Deacon wrote: > > >>>On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > > >>>>On 20/08/2019 11:31, Will Deacon wrote: > > >>>>>On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > > >>>>>>Although it's conceptually nice for the io_pgtable_cfg to provide a > > >>>>>>standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > > >>>>>>looks exactly like an Arm CPU, and they all have various other TCR > > >>>>>>controls which io-pgtable can't be expected to understand. Thus since > > >>>>>>there is an expectation that drivers will have to add to the given TCR > > >>>>>>value anyway, let's strip it down to just the essentials that are > > >>>>>>directly relevant to io-pgatble's inner workings - namely the address > > >>>>>>sizes, walk attributes, and where appropriate, format selection. > > >>>>>> > > >>>>>>Signed-off-by: Robin Murphy <robin.murphy@arm.com> > > >>>>>>--- > > >>>>>> drivers/iommu/arm-smmu-v3.c | 7 +------ > > >>>>>> drivers/iommu/arm-smmu.c | 1 + > > >>>>>> drivers/iommu/arm-smmu.h | 2 ++ > > >>>>>> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > > >>>>>> drivers/iommu/io-pgtable-arm.c | 4 ---- > > >>>>>> drivers/iommu/qcom_iommu.c | 2 +- > > >>>>>> 6 files changed, 7 insertions(+), 15 deletions(-) > > >>>>> > > >>>>>Hmm, so I'm a bit nervous about this one since I think we really should > > >>>>>be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > > >>>>>on the driver to do this worries me. See my comments on the next patch. > > >>>> > > >>>>The whole idea is that we already know we can't provide a *complete* TCR > > >>>>value (not least because anything above bit 31 is the wild west), thus > > >>>>there's really no point in io-pgtable trying to provide anything other than > > >>>>the parts it definitely controls. It makes sense to provide this partial TCR > > >>>>value "as if" for TTBR0, since that's the most common case, but ultimately > > >>>>io-pgatble doesn't know (or need to) which TTBR the caller intends to > > >>>>actually use for this table. Even if the caller *is* allocating it for > > >>>>TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > > >>>>already, so it still wouldn't be in a position to make the EPD1 call either > > >>>>way. > > >>> > > >>>Ok, but the driver can happily rewrite/ignore what it gets back. I suppose > > >>>an alternative would be scrapped the 'u64 tcr' and instead having a bunch > > >>>of named bitfields for the stuff we're actually providing, although I'd > > >>>still like EPDx to be in there. > > >> > > >>I like the bitfield idea; it would certainly emphasise the "you have to do > > >>something more with this" angle that I'm pushing towards here, but still > > >>leave things framed in TCR terms without having to go to some more general > > >>abstraction. It really doesn't play into your EPD argument though - such a > > >>config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. > > >>For a driver to understand that and do the right thing with it is even more > > >>involved than for the driver to just set EPD1 by itself anyway. > > > > > >Having considered the bitfield idea some more, I'm less attached to EPDx > > >because we simply wouldn't be making a statement about them, rather than a > > >(dangerous) zero value and expecting it to be ignored. So I think we're in > > >agreement on that. > > > > Cool, I'll give bitfields a go for v2. > > > > >The only part I'm still stuck to is that I think io-pgtable should know > > >whether it's targetting TTBR0 or TTBR1 so that it can sanitise input > > >addresses correctly. Doing this in the driver code is possible, but I'd > > >rather not start from that position, particularly as it would require things > > >like sign-extension in the TLBI callbacks. > > Bumping this as is our tradition in the -rc1 time frame before we get all > distracted with other stuff. It sounds like the last agreement was for a > TTBR1 hint for the EDP and the sign extension in the functions. If somebody respins this using bitfields and an explicit TTBR1 quirk then I'll merge it. Will
On 2019-10-24 11:51 am, Will Deacon wrote: > On Thu, Oct 03, 2019 at 11:33:52AM -0600, Jordan Crouse wrote: >> On Wed, Aug 21, 2019 at 01:56:20PM +0100, Robin Murphy wrote: >>> On 21/08/2019 13:11, Will Deacon wrote: >>>> On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: >>>>> On 20/08/2019 17:07, Will Deacon wrote: >>>>>> On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: >>>>>>> On 20/08/2019 11:31, Will Deacon wrote: >>>>>>>> On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: >>>>>>>>> Although it's conceptually nice for the io_pgtable_cfg to provide a >>>>>>>>> standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU >>>>>>>>> looks exactly like an Arm CPU, and they all have various other TCR >>>>>>>>> controls which io-pgtable can't be expected to understand. Thus since >>>>>>>>> there is an expectation that drivers will have to add to the given TCR >>>>>>>>> value anyway, let's strip it down to just the essentials that are >>>>>>>>> directly relevant to io-pgatble's inner workings - namely the address >>>>>>>>> sizes, walk attributes, and where appropriate, format selection. >>>>>>>>> >>>>>>>>> Signed-off-by: Robin Murphy <robin.murphy@arm.com> >>>>>>>>> --- >>>>>>>>> drivers/iommu/arm-smmu-v3.c | 7 +------ >>>>>>>>> drivers/iommu/arm-smmu.c | 1 + >>>>>>>>> drivers/iommu/arm-smmu.h | 2 ++ >>>>>>>>> drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- >>>>>>>>> drivers/iommu/io-pgtable-arm.c | 4 ---- >>>>>>>>> drivers/iommu/qcom_iommu.c | 2 +- >>>>>>>>> 6 files changed, 7 insertions(+), 15 deletions(-) >>>>>>>> >>>>>>>> Hmm, so I'm a bit nervous about this one since I think we really should >>>>>>>> be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying >>>>>>>> on the driver to do this worries me. See my comments on the next patch. >>>>>>> >>>>>>> The whole idea is that we already know we can't provide a *complete* TCR >>>>>>> value (not least because anything above bit 31 is the wild west), thus >>>>>>> there's really no point in io-pgtable trying to provide anything other than >>>>>>> the parts it definitely controls. It makes sense to provide this partial TCR >>>>>>> value "as if" for TTBR0, since that's the most common case, but ultimately >>>>>>> io-pgatble doesn't know (or need to) which TTBR the caller intends to >>>>>>> actually use for this table. Even if the caller *is* allocating it for >>>>>>> TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 >>>>>>> already, so it still wouldn't be in a position to make the EPD1 call either >>>>>>> way. >>>>>> >>>>>> Ok, but the driver can happily rewrite/ignore what it gets back. I suppose >>>>>> an alternative would be scrapped the 'u64 tcr' and instead having a bunch >>>>>> of named bitfields for the stuff we're actually providing, although I'd >>>>>> still like EPDx to be in there. >>>>> >>>>> I like the bitfield idea; it would certainly emphasise the "you have to do >>>>> something more with this" angle that I'm pushing towards here, but still >>>>> leave things framed in TCR terms without having to go to some more general >>>>> abstraction. It really doesn't play into your EPD argument though - such a >>>>> config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. >>>>> For a driver to understand that and do the right thing with it is even more >>>>> involved than for the driver to just set EPD1 by itself anyway. >>>> >>>> Having considered the bitfield idea some more, I'm less attached to EPDx >>>> because we simply wouldn't be making a statement about them, rather than a >>>> (dangerous) zero value and expecting it to be ignored. So I think we're in >>>> agreement on that. >>> >>> Cool, I'll give bitfields a go for v2. >>> >>>> The only part I'm still stuck to is that I think io-pgtable should know >>>> whether it's targetting TTBR0 or TTBR1 so that it can sanitise input >>>> addresses correctly. Doing this in the driver code is possible, but I'd >>>> rather not start from that position, particularly as it would require things >>>> like sign-extension in the TLBI callbacks. >> >> Bumping this as is our tradition in the -rc1 time frame before we get all >> distracted with other stuff. It sounds like the last agreement was for a >> TTBR1 hint for the EDP and the sign extension in the functions. > > If somebody respins this using bitfields and an explicit TTBR1 quirk then > I'll merge it. Oops, the ping did register, I just didn't react outwardly ;) I have been working on v2, and plan to have something ready next week - the holdup was that I started refactoring all the argument passing since the number of things we have to carry through from one end of map/unmap to the other is getting a bit silly, but I think I can still finish the TTBR1 quirk without that, so if I don't get it cracked imminently then I'll put it aside to revisit later. Robin.
On Thu, Oct 24, 2019 at 12:23:46PM +0100, Robin Murphy wrote: > On 2019-10-24 11:51 am, Will Deacon wrote: > > On Thu, Oct 03, 2019 at 11:33:52AM -0600, Jordan Crouse wrote: > > > On Wed, Aug 21, 2019 at 01:56:20PM +0100, Robin Murphy wrote: > > > > On 21/08/2019 13:11, Will Deacon wrote: > > > > > On Tue, Aug 20, 2019 at 07:41:52PM +0100, Robin Murphy wrote: > > > > > > On 20/08/2019 17:07, Will Deacon wrote: > > > > > > > On Tue, Aug 20, 2019 at 04:25:56PM +0100, Robin Murphy wrote: > > > > > > > > On 20/08/2019 11:31, Will Deacon wrote: > > > > > > > > > On Mon, Aug 19, 2019 at 07:19:30PM +0100, Robin Murphy wrote: > > > > > > > > > > Although it's conceptually nice for the io_pgtable_cfg to provide a > > > > > > > > > > standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU > > > > > > > > > > looks exactly like an Arm CPU, and they all have various other TCR > > > > > > > > > > controls which io-pgtable can't be expected to understand. Thus since > > > > > > > > > > there is an expectation that drivers will have to add to the given TCR > > > > > > > > > > value anyway, let's strip it down to just the essentials that are > > > > > > > > > > directly relevant to io-pgatble's inner workings - namely the address > > > > > > > > > > sizes, walk attributes, and where appropriate, format selection. > > > > > > > > > > > > > > > > > > > > Signed-off-by: Robin Murphy <robin.murphy@arm.com> > > > > > > > > > > --- > > > > > > > > > > drivers/iommu/arm-smmu-v3.c | 7 +------ > > > > > > > > > > drivers/iommu/arm-smmu.c | 1 + > > > > > > > > > > drivers/iommu/arm-smmu.h | 2 ++ > > > > > > > > > > drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- > > > > > > > > > > drivers/iommu/io-pgtable-arm.c | 4 ---- > > > > > > > > > > drivers/iommu/qcom_iommu.c | 2 +- > > > > > > > > > > 6 files changed, 7 insertions(+), 15 deletions(-) > > > > > > > > > > > > > > > > > > Hmm, so I'm a bit nervous about this one since I think we really should > > > > > > > > > be providing a TCR with EPD1 set if we're only giving you TTBR0. Relying > > > > > > > > > on the driver to do this worries me. See my comments on the next patch. > > > > > > > > > > > > > > > > The whole idea is that we already know we can't provide a *complete* TCR > > > > > > > > value (not least because anything above bit 31 is the wild west), thus > > > > > > > > there's really no point in io-pgtable trying to provide anything other than > > > > > > > > the parts it definitely controls. It makes sense to provide this partial TCR > > > > > > > > value "as if" for TTBR0, since that's the most common case, but ultimately > > > > > > > > io-pgatble doesn't know (or need to) which TTBR the caller intends to > > > > > > > > actually use for this table. Even if the caller *is* allocating it for > > > > > > > > TTBR0, io-pgtable doesn't know that they haven't got something live in TTBR1 > > > > > > > > already, so it still wouldn't be in a position to make the EPD1 call either > > > > > > > > way. > > > > > > > > > > > > > > Ok, but the driver can happily rewrite/ignore what it gets back. I suppose > > > > > > > an alternative would be scrapped the 'u64 tcr' and instead having a bunch > > > > > > > of named bitfields for the stuff we're actually providing, although I'd > > > > > > > still like EPDx to be in there. > > > > > > > > > > > > I like the bitfield idea; it would certainly emphasise the "you have to do > > > > > > something more with this" angle that I'm pushing towards here, but still > > > > > > leave things framed in TCR terms without having to go to some more general > > > > > > abstraction. It really doesn't play into your EPD argument though - such a > > > > > > config would be providing TxSZ/TGx/IRGNx/ORGNx/SHx, but EPDy, for y = !x. > > > > > > For a driver to understand that and do the right thing with it is even more > > > > > > involved than for the driver to just set EPD1 by itself anyway. > > > > > > > > > > Having considered the bitfield idea some more, I'm less attached to EPDx > > > > > because we simply wouldn't be making a statement about them, rather than a > > > > > (dangerous) zero value and expecting it to be ignored. So I think we're in > > > > > agreement on that. > > > > > > > > Cool, I'll give bitfields a go for v2. > > > > > > > > > The only part I'm still stuck to is that I think io-pgtable should know > > > > > whether it's targetting TTBR0 or TTBR1 so that it can sanitise input > > > > > addresses correctly. Doing this in the driver code is possible, but I'd > > > > > rather not start from that position, particularly as it would require things > > > > > like sign-extension in the TLBI callbacks. > > > > > > Bumping this as is our tradition in the -rc1 time frame before we get all > > > distracted with other stuff. It sounds like the last agreement was for a > > > TTBR1 hint for the EDP and the sign extension in the functions. > > > > If somebody respins this using bitfields and an explicit TTBR1 quirk then > > I'll merge it. > > Oops, the ping did register, I just didn't react outwardly ;) > > I have been working on v2, and plan to have something ready next week - the > holdup was that I started refactoring all the argument passing since the > number of things we have to carry through from one end of map/unmap to the > other is getting a bit silly, but I think I can still finish the TTBR1 quirk > without that, so if I don't get it cracked imminently then I'll put it aside > to revisit later. Cheers, Robin! I'll keep an eye out for it. Will
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 2e50cf49c3c4..c8e83f72893f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -270,9 +270,7 @@ #define CTXDESC_CD_0_TCR_SH0 GENMASK_ULL(13, 12) #define ARM64_TCR_SH0 GENMASK_ULL(13, 12) #define CTXDESC_CD_0_TCR_EPD0 (1ULL << 14) -#define ARM64_TCR_EPD0 (1ULL << 7) #define CTXDESC_CD_0_TCR_EPD1 (1ULL << 30) -#define ARM64_TCR_EPD1 (1ULL << 23) #define CTXDESC_CD_0_ENDI (1UL << 15) #define CTXDESC_CD_0_V (1UL << 31) @@ -280,7 +278,6 @@ #define CTXDESC_CD_0_TCR_IPS GENMASK_ULL(34, 32) #define ARM64_TCR_IPS GENMASK_ULL(34, 32) #define CTXDESC_CD_0_TCR_TBI0 (1ULL << 38) -#define ARM64_TCR_TBI0 (1ULL << 37) #define CTXDESC_CD_0_AA64 (1UL << 41) #define CTXDESC_CD_0_S (1UL << 44) @@ -1064,8 +1061,6 @@ static u64 arm_smmu_cpu_tcr_to_cd(u64 tcr) val |= ARM_SMMU_TCR2CD(tcr, IRGN0); val |= ARM_SMMU_TCR2CD(tcr, ORGN0); val |= ARM_SMMU_TCR2CD(tcr, SH0); - val |= ARM_SMMU_TCR2CD(tcr, EPD0); - val |= ARM_SMMU_TCR2CD(tcr, EPD1); val |= ARM_SMMU_TCR2CD(tcr, IPS); return val; @@ -1086,7 +1081,7 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_device *smmu, #endif CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cfg->cd.asid) | - CTXDESC_CD_0_V; + CTXDESC_CD_0_V | CTXDESC_CD_0_TCR_EPD1; /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index 19030c4b5904..49c734a3814b 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -461,6 +461,7 @@ static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain, cb->tcr[0] = pgtbl_cfg->arm_v7s_cfg.tcr; } else { cb->tcr[0] = pgtbl_cfg->arm_lpae_s1_cfg.tcr; + cb->tcr[0] |= TCR_EPD1; cb->tcr[1] = pgtbl_cfg->arm_lpae_s1_cfg.tcr >> 32; cb->tcr[1] |= FIELD_PREP(TCR2_SEP, TCR2_SEP_UPSTREAM); if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64) diff --git a/drivers/iommu/arm-smmu.h b/drivers/iommu/arm-smmu.h index ac9eac966cf5..7b0e4d238558 100644 --- a/drivers/iommu/arm-smmu.h +++ b/drivers/iommu/arm-smmu.h @@ -162,6 +162,8 @@ enum arm_smmu_cbar_type { #define TTBRn_ASID GENMASK_ULL(63, 48) #define ARM_SMMU_CB_TCR 0x30 +#define TCR_EPD1 BIT(23) + #define ARM_SMMU_CB_CONTEXTIDR 0x34 #define ARM_SMMU_CB_S1_MAIR0 0x38 #define ARM_SMMU_CB_S1_MAIR1 0x3c diff --git a/drivers/iommu/io-pgtable-arm-v7s.c b/drivers/iommu/io-pgtable-arm-v7s.c index 7c80e9d00f73..a53f8d904f66 100644 --- a/drivers/iommu/io-pgtable-arm-v7s.c +++ b/drivers/iommu/io-pgtable-arm-v7s.c @@ -147,8 +147,6 @@ #define ARM_V7S_TTBR_IRGN_ATTR(attr) \ ((((attr) & 0x1) << 6) | (((attr) & 0x2) >> 1)) -#define ARM_V7S_TCR_PD1 BIT(5) - #ifdef CONFIG_ZONE_DMA32 #define ARM_V7S_TABLE_GFP_DMA GFP_DMA32 #define ARM_V7S_TABLE_SLAB_FLAGS SLAB_CACHE_DMA32 @@ -751,8 +749,8 @@ static struct io_pgtable *arm_v7s_alloc_pgtable(struct io_pgtable_cfg *cfg, */ cfg->pgsize_bitmap &= SZ_4K | SZ_64K | SZ_1M | SZ_16M; - /* TCR: T0SZ=0, disable TTBR1 */ - cfg->arm_v7s_cfg.tcr = ARM_V7S_TCR_PD1; + /* TCR: T0SZ=0, EAE=0 (if applicable) */ + cfg->arm_v7s_cfg.tcr = 0; /* * TEX remap: the indices used map to the closest equivalent types diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 56e0921331c3..09cb20671fbb 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -112,8 +112,6 @@ #define ARM_32_LPAE_TCR_EAE (1 << 31) #define ARM_64_LPAE_S2_TCR_RES1 (1 << 31) -#define ARM_LPAE_TCR_EPD1 (1 << 23) - #define ARM_LPAE_TCR_TG0_4K (0 << 14) #define ARM_LPAE_TCR_TG0_64K (1 << 14) #define ARM_LPAE_TCR_TG0_16K (2 << 14) @@ -845,8 +843,6 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie) reg |= (64ULL - cfg->ias) << ARM_LPAE_TCR_T0SZ_SHIFT; - /* Disable speculative walks through TTBR1 */ - reg |= ARM_LPAE_TCR_EPD1; cfg->arm_lpae_s1_cfg.tcr = reg; /* MAIRs */ diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index de55b6d82ef1..c58237bb2a86 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -257,7 +257,7 @@ static int qcom_iommu_init_domain(struct iommu_domain *domain, (pgtbl_cfg.arm_lpae_s1_cfg.tcr >> 32) | FIELD_PREP(TCR2_SEP, TCR2_SEP_UPSTREAM)); iommu_writel(ctx, ARM_SMMU_CB_TCR, - pgtbl_cfg.arm_lpae_s1_cfg.tcr); + pgtbl_cfg.arm_lpae_s1_cfg.tcr | TCR_EPD1); /* MAIRs (stage-1 only) */ iommu_writel(ctx, ARM_SMMU_CB_S1_MAIR0,
Although it's conceptually nice for the io_pgtable_cfg to provide a standard VMSA TCR value, the reality is that no VMSA-compliant IOMMU looks exactly like an Arm CPU, and they all have various other TCR controls which io-pgtable can't be expected to understand. Thus since there is an expectation that drivers will have to add to the given TCR value anyway, let's strip it down to just the essentials that are directly relevant to io-pgatble's inner workings - namely the address sizes, walk attributes, and where appropriate, format selection. Signed-off-by: Robin Murphy <robin.murphy@arm.com> --- drivers/iommu/arm-smmu-v3.c | 7 +------ drivers/iommu/arm-smmu.c | 1 + drivers/iommu/arm-smmu.h | 2 ++ drivers/iommu/io-pgtable-arm-v7s.c | 6 ++---- drivers/iommu/io-pgtable-arm.c | 4 ---- drivers/iommu/qcom_iommu.c | 2 +- 6 files changed, 7 insertions(+), 15 deletions(-)