| Message ID | 20200920110010.16796-2-Sergey.Semin@baikalelectronics.ru (mailing list archive) |
|---|---|
| State | Deferred |
| Headers | show |
| Series | mips: Introduce some IO-accessors optimizations | expand |
在 2020/9/20 19:00, Serge Semin 写道: > In accordance with [1, 2] memory transactions using CCA=2 (Uncached > Cacheability and Coherency Attribute) are always strongly ordered. This > means the younger memory accesses using CCA=2 are never allowed to be > executed before older memory accesses using CCA=2 (no bypassing is > allowed), and Loads and Stores using CCA=2 are never speculative. It is > expected by the specification that the rest of the system maintains these > properties for processor initiated uncached accesses. So the system IO > interconnect doesn't reorder uncached transactions once they have left the > processor subsystem. Taking into account these properties and what [3] > says about the relaxed IO-accessors we can infer that normal Loads and > Stores from/to CCA=2 memory and without any additional execution barriers > will fully comply with the {read,write}X_relaxed() methods requirements. > > Let's convert then currently generated relaxed IO-accessors to being pure > Loads and Stores. Seeing the commit 3d474dacae72 ("MIPS: Enforce strong > ordering for MMIO accessors") and commit 8b656253a7a4 ("MIPS: Provide > actually relaxed MMIO accessors") have already made a preparation in the > corresponding macro, we can do that just by replacing the "barrier" > parameter utilization with the "relax" one. Note the "barrier" macro > argument can be removed, since it isn't fully used anyway other than being > always assigned to 1. > > Of course it would be fullish to believe that all the available MIPS-based > CPUs completely follow the denoted specification, especially considering > how old the architecture is. Instead we introduced a dedicated kernel > config, which when enabled will convert the relaxed IO-accessors to being > pure Loads and Stores without any additional barriers around. So if some > CPU supports the strongly ordered UC memory access, it can enable that > config and use a fully optimized relaxed IO-methods. For instance, > Baikal-T1 architecture support code will do that. > > [1] MIPS Coherence Protocol Specification, Document Number: MD00605, > Revision 01.01. September 14, 2015, 4.2 Execution Order Behavior, > p. 33 > > [2] MIPS Coherence Protocol Specification, Document Number: MD00605, > Revision 01.01. September 14, 2015, 4.8.1 IO Device Access, p. 58 > > [3] "LINUX KERNEL MEMORY BARRIERS", Documentation/memory-barriers.txt, > Section "KERNEL I/O BARRIER EFFECTS" > > Signed-off-by: Serge Semin <Sergey.Semin@baikalelectronics.ru> > Cc: Maciej W. Rozycki <macro@linux-mips.org> Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com> Based on #mipslinus discussions, I suspect this option can be selected by most modern MIPS processors including all IMG/MTI cores, Ingenic and Loongson. Thanks. - Jiaxun > --- > arch/mips/Kconfig | 8 ++++++++ > arch/mips/include/asm/io.h | 20 ++++++++++---------- > 2 files changed, 18 insertions(+), 10 deletions(-) >
On Fri, Sep 25, 2020 at 11:54:20AM +0800, Jiaxun Yang wrote: > > > 在 2020/9/20 19:00, Serge Semin 写道: > > In accordance with [1, 2] memory transactions using CCA=2 (Uncached > > Cacheability and Coherency Attribute) are always strongly ordered. This > > means the younger memory accesses using CCA=2 are never allowed to be > > executed before older memory accesses using CCA=2 (no bypassing is > > allowed), and Loads and Stores using CCA=2 are never speculative. It is > > expected by the specification that the rest of the system maintains these > > properties for processor initiated uncached accesses. So the system IO > > interconnect doesn't reorder uncached transactions once they have left the > > processor subsystem. Taking into account these properties and what [3] > > says about the relaxed IO-accessors we can infer that normal Loads and > > Stores from/to CCA=2 memory and without any additional execution barriers > > will fully comply with the {read,write}X_relaxed() methods requirements. > > > > Let's convert then currently generated relaxed IO-accessors to being pure > > Loads and Stores. Seeing the commit 3d474dacae72 ("MIPS: Enforce strong > > ordering for MMIO accessors") and commit 8b656253a7a4 ("MIPS: Provide > > actually relaxed MMIO accessors") have already made a preparation in the > > corresponding macro, we can do that just by replacing the "barrier" > > parameter utilization with the "relax" one. Note the "barrier" macro > > argument can be removed, since it isn't fully used anyway other than being > > always assigned to 1. > > > > Of course it would be fullish to believe that all the available MIPS-based > > CPUs completely follow the denoted specification, especially considering > > how old the architecture is. Instead we introduced a dedicated kernel > > config, which when enabled will convert the relaxed IO-accessors to being > > pure Loads and Stores without any additional barriers around. So if some > > CPU supports the strongly ordered UC memory access, it can enable that > > config and use a fully optimized relaxed IO-methods. For instance, > > Baikal-T1 architecture support code will do that. > > > > [1] MIPS Coherence Protocol Specification, Document Number: MD00605, > > Revision 01.01. September 14, 2015, 4.2 Execution Order Behavior, > > p. 33 > > > > [2] MIPS Coherence Protocol Specification, Document Number: MD00605, > > Revision 01.01. September 14, 2015, 4.8.1 IO Device Access, p. 58 > > > > [3] "LINUX KERNEL MEMORY BARRIERS", Documentation/memory-barriers.txt, > > Section "KERNEL I/O BARRIER EFFECTS" > > > > Signed-off-by: Serge Semin <Sergey.Semin@baikalelectronics.ru> > > Cc: Maciej W. Rozycki <macro@linux-mips.org> > Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com> > > > Based on #mipslinus discussions, I suspect this option can be selected by > most modern MIPS processors including all IMG/MTI cores, > Ingenic and Loongson. Thanks for reviewing the patch. Regarding the option. Alas it's not that easy and we must be very careful with assumption whether some processor supports the denoted feature. Even if the MIPS cores do imply the strict UC load/stores ordering, the system interconnects may still perform the out-of-order requests execution. For instance, the P5600 cores installed into our Baikal-T1 SoC do support the strong UC ordering, but there is a cascade of the OCP2AXI, AXI2AXI and AXI2APB bridges behind the CPU memory interface, each of which is equipped with an internal FIFO and some complicated logic of the traffic routing. So each platform should be carefully analyzed and tested (if it's possible) before enabling the suggested feature, otherwise we'll risk to end up with in general working, but at some point buggy, systems. Needless to say, that out-of-order exec problems is very hard to track and debug due to a random nature of impact on the system. -Sergey > > Thanks. > > - Jiaxun > > > --- > > arch/mips/Kconfig | 8 ++++++++ > > arch/mips/include/asm/io.h | 20 ++++++++++---------- > > 2 files changed, 18 insertions(+), 10 deletions(-) > >
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index c95fa3a2484c..2c82d927347d 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -2066,6 +2066,14 @@ config WEAK_ORDERING # config WEAK_REORDERING_BEYOND_LLSC bool + +# +# CPU may not reorder reads and writes R->R, R->W, W->R, W->W within Uncached +# Cacheability and Coherency Attribute (CCA=2) +# +config STRONG_UC_ORDERING + bool + endmenu # diff --git a/arch/mips/include/asm/io.h b/arch/mips/include/asm/io.h index 78537aa23500..130c4b6458fc 100644 --- a/arch/mips/include/asm/io.h +++ b/arch/mips/include/asm/io.h @@ -213,7 +213,7 @@ void iounmap(const volatile void __iomem *addr); #define war_io_reorder_wmb() barrier() #endif -#define __BUILD_MEMORY_SINGLE(pfx, bwlq, type, barrier, relax, irq) \ +#define __BUILD_MEMORY_SINGLE(pfx, bwlq, type, relax, irq) \ \ static inline void pfx##write##bwlq(type val, \ volatile void __iomem *mem) \ @@ -221,7 +221,7 @@ static inline void pfx##write##bwlq(type val, \ volatile type *__mem; \ type __val; \ \ - if (barrier) \ + if (!(relax && IS_ENABLED(CONFIG_STRONG_UC_ORDERING))) \ iobarrier_rw(); \ else \ war_io_reorder_wmb(); \ @@ -262,7 +262,7 @@ static inline type pfx##read##bwlq(const volatile void __iomem *mem) \ \ __mem = (void *)__swizzle_addr_##bwlq((unsigned long)(mem)); \ \ - if (barrier) \ + if (!(relax && IS_ENABLED(CONFIG_STRONG_UC_ORDERING))) \ iobarrier_rw(); \ \ if (sizeof(type) != sizeof(u64) || sizeof(u64) == sizeof(long)) \ @@ -294,14 +294,14 @@ static inline type pfx##read##bwlq(const volatile void __iomem *mem) \ return pfx##ioswab##bwlq(__mem, __val); \ } -#define __BUILD_IOPORT_SINGLE(pfx, bwlq, type, barrier, relax, p) \ +#define __BUILD_IOPORT_SINGLE(pfx, bwlq, type, relax, p) \ \ static inline void pfx##out##bwlq##p(type val, unsigned long port) \ { \ volatile type *__addr; \ type __val; \ \ - if (barrier) \ + if (!(relax && IS_ENABLED(CONFIG_STRONG_UC_ORDERING))) \ iobarrier_rw(); \ else \ war_io_reorder_wmb(); \ @@ -325,7 +325,7 @@ static inline type pfx##in##bwlq##p(unsigned long port) \ \ BUILD_BUG_ON(sizeof(type) > sizeof(unsigned long)); \ \ - if (barrier) \ + if (!(relax && IS_ENABLED(CONFIG_STRONG_UC_ORDERING))) \ iobarrier_rw(); \ \ __val = *__addr; \ @@ -338,7 +338,7 @@ static inline type pfx##in##bwlq##p(unsigned long port) \ #define __BUILD_MEMORY_PFX(bus, bwlq, type, relax) \ \ -__BUILD_MEMORY_SINGLE(bus, bwlq, type, 1, relax, 1) +__BUILD_MEMORY_SINGLE(bus, bwlq, type, relax, 1) #define BUILDIO_MEM(bwlq, type) \ \ @@ -358,8 +358,8 @@ __BUILD_MEMORY_PFX(__mem_, q, u64, 0) #endif #define __BUILD_IOPORT_PFX(bus, bwlq, type) \ - __BUILD_IOPORT_SINGLE(bus, bwlq, type, 1, 0,) \ - __BUILD_IOPORT_SINGLE(bus, bwlq, type, 1, 0, _p) + __BUILD_IOPORT_SINGLE(bus, bwlq, type, 0,) \ + __BUILD_IOPORT_SINGLE(bus, bwlq, type, 0, _p) #define BUILDIO_IOPORT(bwlq, type) \ __BUILD_IOPORT_PFX(, bwlq, type) \ @@ -374,7 +374,7 @@ BUILDIO_IOPORT(q, u64) #define __BUILDIO(bwlq, type) \ \ -__BUILD_MEMORY_SINGLE(____raw_, bwlq, type, 1, 0, 0) +__BUILD_MEMORY_SINGLE(____raw_, bwlq, type, 0, 0) __BUILDIO(q, u64)
In accordance with [1, 2] memory transactions using CCA=2 (Uncached Cacheability and Coherency Attribute) are always strongly ordered. This means the younger memory accesses using CCA=2 are never allowed to be executed before older memory accesses using CCA=2 (no bypassing is allowed), and Loads and Stores using CCA=2 are never speculative. It is expected by the specification that the rest of the system maintains these properties for processor initiated uncached accesses. So the system IO interconnect doesn't reorder uncached transactions once they have left the processor subsystem. Taking into account these properties and what [3] says about the relaxed IO-accessors we can infer that normal Loads and Stores from/to CCA=2 memory and without any additional execution barriers will fully comply with the {read,write}X_relaxed() methods requirements. Let's convert then currently generated relaxed IO-accessors to being pure Loads and Stores. Seeing the commit 3d474dacae72 ("MIPS: Enforce strong ordering for MMIO accessors") and commit 8b656253a7a4 ("MIPS: Provide actually relaxed MMIO accessors") have already made a preparation in the corresponding macro, we can do that just by replacing the "barrier" parameter utilization with the "relax" one. Note the "barrier" macro argument can be removed, since it isn't fully used anyway other than being always assigned to 1. Of course it would be fullish to believe that all the available MIPS-based CPUs completely follow the denoted specification, especially considering how old the architecture is. Instead we introduced a dedicated kernel config, which when enabled will convert the relaxed IO-accessors to being pure Loads and Stores without any additional barriers around. So if some CPU supports the strongly ordered UC memory access, it can enable that config and use a fully optimized relaxed IO-methods. For instance, Baikal-T1 architecture support code will do that. [1] MIPS Coherence Protocol Specification, Document Number: MD00605, Revision 01.01. September 14, 2015, 4.2 Execution Order Behavior, p. 33 [2] MIPS Coherence Protocol Specification, Document Number: MD00605, Revision 01.01. September 14, 2015, 4.8.1 IO Device Access, p. 58 [3] "LINUX KERNEL MEMORY BARRIERS", Documentation/memory-barriers.txt, Section "KERNEL I/O BARRIER EFFECTS" Signed-off-by: Serge Semin <Sergey.Semin@baikalelectronics.ru> Cc: Maciej W. Rozycki <macro@linux-mips.org> --- arch/mips/Kconfig | 8 ++++++++ arch/mips/include/asm/io.h | 20 ++++++++++---------- 2 files changed, 18 insertions(+), 10 deletions(-)