| Message ID | 20170109213944.31970-1-jarod@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
On Mon, Jan 09, 2017 at 04:39:44PM -0500, Jarod Wilson wrote: > Building for 32-bit arm, things fall down, due to lack of arch-specific > memory barriers. Since we now have rxe that should work on ARM I think we need to fix this upstream.. Do you have time to test some patches on ARM? Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2017-01-09 5:31 PM, Jason Gunthorpe wrote: > On Mon, Jan 09, 2017 at 04:39:44PM -0500, Jarod Wilson wrote: >> Building for 32-bit arm, things fall down, due to lack of arch-specific >> memory barriers. > > Since we now have rxe that should work on ARM I think we need to fix > this upstream.. > > Do you have time to test some patches on ARM? Afraid I don't have hardware. To the best of my knowledge, the only ARM systems my group has (and/or are getting soon) with RDMA hardware is all ARM64.
On Mon, Jan 09, 2017 at 10:27:52PM -0500, Jarod Wilson wrote: > On 2017-01-09 5:31 PM, Jason Gunthorpe wrote: > >On Mon, Jan 09, 2017 at 04:39:44PM -0500, Jarod Wilson wrote: > >>Building for 32-bit arm, things fall down, due to lack of arch-specific > >>memory barriers. > > > >Since we now have rxe that should work on ARM I think we need to fix > >this upstream.. > > > >Do you have time to test some patches on ARM? > > Afraid I don't have hardware. To the best of my knowledge, the only ARM > systems my group has (and/or are getting soon) with RDMA hardware is all > ARM64. RDMA hardware doesn't work in ARM so that is OK. Cross compiling is fine, ideal would be to test RXE on an arbitary ARM system, but I wouldn't call that critical. I just don't want to setup a cross compiler for this :\ Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 1/9/2017 5:31 PM, Jason Gunthorpe wrote: > On Mon, Jan 09, 2017 at 04:39:44PM -0500, Jarod Wilson wrote: >> Building for 32-bit arm, things fall down, due to lack of arch-specific >> memory barriers. > > Since we now have rxe that should work on ARM I think we need to fix > this upstream.. > > Do you have time to test some patches on ARM? Looks to me that issue was introduced by: commit 1df0888f6a736e1612ce8b054d6c17651ebd003f Author: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> Date: Fri Sep 2 12:57:57 2016 -0600 Remove most checks of __BYTE_ORDER For a long time now endian.h has defined sane fixed with conversion macros, so lets just use them instead of rolling our own. Also, htonll is defined in this source tree under infiniband/arch.h, so all users of that macro can just use the header. Someday we should also get rid of all the endless wrappers.. Signed-off-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> where byte order macros are from libibverbs arch.h which also includes the memory barrier stuff. So can't arch.h be separated out into 2 headers and have the various parts of rdma-core include one or both of these headers as needed ? -- Hal > Jason > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2017-01-10 at 11:30 -0500, Hal Rosenstock wrote: > On 1/9/2017 5:31 PM, Jason Gunthorpe wrote: > > > > On Mon, Jan 09, 2017 at 04:39:44PM -0500, Jarod Wilson wrote: > > > > > > Building for 32-bit arm, things fall down, due to lack of arch- > > > specific > > > memory barriers. > > > > Since we now have rxe that should work on ARM I think we need to > > fix > > this upstream.. > > > > Do you have time to test some patches on ARM? > > Looks to me that issue was introduced by: > > commit 1df0888f6a736e1612ce8b054d6c17651ebd003f > Author: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> > Date: Fri Sep 2 12:57:57 2016 -0600 > > Remove most checks of __BYTE_ORDER > > For a long time now endian.h has defined sane fixed with > conversion > macros, so lets just use them instead of rolling our own. > > Also, htonll is defined in this source tree under > infiniband/arch.h, > so all users of that macro can just use the header. > > Someday we should also get rid of all the endless wrappers.. > > Signed-off-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> > > where byte order macros are from libibverbs arch.h which also > includes > the memory barrier stuff. > > So can't arch.h be separated out into 2 headers and have the various > parts of rdma-core include one or both of these headers as needed ? No, that won't help. At most it would shift where the compile breaks. When Steve Wise did the patch to add the arm64 memory barriers, we discussed on list the fact that not having a proper memory barrier define for an arch should be a fatal error, so we also made it such that without an arch specific memory barrier define, we fail the build. If you split the headers, then somewhere you are going to come to a file that needs the memory barriers defined in the headers, and when you include the header, the compile error returns. The proper fix is just to get 32bit arm defines in place.
On Tue, Jan 10, 2017 at 12:36:04PM -0500, Doug Ledford wrote: > file that needs the memory barriers defined in the headers, and when > you include the header, the compile error returns. ??The proper fix is > just to get 32bit arm defines in place. It is a little worse than that, 32 bit ARM is not DMA coherent at all so it simply cannot work. The fix is to not compile providers that require DMA coherence on arches that cannot provide it. Bit tricky, but almost done.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> RDMA hardware doesn't work in ARM so that is OK. Cross compiling is fine, > ideal would be to test RXE on an arbitary ARM system, but I wouldn't call > that critical. > > I just don't want to setup a cross compiler for this :\ > > Jason > -- Hi Jason, Doug, All, I wanted to bring to your attention that our RDMA solution *is* working in an ARM based environment: the Cavium ThunderX platform. This is a 64bit ARMv8 *with* cache coherency. Please keep that in mind when considering RDMA-relevant platforms. Thanks, Ram -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jan 12, 2017 at 12:32:37PM +0000, Amrani, Ram wrote: > > RDMA hardware doesn't work in ARM so that is OK. Cross compiling is fine, > > ideal would be to test RXE on an arbitary ARM system, but I wouldn't call > > that critical. > > > > I just don't want to setup a cross compiler for this :\ > > > > Jason > > -- > > Hi Jason, Doug, All, > > I wanted to bring to your attention that our RDMA solution *is* working in an ARM based environment: > the Cavium ThunderX platform. This is a 64bit ARMv8 *with* cache coherency. Please keep that in mind > when considering RDMA-relevant platforms. They are referring to 32bit ARM in their discussion. AFAIK all devices in RDMA are working and supported in 64bit ARM environment. > > Thanks, > Ram > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > RDMA hardware doesn't work in ARM so that is OK. Cross compiling is > > > fine, ideal would be to test RXE on an arbitary ARM system, but I > > > wouldn't call that critical. > > > > > > I just don't want to setup a cross compiler for this :\ > > > > > > Jason > > > -- > > > > Hi Jason, Doug, All, > > > > I wanted to bring to your attention that our RDMA solution *is* working > in an ARM based environment: > > the Cavium ThunderX platform. This is a 64bit ARMv8 *with* cache > > coherency. Please keep that in mind when considering RDMA-relevant > platforms. > > They are referring to 32bit ARM in their discussion. AFAIK all devices in > RDMA are working and supported in 64bit ARM environment. I understand that the thread relates to 32-bit. I wanted to make sure that it doesn't impact 64-bit, as the sentence - "RDMA hardware doesn't work in ARM " may imply that ARM doesn't support RDMA at all and that 64-bit may be dropped. Ram -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jan 12, 2017 at 01:15:19PM +0000, Amrani, Ram wrote: > > They are referring to 32bit ARM in their discussion. AFAIK all devices in > > RDMA are working and supported in 64bit ARM environment. > > I understand that the thread relates to 32-bit. I wanted to make sure that it > doesn't impact 64-bit, as the sentence - "RDMA hardware doesn't work > in ARM " may imply that ARM doesn't support RDMA at all and that 64-bit > may be dropped. hhns is already 64 bit ARM only hardware, it will be supported.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec index 246079e..4d31d53 100644 --- a/redhat/rdma-core.spec +++ b/redhat/rdma-core.spec @@ -166,6 +166,7 @@ level work required to open an RDMA connection between two machines. %package -n libibumad Summary: OpenFabrics Alliance InfiniBand umad (userspace management datagram) library +ExcludeArch: %{arm} Requires: %{name}%{?_isa} = %{version}-%{release} %description -n libibumad
Building for 32-bit arm, things fall down, due to lack of arch-specific memory barriers. In file included from /builddir/build/BUILD/rdma-core-12/libibumad/sysfs.c:46:0: include/infiniband/arch.h:132:2: error: #error No architecture specific memory barrier defines found! #error No architecture specific memory barrier defines found! Reported-by: Honggang Li <honli@redhat.com> Signed-off-by: Jarod Wilson <jarod@redhat.com> --- redhat/rdma-core.spec | 1 + 1 file changed, 1 insertion(+)