| Message ID | 20250325121624.523258-1-guoren@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | rv64ilp32_abi: Build CONFIG_64BIT kernel-self with ILP32 ABI | expand |
On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: > From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> > > Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, > but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI I'm thinking you're going to be finding a metric ton of assumptions about 'unsigned long' being 64bit when 64BIT=y throughout the kernel. I know of a couple of places where 64BIT will result in different math such that a 32bit 'unsigned long' will trivially overflow. Please, don't do this. This adds a significant maintenance burden on all of us.
On Tue, Mar 25, 2025 at 8:27 PM Peter Zijlstra <peterz@infradead.org> wrote: > > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: > > From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> > > > > Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, > > but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > > I'm thinking you're going to be finding a metric ton of assumptions > about 'unsigned long' being 64bit when 64BIT=y throughout the kernel. Less than you imagined. Most code is compatible with ILP32 ABI due to the CONFIG_32BIT. In my practice, it's deemed acceptable. > > I know of a couple of places where 64BIT will result in different math > such that a 32bit 'unsigned long' will trivially overflow. I would be grateful if you could share some with me. > > Please, don't do this. This adds a significant maintenance burden on all > of us. The 64ILP32 ABI would bear the maintenance burden, not traditional 64-bit or 32-bit ABIs. The patch set won't impact other CONFIG_64BIT or CONFIG_32BIT. Numerous RV64 chips require the RV64ILP32 ABI to reduce the memory and cache footprint; we will bear the burden. The core code maintainers would receive patches that would make them use BITS_PER_LONG and CONFIG_64BIT more accurately.
On Tue, Mar 25, 2025, at 13:26, Peter Zijlstra wrote: > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: >> From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> >> >> Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, >> but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > > Please, don't do this. This adds a significant maintenance burden on all > of us. It would be easier to this with CONFIG_64BIT disabled and continue treating CONFIG_64BIT to be the same as BITS_PER_LONG=64, but I still think it's fundamentally a bad idea to support this in mainline kernels in any variation, other than supporting regular 32-bit compat mode tasks on a regular 64-bit kernel. >> The patchset targets RISC-V and is built on the RV64ILP32 ABI, which >> was introduced into RISC-V's psABI in January 2025 [1]. This patchset >> equips an rv64ilp32-abi kernel with all the functionalities of a >> traditional lp64-abi kernel, yet restricts the address space to 2GiB. >> Hence, the rv64ilp32-abi kernel simultaneously supports lp64-abi >> userspace and ilp32-abi (compat) userspace, the same as the >> traditional lp64-abi kernel. You declare the syscall ABI to be the native 64-bit ABI, but this is fundamentally not true because a many uapi structures are defined in terms of 'long' or pointer values, in particular in the ioctl call. This might work for an rv64ilp32 userspace that uses the same headers and the same types, but you explicitly say that the goal is to run native rv64 or compat rv32 tasks, not rv64ilp32 (thanks!). As far as I can tell, there is no way to rectify this design flaw other than to drop support for 64-bit userspace and only support regular rv32 userspace. I'm also skeptical that supporting rv64 userspace helps in practice other than for testing, since generally most memory overhead is in userspace rather than the kernel, and there is much more to gain from shrinking the larger userspace by running rv32 compat mode binaries on a 64-bit kernel than the other way round. If you remove the CONFIG_64BIT changes that Peter mentioned and the support for ilp64 userland from your series, you end up with a kernel that is very similar to a native rv32 kernel but executes as rv64ilp32 and runs rv32 userspace. I don't have any objections to that approach, and the same thing has come up on arm64 as a possible idea as well, but I don't know if that actually brings any notable advantage over an rv32 kernel. Are there CPUs that can run rv64 kernels and rv32 userspace but not rv32 kernels, similar to what we have on Arm Cortex-A76 and Cortex-A510? Arnd
On 25.03.25 13:26, Peter Zijlstra wrote: > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: >> From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> >> >> Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, >> but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > > I'm thinking you're going to be finding a metric ton of assumptions > about 'unsigned long' being 64bit when 64BIT=y throughout the kernel. > > I know of a couple of places where 64BIT will result in different math > such that a 32bit 'unsigned long' will trivially overflow. > > Please, don't do this. This adds a significant maintenance burden on all > of us. > Fully agreed.
* David Hildenbrand <david@redhat.com> [250325 14:52]: > On 25.03.25 13:26, Peter Zijlstra wrote: > > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: > > > From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> > > > > > > Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, > > > but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > > > > I'm thinking you're going to be finding a metric ton of assumptions > > about 'unsigned long' being 64bit when 64BIT=y throughout the kernel. > > > > I know of a couple of places where 64BIT will result in different math > > such that a 32bit 'unsigned long' will trivially overflow. > > > > Please, don't do this. This adds a significant maintenance burden on all > > of us. > > > > Fully agreed. I would go further and say I do not want this to go in. The open ended maintenance burden is not worth extending hardware life of a board with 16mb of ram (If I understand your 2023 LPC slides correctly). Thank you, Liam
On Tue, Mar 25, 2025 at 9:18 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Mar 25, 2025, at 13:26, Peter Zijlstra wrote: > > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: > >> From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> > >> > >> Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, > >> but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > > > > Please, don't do this. This adds a significant maintenance burden on all > > of us. > > It would be easier to this with CONFIG_64BIT disabled and continue > treating CONFIG_64BIT to be the same as BITS_PER_LONG=64, but I still > think it's fundamentally a bad idea to support this in mainline > kernels in any variation, other than supporting regular 32-bit > compat mode tasks on a regular 64-bit kernel. > > >> The patchset targets RISC-V and is built on the RV64ILP32 ABI, which > >> was introduced into RISC-V's psABI in January 2025 [1]. This patchset > >> equips an rv64ilp32-abi kernel with all the functionalities of a > >> traditional lp64-abi kernel, yet restricts the address space to 2GiB. > >> Hence, the rv64ilp32-abi kernel simultaneously supports lp64-abi > >> userspace and ilp32-abi (compat) userspace, the same as the > >> traditional lp64-abi kernel. > > You declare the syscall ABI to be the native 64-bit ABI, but this > is fundamentally not true because a many uapi structures are > defined in terms of 'long' or pointer values, in particular in > the ioctl call. I modified uapi with void __user *msg_name; -> union {void __user *msg_name; u64 __msg_name;}; to make native 64-bit ABI. I would look at compat stuff instead of using __riscv_xlen macro. > This might work for an rv64ilp32 userspace that > uses the same headers and the same types, but you explicitly > say that the goal is to run native rv64 or compat rv32 tasks, > not rv64ilp32 (thanks!). It's not for rv64ilp32-abi userspace, no rv64ilp32-abi userspace introduced in the patch set. It's for native lp64-abi. Let's discuss this in the first patch thread: uapi: Reuse lp64 ABI interface > > As far as I can tell, there is no way to rectify this design flaw > other than to drop support for 64-bit userspace and only support > regular rv32 userspace. I'm also skeptical that supporting rv64 > userspace helps in practice other than for testing, since > generally most memory overhead is in userspace rather than the > kernel, and there is much more to gain from shrinking the larger > userspace by running rv32 compat mode binaries on a 64-bit kernel > than the other way round. The lp64-abi userspace rootfs works fine in this patch set, which proves the technique is valid. But the modification on uapi is raw, and I'm looking at compat stuff. Supporting lp64-abi userspace is essential because riscv lp64-abi and ilp32-abi userspace are hybrid deployments when the target is ilp32-abi userspace. The lp64-abi provides a good supplement to ilp32-abi which eases the development. > > If you remove the CONFIG_64BIT changes that Peter mentioned and > the support for ilp64 userland from your series, you end up > with a kernel that is very similar to a native rv32 kernel > but executes as rv64ilp32 and runs rv32 userspace. I don't have > any objections to that approach, and the same thing has come > up on arm64 as a possible idea as well, but I don't know if > that actually brings any notable advantage over an rv32 kernel. > > Are there CPUs that can run rv64 kernels and rv32 userspace > but not rv32 kernels, similar to what we have on Arm Cortex-A76 > and Cortex-A510? Yes, there is, and it only supports rv32 userspace, not rv32 kernel. https://www.xrvm.com/product/xuantie/C908 Here are the products: https://developer.canaan-creative.com/k230_canmv/en/dev/userguide/boards/canmv_k230d.html http://riscv.org/ecosystem-news/2024/07/unpacking-the-canmv-k230-risc-v-board/
On Wed, Mar 26, 2025, at 07:07, Guo Ren wrote: > On Tue, Mar 25, 2025 at 9:18 PM Arnd Bergmann <arnd@arndb.de> wrote: >> On Tue, Mar 25, 2025, at 13:26, Peter Zijlstra wrote: >> > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: >> >> You declare the syscall ABI to be the native 64-bit ABI, but this >> is fundamentally not true because a many uapi structures are >> defined in terms of 'long' or pointer values, in particular in >> the ioctl call. > > I modified uapi with > void __user *msg_name; > -> > union {void __user *msg_name; u64 __msg_name;}; > to make native 64-bit ABI. > > I would look at compat stuff instead of using __riscv_xlen macro. The problem I see here is that there are many more drivers that you did not modify than drivers that you did change this way. The union is particularly ugly, but even if you find a nicer method of doing this, you now also put the burden on future driver writers to do this right for your platform. >> As far as I can tell, there is no way to rectify this design flaw >> other than to drop support for 64-bit userspace and only support >> regular rv32 userspace. I'm also skeptical that supporting rv64 >> userspace helps in practice other than for testing, since >> generally most memory overhead is in userspace rather than the >> kernel, and there is much more to gain from shrinking the larger >> userspace by running rv32 compat mode binaries on a 64-bit kernel >> than the other way round. > > The lp64-abi userspace rootfs works fine in this patch set, which > proves the technique is valid. But the modification on uapi is raw, > and I'm looking at compat stuff. There is a big difference between making it work for a particular set of userspace binaries and making it correct for the entire kernel ABI. I agree that limiting the hacks to the compat side while keeping the native ABI as ilp32 as in your previous versions is better, but I also don't think this can be easily done without major changes to how compat mode works in general, and that still seems like a show-stopper for two reasons: - it still puts the burden on driver writers to get it right for your platform. The scope is a bit smaller than in the current version because that would be limited to the compat handlers and not change the native codepath, but that's still a lot of drivers. - the way that I would imagine this to be implemented in practice would require changing the compat code in a way that allows multiple compat ABIs, so drivers can separate the normal 32-on-64 handling from the 64-on-32 version you need. We have discussed something like this in the past, but Linus has already made it very clear that he doesn't want it done that way. Whichever way you do it, this is unlikely to find consensus. > Supporting lp64-abi userspace is essential because riscv lp64-abi and > ilp32-abi userspace are hybrid deployments when the target is > ilp32-abi userspace. The lp64-abi provides a good supplement to > ilp32-abi which eases the development. I'm not following here, please clarify. I do understand that having a mixed 32/64 userspace can help for development, but that can already be done on a 64-bit kernel and it doesn't seem to be useful for deployment because having two sets of support libraries makes this counterproductive for the goal of saving RAM. >> If you remove the CONFIG_64BIT changes that Peter mentioned and >> the support for ilp64 userland from your series, you end up >> with a kernel that is very similar to a native rv32 kernel >> but executes as rv64ilp32 and runs rv32 userspace. I don't have >> any objections to that approach, and the same thing has come >> up on arm64 as a possible idea as well, but I don't know if >> that actually brings any notable advantage over an rv32 kernel. >> >> Are there CPUs that can run rv64 kernels and rv32 userspace >> but not rv32 kernels, similar to what we have on Arm Cortex-A76 >> and Cortex-A510? > > Yes, there is, and it only supports rv32 userspace, not rv32 kernel. > https://www.xrvm.com/product/xuantie/C908 Ok, thanks for the link. Arnd
On Wed, Mar 26, 2025 at 2:56 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Wed, Mar 26, 2025, at 07:07, Guo Ren wrote: > > On Tue, Mar 25, 2025 at 9:18 PM Arnd Bergmann <arnd@arndb.de> wrote: > >> On Tue, Mar 25, 2025, at 13:26, Peter Zijlstra wrote: > >> > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: > >> > >> You declare the syscall ABI to be the native 64-bit ABI, but this > >> is fundamentally not true because a many uapi structures are > >> defined in terms of 'long' or pointer values, in particular in > >> the ioctl call. > > > > I modified uapi with > > void __user *msg_name; > > -> > > union {void __user *msg_name; u64 __msg_name;}; > > to make native 64-bit ABI. > > > > I would look at compat stuff instead of using __riscv_xlen macro. > > The problem I see here is that there are many more drivers > that you did not modify than drivers that you did change this > way. The union is particularly ugly, but even if you find > a nicer method of doing this, you now also put the burden > on future driver writers to do this right for your platform. Got it. > > >> As far as I can tell, there is no way to rectify this design flaw > >> other than to drop support for 64-bit userspace and only support > >> regular rv32 userspace. I'm also skeptical that supporting rv64 > >> userspace helps in practice other than for testing, since > >> generally most memory overhead is in userspace rather than the > >> kernel, and there is much more to gain from shrinking the larger > >> userspace by running rv32 compat mode binaries on a 64-bit kernel > >> than the other way round. > > > > The lp64-abi userspace rootfs works fine in this patch set, which > > proves the technique is valid. But the modification on uapi is raw, > > and I'm looking at compat stuff. > > There is a big difference between making it work for a particular > set of userspace binaries and making it correct for the entire > kernel ABI. > > I agree that limiting the hacks to the compat side while keeping > the native ABI as ilp32 as in your previous versions is better, > but I also don't think this can be easily done without major > changes to how compat mode works in general, and that still > seems like a show-stopper for two reasons: > > - it still puts the burden on driver writers to get it right > for your platform. The scope is a bit smaller than in the > current version because that would be limited to the compat > handlers and not change the native codepath, but that's > still a lot of drivers. > > - the way that I would imagine this to be implemented in > practice would require changing the compat code in a way that > allows multiple compat ABIs, so drivers can separate the > normal 32-on-64 handling from the 64-on-32 version you need. > We have discussed something like this in the past, but Linus > has already made it very clear that he doesn't want it done > that way. Whichever way you do it, this is unlikely to > find consensus. Got it, thanks for analysing. > > > Supporting lp64-abi userspace is essential because riscv lp64-abi and > > ilp32-abi userspace are hybrid deployments when the target is > > ilp32-abi userspace. The lp64-abi provides a good supplement to > > ilp32-abi which eases the development. > > I'm not following here, please clarify. I do understand that > having a mixed 32/64 userspace can help for development, but > that can already be done on a 64-bit kernel and it doesn't > seem to be useful for deployment because having two sets of > support libraries makes this counterproductive for the goal > of saving RAM. In my case, most binaries and libraries are based on 32-bit, but a small part would remain on 64-bit, which may be statically linked. For RISC-V, the rv64 ecosystem is more complete than the rv32's. So, rv64-abi is always necessary, and rv32-abi is a supplement. > > >> If you remove the CONFIG_64BIT changes that Peter mentioned and > >> the support for ilp64 userland from your series, you end up > >> with a kernel that is very similar to a native rv32 kernel > >> but executes as rv64ilp32 and runs rv32 userspace. I don't have > >> any objections to that approach, and the same thing has come > >> up on arm64 as a possible idea as well, but I don't know if > >> that actually brings any notable advantage over an rv32 kernel. > >> > >> Are there CPUs that can run rv64 kernels and rv32 userspace > >> but not rv32 kernels, similar to what we have on Arm Cortex-A76 > >> and Cortex-A510? > > > > Yes, there is, and it only supports rv32 userspace, not rv32 kernel. > > https://www.xrvm.com/product/xuantie/C908 > > Ok, thanks for the link. > > Arnd >
On Tue, 25 Mar 2025 12:23:39 PDT (-0700), Liam.Howlett@oracle.com wrote: > * David Hildenbrand <david@redhat.com> [250325 14:52]: >> On 25.03.25 13:26, Peter Zijlstra wrote: >> > On Tue, Mar 25, 2025 at 08:15:41AM -0400, guoren@kernel.org wrote: >> > > From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> >> > > >> > > Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, >> > > but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI >> > >> > I'm thinking you're going to be finding a metric ton of assumptions >> > about 'unsigned long' being 64bit when 64BIT=y throughout the kernel. >> > >> > I know of a couple of places where 64BIT will result in different math >> > such that a 32bit 'unsigned long' will trivially overflow. Ya, I write code that assumes "unsigned long" is the size of a register pretty regularly. >> > >> > Please, don't do this. This adds a significant maintenance burden on all >> > of us. >> > >> >> Fully agreed. > > I would go further and say I do not want this to go in. Seems reasonable to me, and I think it's also been the general sentiment when this stuff comes up. This specific implementation seems particularly clunky, but I agree that it's going to be painful to do this sort of thing. > The open ended maintenance burden is not worth extending hardware life > of a board with 16mb of ram (If I understand your 2023 LPC slides > correctly). We can already run full 32-bit kernels on 64-bit hardware. The hardware needs to support configurable XLEN, but there's systems out there that do already. It's not like any of the existing RISC-V stuff ships in meaningful volumes. So I think it's fine to just say that vendors who want tiny memories should build hardware that plays nice with those constraints, and vendors who build hardware that doesn't make any sense get to pick up the pieces. I get RISC-V is where people go to have crazy ideas, but there's got to be a line somewhere... > > Thank you, > Liam
On Tue, 25 Mar 2025 08:15:41 -0400 guoren@kernel.org wrote: > From: "Guo Ren (Alibaba DAMO Academy)" <guoren@kernel.org> > > Since 2001, the CONFIG_64BIT kernel has been built with the LP64 ABI, > but this patchset allows the CONFIG_64BIT kernel to use an ILP32 ABI > for construction to reduce cache & memory footprint (Compared to > kernel-lp64-abi, kernel-rv64ilp32-abi decreased the used memory by > about 20%, as shown in "free -h" in the following demo.) ... Why on earth would you want to run a 64bit application on a 32bit kernel. IIRC the main justification for 64bit was to get a larger address space. Now you might want to compile a 32bit (ILP32) system that actually runs in 64bit mode (c/f x32) so that 64bit maths (long long) is more efficient - but that is a different issue. (I suspect you'd need to change the process switch code to save all 64bits of the registers - but maybe not much else??) David