| Message ID | 20240618022422.804305-2-jhubbard@nvidia.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 504d8a5e0fd4def825a58d69ca783dcfa424d012 |
| Headers | show |
| Series | cleanups, fixes, and progress towards avoiding "make headers" | expand |
On 18.06.24 04:24, John Hubbard wrote: > The selftests/mm build isn't exactly "broken", according to the current > documentation, which still claims that one must run "make headers", > before building the kselftests. However, according to the new plan to > get rid of that requirement [1], they are future-broken: attempting to > build selftests/mm *without* first running "make headers" will fail due > to not finding __NR_mseal. > > Therefore, include asm-generic/unistd.h, which has all of the system > call numbers that are needed, abstracted across the various CPU arches. > > [1] commit e076eaca5906 ("selftests: break the dependency upon local > header files") > > Fixes: 4926c7a52de7 ("selftest mm/mseal memory sealing") > Cc: Jeff Xu <jeffxu@chromium.org> > Cc: David Hildenbrand <david@redhat.com> > Signed-off-by: John Hubbard <jhubbard@nvidia.com> > --- > tools/testing/selftests/mm/mseal_test.c | 2 +- > tools/testing/selftests/mm/seal_elf.c | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tools/testing/selftests/mm/mseal_test.c b/tools/testing/selftests/mm/mseal_test.c > index 41998cf1dcf5..58c888529f42 100644 > --- a/tools/testing/selftests/mm/mseal_test.c > +++ b/tools/testing/selftests/mm/mseal_test.c > @@ -3,7 +3,7 @@ > #include <linux/mman.h> > #include <sys/mman.h> > #include <stdint.h> > -#include <unistd.h> > +#include <asm-generic/unistd.h> > #include <string.h> > #include <sys/time.h> > #include <sys/resource.h> > diff --git a/tools/testing/selftests/mm/seal_elf.c b/tools/testing/selftests/mm/seal_elf.c > index f2babec79bb6..27bf2f84231d 100644 > --- a/tools/testing/selftests/mm/seal_elf.c > +++ b/tools/testing/selftests/mm/seal_elf.c > @@ -2,7 +2,7 @@ > #define _GNU_SOURCE > #include <sys/mman.h> > #include <stdint.h> > -#include <unistd.h> > +#include <asm-generic/unistd.h> > #include <string.h> > #include <sys/time.h> > #include <sys/resource.h> Still confused. Let's take a look at "microblaze". arch/microblaze/include/asm/unistd.h -> #include <uapi/asm/unistd.h> arch/microblaze/include/uapi/asm/unistd.h -> #include <asm/unistd_32.h> -> Generated during "make headers" usr/include/asm/unistd_32.h is generated via arch/microblaze/kernel/syscalls/Makefile with the syshdr command. So we never end up including asm-generic/unistd.h directly on microblaze, but rather converts it (IIUC) to something else. That will work as expected here?
On 6/17/24 11:56 PM, David Hildenbrand wrote: > On 18.06.24 04:24, John Hubbard wrote: ... >> diff --git a/tools/testing/selftests/mm/seal_elf.c b/tools/testing/selftests/mm/seal_elf.c >> index f2babec79bb6..27bf2f84231d 100644 >> --- a/tools/testing/selftests/mm/seal_elf.c >> +++ b/tools/testing/selftests/mm/seal_elf.c >> @@ -2,7 +2,7 @@ >> #define _GNU_SOURCE >> #include <sys/mman.h> >> #include <stdint.h> >> -#include <unistd.h> >> +#include <asm-generic/unistd.h> >> #include <string.h> >> #include <sys/time.h> >> #include <sys/resource.h> > > Still confused. Let's take a look at "microblaze". > > arch/microblaze/include/asm/unistd.h > -> #include <uapi/asm/unistd.h> > > arch/microblaze/include/uapi/asm/unistd.h > -> #include <asm/unistd_32.h> > -> Generated during "make headers" > > usr/include/asm/unistd_32.h is generated via > arch/microblaze/kernel/syscalls/Makefile with the syshdr command. > > So we never end up including asm-generic/unistd.h directly on microblaze, but rather converts it (IIUC) to something else. > Yes. > That will work as expected here? > No. :) The problem, and the source of confusion here, is that for most user space programs, the header file inclusion behaves as you've mentioned above. However, those programs are installed on a single computer that has a single set of asm and kernel headers installed. We are quite special here, because we are building a set of user space programs that: a) Mostly avoids using the installed (distro) system header files. b) Must build (and run) on all supported CPU architectures c) Must occasionally use symbols that have so new that they have not yet been included in the distro's header files. Doing (a) creates a new problem: how to get a set of cross-platform headers that works in all cases. Fortunately, asm-generic headers solve that one. Which is why we need to use them here. The reason this hasn't really come up yet, is that until now, the kselftests requirement (which I'm trying to remove) was that "make headers" must first be run. That allowed the selftests to get a snapshot of sufficiently new header files that looked just like (and conflict with) the installed system headers. I can update the commit description with some of the above, if it helps. thanks,
On 18.06.24 22:14, John Hubbard wrote: > On 6/17/24 11:56 PM, David Hildenbrand wrote: >> On 18.06.24 04:24, John Hubbard wrote: > ... >>> diff --git a/tools/testing/selftests/mm/seal_elf.c b/tools/testing/selftests/mm/seal_elf.c >>> index f2babec79bb6..27bf2f84231d 100644 >>> --- a/tools/testing/selftests/mm/seal_elf.c >>> +++ b/tools/testing/selftests/mm/seal_elf.c >>> @@ -2,7 +2,7 @@ >>> #define _GNU_SOURCE >>> #include <sys/mman.h> >>> #include <stdint.h> >>> -#include <unistd.h> >>> +#include <asm-generic/unistd.h> >>> #include <string.h> >>> #include <sys/time.h> >>> #include <sys/resource.h> >> >> Still confused. Let's take a look at "microblaze". >> >> arch/microblaze/include/asm/unistd.h >> -> #include <uapi/asm/unistd.h> >> >> arch/microblaze/include/uapi/asm/unistd.h >> -> #include <asm/unistd_32.h> >> -> Generated during "make headers" >> >> usr/include/asm/unistd_32.h is generated via >> arch/microblaze/kernel/syscalls/Makefile with the syshdr command. >> >> So we never end up including asm-generic/unistd.h directly on microblaze, but rather converts it (IIUC) to something else. >> > > Yes. > >> That will work as expected here? >> > > No. :) > > The problem, and the source of confusion here, is that for most user > space programs, the header file inclusion behaves as you've mentioned > above. However, those programs are installed on a single computer that > has a single set of asm and kernel headers installed. > > We are quite special here, because we are building a set of user space > programs that: > > a) Mostly avoids using the installed (distro) system header files. > > b) Must build (and run) on all supported CPU architectures > > c) Must occasionally use symbols that have so new that they have not > yet been included in the distro's header files. > > Doing (a) creates a new problem: how to get a set of cross-platform > headers that works in all cases. > > Fortunately, asm-generic headers solve that one. Which is why we need to > use them here. > > The reason this hasn't really come up yet, is that until now, the > kselftests requirement (which I'm trying to remove) was that "make > headers" must first be run. That allowed the selftests to get a snapshot > of sufficiently new header files that looked just like (and conflict > with) the installed system headers. > > I can update the commit description with some of the above, if it helps. I think it will. The main concern I had was that we could be ending up including headers with *wrong* data. As long as (a) it compiles where it's supposed to compile (b) it runs where it's supposed to run, we're good :)
On 6/18/24 1:54 PM, David Hildenbrand wrote: > On 18.06.24 22:14, John Hubbard wrote: >> On 6/17/24 11:56 PM, David Hildenbrand wrote: >>> On 18.06.24 04:24, John Hubbard wrote: >> ... ... >> I can update the commit description with some of the above, if it helps. > > I think it will. The main concern I had was that we could be ending up including headers with *wrong* data. As long as (a) it compiles where it's supposed to compile (b) it runs where it's supposed to run, we're good :) > OK, I've drafted an updated commit description (below), and in order to reduce email churn perhaps it's best for me to hold onto it for a day or two, while we see how v3 fares in linux-next. (Thanks, Andrew, for patching that up with my Makefile fix.) Here's the draft: selftests/mm: mseal, self_elf: fix missing __NR_mseal The selftests/mm build isn't exactly "broken", according to the current documentation, which still claims that one must run "make headers", before building the kselftests. However, according to the new plan to get rid of that requirement [1], they are future-broken: attempting to build selftests/mm *without* first running "make headers" will fail due to not finding __NR_mseal. Therefore, include asm-generic/unistd.h, which has all of the system call numbers that are needed, abstracted across the various CPU arches. Some explanation in support of this "asm-generic" approach: For most user space programs, the header file inclusion behaves as per this microblaze example, which comes from David Hildenbrand (thanks!) : arch/microblaze/include/asm/unistd.h -> #include <uapi/asm/unistd.h> arch/microblaze/include/uapi/asm/unistd.h -> #include <asm/unistd_32.h> -> Generated during "make headers" usr/include/asm/unistd_32.h is generated via arch/microblaze/kernel/syscalls/Makefile with the syshdr command. So we never end up including asm-generic/unistd.h directly on microblaze... [2] However, those programs are installed on a single computer that has a single set of asm and kernel headers installed. In contrast, the kselftests are quite special, because they must provide a set of user space programs that: a) Mostly avoid using the installed (distro) system header files. b) Build (and run) on all supported CPU architectures c) Occasionally use symbols that have so new that they have not yet been included in the distro's header files. Doing (a) creates a new problem: how to get a set of cross-platform headers that works in all cases. Fortunately, asm-generic headers solve that one. Which is why we need to use them here--at least, for particularly difficult headers such as unistd.h. The reason this hasn't really come up yet, is that until now, the kselftests requirement (which I'm trying to eventually remove) was that "make headers" must first be run. That allowed the selftests to get a snapshot of sufficiently new header files that looked just like (and conflict with) the installed system headers. And as an aside, this is also an improvement over past practices of simply open-coding in a single (not per-arch) definition of a new symbol, directly into the selftest code. [1] commit e076eaca5906 ("selftests: break the dependency upon local header files") [2] https://lore.kernel.org/all/0b152bea-ccb6-403e-9c57-08ed5e828135@redhat.com/ Fixes: 4926c7a52de7 ("selftest mm/mseal memory sealing") Cc: Jeff Xu <jeffxu@chromium.org> Cc: David Hildenbrand <david@redhat.com> Signed-off-by: John Hubbard <jhubbard@nvidia.com> thanks,
On Tue, 18 Jun 2024 14:29:32 -0700 John Hubbard <jhubbard@nvidia.com> wrote:
> OK, I've drafted an updated commit description (below)
<copy><paste>
On 6/18/24 2:53 PM, Andrew Morton wrote: > On Tue, 18 Jun 2024 14:29:32 -0700 John Hubbard <jhubbard@nvidia.com> wrote: > >> OK, I've drafted an updated commit description (below) > > <copy><paste> Awesome! :) thanks,
On 18.06.24 23:29, John Hubbard wrote: > On 6/18/24 1:54 PM, David Hildenbrand wrote: >> On 18.06.24 22:14, John Hubbard wrote: >>> On 6/17/24 11:56 PM, David Hildenbrand wrote: >>>> On 18.06.24 04:24, John Hubbard wrote: >>> ... > ... >>> I can update the commit description with some of the above, if it helps. >> >> I think it will. The main concern I had was that we could be ending up including headers with *wrong* data. As long as (a) it compiles where it's supposed to compile (b) it runs where it's supposed to run, we're good :) >> > > OK, I've drafted an updated commit description (below), and in order > to reduce email churn perhaps it's best for me to hold onto it for a > day or two, while we see how v3 fares in linux-next. (Thanks, Andrew, > for patching that up with my Makefile fix.) > > Here's the draft: > selftests/mm: mseal, self_elf: fix missing __NR_mseal > > The selftests/mm build isn't exactly "broken", according to the current > documentation, which still claims that one must run "make headers", > before building the kselftests. However, according to the new plan to > get rid of that requirement [1], they are future-broken: attempting to > build selftests/mm *without* first running "make headers" will fail due > to not finding __NR_mseal. > > Therefore, include asm-generic/unistd.h, which has all of the system > call numbers that are needed, abstracted across the various CPU arches. > > Some explanation in support of this "asm-generic" approach: > > For most user space programs, the header file inclusion behaves as > per this microblaze example, which comes from David Hildenbrand > (thanks!) : > > arch/microblaze/include/asm/unistd.h > -> #include <uapi/asm/unistd.h> > > arch/microblaze/include/uapi/asm/unistd.h > -> #include <asm/unistd_32.h> > -> Generated during "make headers" > > usr/include/asm/unistd_32.h is generated via > arch/microblaze/kernel/syscalls/Makefile with the syshdr command. > > So we never end up including asm-generic/unistd.h directly on > microblaze... [2] > > However, those programs are installed on a single computer that has a > single set of asm and kernel headers installed. > > In contrast, the kselftests are quite special, because they must provide > a set of user space programs that: > > a) Mostly avoid using the installed (distro) system header files. > > b) Build (and run) on all supported CPU architectures > > c) Occasionally use symbols that have so new that they have not > yet been included in the distro's header files. > > Doing (a) creates a new problem: how to get a set of cross-platform > headers that works in all cases. > > Fortunately, asm-generic headers solve that one. Which is why we need to > use them here--at least, for particularly difficult headers such as > unistd.h. > > The reason this hasn't really come up yet, is that until now, the > kselftests requirement (which I'm trying to eventually remove) was that > "make headers" must first be run. That allowed the selftests to get a > snapshot of sufficiently new header files that looked just like (and > conflict with) the installed system headers. > > And as an aside, this is also an improvement over past practices of > simply open-coding in a single (not per-arch) definition of a new > symbol, directly into the selftest code. > > [1] commit e076eaca5906 ("selftests: break the dependency upon local > header files") > > [2] https://lore.kernel.org/all/0b152bea-ccb6-403e-9c57-08ed5e828135@redhat.com/ > > Fixes: 4926c7a52de7 ("selftest mm/mseal memory sealing") > Cc: Jeff Xu <jeffxu@chromium.org> > Cc: David Hildenbrand <david@redhat.com> > Signed-off-by: John Hubbard <jhubbard@nvidia.com> Thanks! Acked-by: David Hildenbrand <david@redhat.com>
diff --git a/tools/testing/selftests/mm/mseal_test.c b/tools/testing/selftests/mm/mseal_test.c index 41998cf1dcf5..58c888529f42 100644 --- a/tools/testing/selftests/mm/mseal_test.c +++ b/tools/testing/selftests/mm/mseal_test.c @@ -3,7 +3,7 @@ #include <linux/mman.h> #include <sys/mman.h> #include <stdint.h> -#include <unistd.h> +#include <asm-generic/unistd.h> #include <string.h> #include <sys/time.h> #include <sys/resource.h> diff --git a/tools/testing/selftests/mm/seal_elf.c b/tools/testing/selftests/mm/seal_elf.c index f2babec79bb6..27bf2f84231d 100644 --- a/tools/testing/selftests/mm/seal_elf.c +++ b/tools/testing/selftests/mm/seal_elf.c @@ -2,7 +2,7 @@ #define _GNU_SOURCE #include <sys/mman.h> #include <stdint.h> -#include <unistd.h> +#include <asm-generic/unistd.h> #include <string.h> #include <sys/time.h> #include <sys/resource.h>
The selftests/mm build isn't exactly "broken", according to the current documentation, which still claims that one must run "make headers", before building the kselftests. However, according to the new plan to get rid of that requirement [1], they are future-broken: attempting to build selftests/mm *without* first running "make headers" will fail due to not finding __NR_mseal. Therefore, include asm-generic/unistd.h, which has all of the system call numbers that are needed, abstracted across the various CPU arches. [1] commit e076eaca5906 ("selftests: break the dependency upon local header files") Fixes: 4926c7a52de7 ("selftest mm/mseal memory sealing") Cc: Jeff Xu <jeffxu@chromium.org> Cc: David Hildenbrand <david@redhat.com> Signed-off-by: John Hubbard <jhubbard@nvidia.com> --- tools/testing/selftests/mm/mseal_test.c | 2 +- tools/testing/selftests/mm/seal_elf.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)