| Message ID | 3132727fea08e81e834104761b5a5630d337340a.1725636560.git.sam@gentoo.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | fixdep: handle short reads in read_file | expand |
On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > 50% or so of kernel builds within our package manager fail for me with > 'fixdep: read: success' because read(), for some reason - possibly ptrace, > only read a short amount, not the full size. > > Unfortunately, this didn't trigger a -Wunused-result warning because > we _are_ checking the return value, but with a bad comparison (it's completely > fine for read() to not read the whole file in one gulp). > > Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file to buffer") I guess, another approach would be to use fread() instead of read(). Does the attached diff fix the issue too? > Signed-off-by: Sam James <sam@gentoo.org> > --- > scripts/basic/fixdep.c | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) > > diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c > index 84b6efa849f4d..04d7742c99ac2 100644 > --- a/scripts/basic/fixdep.c > +++ b/scripts/basic/fixdep.c > @@ -233,9 +233,15 @@ static void *read_file(const char *filename) > perror("fixdep: malloc"); > exit(2); > } > - if (read(fd, buf, st.st_size) != st.st_size) { > - perror("fixdep: read"); > - exit(2); > + ssize_t bytes = 0; > + while (bytes < st.st_size) { > + ssize_t cur = read(fd, buf + bytes, st.st_size - bytes); > + if (cur == -1) { > + perror("fixdep: read"); > + exit(2); > + } else { > + bytes += cur; > + } > } > buf[st.st_size] = '\0'; > close(fd); > -- > 2.46.0 >
Masahiro Yamada <masahiroy@kernel.org> writes: > On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: Hi Masahiro, >> >> 50% or so of kernel builds within our package manager fail for me with >> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >> only read a short amount, not the full size. >> >> Unfortunately, this didn't trigger a -Wunused-result warning because >> we _are_ checking the return value, but with a bad comparison (it's completely >> fine for read() to not read the whole file in one gulp). >> >> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e > > > Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file > to buffer") > Ah, thanks. I'll fix that and send v2 depending on how we decide to move forward wrt below. > > I guess, another approach would be to use fread() instead of read(). > > Does the attached diff fix the issue too? > > Unfortunately no. It failed for me in the same way as before :( The man page mentions: > On success, fread() and fwrite() return the number of items read or > written. This number equals the number of bytes transferred only when size is 1. so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well which says: > If a partial element is read, its value is unspecified. and > The fread() function shall return the number of elements successfully > read, which shall be less than nitems only if an error or end-of-file > is encountered, or size is zero. The error reference is kind of mysterious there though. It kind of looks like fread *should* work. I'll send this mail and then think about it a bit later and ask around to see if I'm missing something obvious? > [...] thanks, sam
Sam James <sam@gentoo.org> writes: > Masahiro Yamada <masahiroy@kernel.org> writes: > >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > Hi Masahiro, > >>> >>> 50% or so of kernel builds within our package manager fail for me with >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >>> only read a short amount, not the full size. >>> >>> Unfortunately, this didn't trigger a -Wunused-result warning because >>> we _are_ checking the return value, but with a bad comparison (it's completely >>> fine for read() to not read the whole file in one gulp). >>> >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e >> >> >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file >> to buffer") >> > > Ah, thanks. I'll fix that and send v2 depending on how we decide to move > forward wrt below. > >> >> I guess, another approach would be to use fread() instead of read(). >> >> Does the attached diff fix the issue too? >> >> > > Unfortunately no. It failed for me in the same way as before :( > > The man page mentions: >> On success, fread() and fwrite() return the number of items read or >> written. This number equals the number of bytes transferred only when size is 1. > > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well > which says: >> If a partial element is read, its value is unspecified. > and >> The fread() function shall return the number of elements successfully >> read, which shall be less than nitems only if an error or end-of-file >> is encountered, or size is zero. > > The error reference is kind of mysterious there though. > > It kind of looks like fread *should* work. I'll send this mail and then > think about it a bit later and ask around to see if I'm missing > something obvious? OK, others disagree with my reading of fread and think it is ambiguous. With your patch, I was able to get failures albeit possibly less frequently. I'm trying my patch again in a loop now. > >> [...] > > thanks, > sam
On Sat, Sep 7, 2024 at 10:26 PM Sam James <sam@gentoo.org> wrote: > > Sam James <sam@gentoo.org> writes: > > > Masahiro Yamada <masahiroy@kernel.org> writes: > > > >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > > > Hi Masahiro, > > > >>> > >>> 50% or so of kernel builds within our package manager fail for me with > >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, > >>> only read a short amount, not the full size. > >>> > >>> Unfortunately, this didn't trigger a -Wunused-result warning because > >>> we _are_ checking the return value, but with a bad comparison (it's completely > >>> fine for read() to not read the whole file in one gulp). > >>> > >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e > >> > >> > >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file > >> to buffer") > >> > > > > Ah, thanks. I'll fix that and send v2 depending on how we decide to move > > forward wrt below. > > > >> > >> I guess, another approach would be to use fread() instead of read(). > >> > >> Does the attached diff fix the issue too? > >> > >> > > > > Unfortunately no. It failed for me in the same way as before :( > > > > The man page mentions: > >> On success, fread() and fwrite() return the number of items read or > >> written. This number equals the number of bytes transferred only when size is 1. > > > > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well > > which says: > >> If a partial element is read, its value is unspecified. > > and > >> The fread() function shall return the number of elements successfully > >> read, which shall be less than nitems only if an error or end-of-file > >> is encountered, or size is zero. > > > > The error reference is kind of mysterious there though. > > > > It kind of looks like fread *should* work. I'll send this mail and then > > think about it a bit later and ask around to see if I'm missing > > something obvious? > > OK, others disagree with my reading of fread and think it is ambiguous. > > With your patch, I was able to get failures albeit possibly less > frequently. I'm trying my patch again in a loop now. > > > > >> [...] > > > > thanks, > > sam > Your quotation of the POSIX fread() spec: (https://pubs.opengroup.org/onlinepubs/000095399/functions/fread.html) > If a partial element is read, its value is unspecified. and > The fread() function shall return the number of elements successfully > read, which shall be less than nitems only if an error or end-of-file > is encountered, or size is zero. I think this is clear enough. The end-of-file should not be encountered, as we check the file size in advance. So, some error might be happening.
Masahiro Yamada <masahiroy@kernel.org> writes: > On Sat, Sep 7, 2024 at 10:26 PM Sam James <sam@gentoo.org> wrote: >> >> Sam James <sam@gentoo.org> writes: >> >> > Masahiro Yamada <masahiroy@kernel.org> writes: >> > >> >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: >> > >> > Hi Masahiro, >> > >> >>> >> >>> 50% or so of kernel builds within our package manager fail for me with >> >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >> >>> only read a short amount, not the full size. >> >>> >> >>> Unfortunately, this didn't trigger a -Wunused-result warning because >> >>> we _are_ checking the return value, but with a bad comparison (it's completely >> >>> fine for read() to not read the whole file in one gulp). >> >>> >> >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e >> >> >> >> >> >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file >> >> to buffer") >> >> >> > >> > Ah, thanks. I'll fix that and send v2 depending on how we decide to move >> > forward wrt below. >> > >> >> >> >> I guess, another approach would be to use fread() instead of read(). >> >> >> >> Does the attached diff fix the issue too? >> >> >> >> >> > >> > Unfortunately no. It failed for me in the same way as before :( >> > >> > The man page mentions: >> >> On success, fread() and fwrite() return the number of items read or >> >> written. This number equals the number of bytes transferred only when size is 1. >> > >> > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well >> > which says: >> >> If a partial element is read, its value is unspecified. >> > and >> >> The fread() function shall return the number of elements successfully >> >> read, which shall be less than nitems only if an error or end-of-file >> >> is encountered, or size is zero. >> > >> > The error reference is kind of mysterious there though. >> > >> > It kind of looks like fread *should* work. I'll send this mail and then >> > think about it a bit later and ask around to see if I'm missing >> > something obvious? >> >> OK, others disagree with my reading of fread and think it is ambiguous. >> >> With your patch, I was able to get failures albeit possibly less >> frequently. I'm trying my patch again in a loop now. >> >> > >> >> [...] >> > >> > thanks, >> > sam >> > > > > > > > Your quotation of the POSIX fread() spec: > (https://pubs.opengroup.org/onlinepubs/000095399/functions/fread.html) > >> If a partial element is read, its value is unspecified. > and >> The fread() function shall return the number of elements successfully >> read, which shall be less than nitems only if an error or end-of-file >> is encountered, or size is zero. > > > I think this is clear enough. > > > The end-of-file should not be encountered, as we check the file > size in advance. I believe that it's referring to the number of records, not if you read *1* record of size N. I looked at the musl and glibc sources and neither seem to retry partial reads in that case. I don't see any error indicator set.
diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c index 84b6efa849f4d..04d7742c99ac2 100644 --- a/scripts/basic/fixdep.c +++ b/scripts/basic/fixdep.c @@ -233,9 +233,15 @@ static void *read_file(const char *filename) perror("fixdep: malloc"); exit(2); } - if (read(fd, buf, st.st_size) != st.st_size) { - perror("fixdep: read"); - exit(2); + ssize_t bytes = 0; + while (bytes < st.st_size) { + ssize_t cur = read(fd, buf + bytes, st.st_size - bytes); + if (cur == -1) { + perror("fixdep: read"); + exit(2); + } else { + bytes += cur; + } } buf[st.st_size] = '\0'; close(fd);
50% or so of kernel builds within our package manager fail for me with 'fixdep: read: success' because read(), for some reason - possibly ptrace, only read a short amount, not the full size. Unfortunately, this didn't trigger a -Wunused-result warning because we _are_ checking the return value, but with a bad comparison (it's completely fine for read() to not read the whole file in one gulp). Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e Signed-off-by: Sam James <sam@gentoo.org> --- scripts/basic/fixdep.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)