mbox series

[bpf-next,0/5] Support local vmtest for riscv64

Message ID 20240328124916.293173-1-pulehui@huaweicloud.com (mailing list archive)
Headers show
Series Support local vmtest for riscv64 | expand

Message

Pu Lehui March 28, 2024, 12:49 p.m. UTC 
Patch 1 is to enable cross platform testing for local vmtest. The
remaining patch adds local vmtest support for riscv64. It relies on
commit [0] [1] for better regression.

We can now perform cross platform testing for riscv64 bpf using the
following command:

PLATFORM=riscv64 CROSS_COMPILE=riscv64-linux-gnu- \
    tools/testing/selftests/bpf/vmtest.sh -- \
        ./test_progs -d \
            \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \
                | cut -d'#' -f1 \
                | sed -e 's/^[[:space:]]*//' \
                      -e 's/[[:space:]]*$//' \
                | tr -s '\n' ','\
            )\"

The test platform is x86_64 architecture, and the versions of relevant
components are as follows:
    QEMU: 8.2.0
    CLANG: 17.0.6 (align to BPF CI)
    OpenSBI: 1.3.1 (default by QEMU)
    ROOTFS: ubuntu jammy (generated by [2])

Link: https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/commit/?id=ea6873118493 [0]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=443574b033876c85 [1]
Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [2]

Pu Lehui (5):
  selftests/bpf: Enable cross platform testing for local vmtest
  riscv, bpf: Relax restrictions on Zbb instructions
  selftests/bpf: Add config.riscv64
  selftests/bpf: Add DENYLIST.riscv64
  selftests/bpf: Add riscv64 configurations to local vmtest

 arch/riscv/net/bpf_jit.h                     |  2 +-
 tools/testing/selftests/bpf/DENYLIST.riscv64 |  5 ++
 tools/testing/selftests/bpf/config.riscv64   | 85 ++++++++++++++++++++
 tools/testing/selftests/bpf/vmtest.sh        | 48 ++++++++---
 4 files changed, 127 insertions(+), 13 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/DENYLIST.riscv64
 create mode 100644 tools/testing/selftests/bpf/config.riscv64

Comments

Eduard Zingerman March 29, 2024, 9:08 a.m. UTC | #1 
On Thu, 2024-03-28 at 12:49 +0000, Pu Lehui wrote:
> Patch 1 is to enable cross platform testing for local vmtest. The
> remaining patch adds local vmtest support for riscv64. It relies on
> commit [0] [1] for better regression.
> 
> We can now perform cross platform testing for riscv64 bpf using the
> following command:
> 
> PLATFORM=riscv64 CROSS_COMPILE=riscv64-linux-gnu- \
>     tools/testing/selftests/bpf/vmtest.sh -- \
>         ./test_progs -d \
>             \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \
>                 | cut -d'#' -f1 \
>                 | sed -e 's/^[[:space:]]*//' \
>                       -e 's/[[:space:]]*$//' \
>                 | tr -s '\n' ','\
>             )\"
> 
> The test platform is x86_64 architecture, and the versions of relevant
> components are as follows:
>     QEMU: 8.2.0
>     CLANG: 17.0.6 (align to BPF CI)
>     OpenSBI: 1.3.1 (default by QEMU)
>     ROOTFS: ubuntu jammy (generated by [2])
> 
> Link: https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/commit/?id=ea6873118493 [0]
> Link: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=443574b033876c85 [1]
> Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [2]

Hello,

I wanted to do a test run for this patch-set but did not figure out
how to build rootfs for riscv64 system.

I modified mkrootfs_debian.sh as below, but build command fails:

$ ./rootfs/mkrootfs_debian.sh -d jammy -a riscv64 -m http://de.archive.ubuntu.com/ubuntu
...
E: Couldn't download http://de.archive.ubuntu.com/ubuntu/dists/jammy/main/binary-riscv64/Packages

Apparently jammy does not have binaries built for riscv64, or I'm failing to find correct mirror.
Could you please provide some instructions on how to prepare rootfs?

Thanks,
Eduard

--

diff --git a/rootfs/mkrootfs_debian.sh b/rootfs/mkrootfs_debian.sh
index dfe957e..1d5b769 100755
--- a/rootfs/mkrootfs_debian.sh
+++ b/rootfs/mkrootfs_debian.sh
@@ -16,6 +16,7 @@ CPUTABLE="${CPUTABLE:-/usr/share/dpkg/cputable}"
 
 deb_arch=$(dpkg --print-architecture)
 distro="bullseye"
+mirror=""
 
 function usage() {
     echo "Usage: $0 [-a | --arch architecture] [-h | --help]
@@ -25,6 +26,7 @@ By default build an image for the architecture of the host running the script.
 
     -a | --arch:    architecture to build the image for. Default (${deb_arch})
     -d | --distro:  distribution to build. Default (${distro})
+    -m | --mirror:  mirror for distribution to build. Default (${mirror})
 "
 }
 
@@ -44,7 +46,7 @@ function qemu_static() {
     # Given a Debian architecture find the location of the matching
     # qemu-${gnu_arch}-static binary.
     gnu_arch=$(debian_to_gnu "${1}")
-    echo "qemu-${gnu_arch}-static"
+    echo "qemu-${gnu_arch}"
 }
 
 function check_requirements() {
@@ -95,7 +97,7 @@ function check_requirements() {
     fi
 }
 
-TEMP=$(getopt  -l "arch:,distro:,help" -o "a:d:h" -- "$@")
+TEMP=$(getopt  -l "arch:,distro:,mirror:,help" -o "a:d:m:h" -- "$@")
 if [ $? -ne 0 ]; then
     usage
 fi
@@ -113,6 +115,10 @@ while true; do
             distro="$2"
             shift 2
             ;;
+        --mirror | -m)
+            mirror="$2"
+            shift 2
+            ;;
         --help | -h)
             usage
             exit
@@ -162,7 +168,8 @@ debootstrap --include="$packages" \
     --arch="${deb_arch}" \
     "$@" \
     "${distro}" \
-    "$root"
+    "$root" \
+    "${mirror}"
 
 qemu=$(which $(qemu_static ${deb_arch}))
Pu Lehui March 29, 2024, 10:10 a.m. UTC | #2 
On 2024/3/29 17:08, Eduard Zingerman wrote:
> On Thu, 2024-03-28 at 12:49 +0000, Pu Lehui wrote:
>> Patch 1 is to enable cross platform testing for local vmtest. The
>> remaining patch adds local vmtest support for riscv64. It relies on
>> commit [0] [1] for better regression.
>>
>> We can now perform cross platform testing for riscv64 bpf using the
>> following command:
>>
>> PLATFORM=riscv64 CROSS_COMPILE=riscv64-linux-gnu- \
>>      tools/testing/selftests/bpf/vmtest.sh -- \
>>          ./test_progs -d \
>>              \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \
>>                  | cut -d'#' -f1 \
>>                  | sed -e 's/^[[:space:]]*//' \
>>                        -e 's/[[:space:]]*$//' \
>>                  | tr -s '\n' ','\
>>              )\"
>>
>> The test platform is x86_64 architecture, and the versions of relevant
>> components are as follows:
>>      QEMU: 8.2.0
>>      CLANG: 17.0.6 (align to BPF CI)
>>      OpenSBI: 1.3.1 (default by QEMU)
>>      ROOTFS: ubuntu jammy (generated by [2])
>>
>> Link: https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/commit/?id=ea6873118493 [0]
>> Link: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=443574b033876c85 [1]
>> Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [2]
> 
> Hello,
> 
> I wanted to do a test run for this patch-set but did not figure out
> how to build rootfs for riscv64 system.
> 
> I modified mkrootfs_debian.sh as below, but build command fails:
> 
> $ ./rootfs/mkrootfs_debian.sh -d jammy -a riscv64 -m http://de.archive.ubuntu.com/ubuntu
> ...
> E: Couldn't download http://de.archive.ubuntu.com/ubuntu/dists/jammy/main/binary-riscv64/Packages
> 
> Apparently jammy does not have binaries built for riscv64, or I'm failing to find correct mirror.
> Could you please provide some instructions on how to prepare rootfs?

Hi Eduard, We need the mirror repository of ubuntu-ports, you could try 
http://de.ports.ubuntu.com/.

> 
> Thanks,
> Eduard
> 
> --
> 
> diff --git a/rootfs/mkrootfs_debian.sh b/rootfs/mkrootfs_debian.sh
> index dfe957e..1d5b769 100755
> --- a/rootfs/mkrootfs_debian.sh
> +++ b/rootfs/mkrootfs_debian.sh
> @@ -16,6 +16,7 @@ CPUTABLE="${CPUTABLE:-/usr/share/dpkg/cputable}"
>   
>   deb_arch=$(dpkg --print-architecture)
>   distro="bullseye"
> +mirror=""
>   
>   function usage() {
>       echo "Usage: $0 [-a | --arch architecture] [-h | --help]
> @@ -25,6 +26,7 @@ By default build an image for the architecture of the host running the script.
>   
>       -a | --arch:    architecture to build the image for. Default (${deb_arch})
>       -d | --distro:  distribution to build. Default (${distro})
> +    -m | --mirror:  mirror for distribution to build. Default (${mirror})
>   "
>   }
>   
> @@ -44,7 +46,7 @@ function qemu_static() {
>       # Given a Debian architecture find the location of the matching
>       # qemu-${gnu_arch}-static binary.
>       gnu_arch=$(debian_to_gnu "${1}")
> -    echo "qemu-${gnu_arch}-static"
> +    echo "qemu-${gnu_arch}"
>   }
>   
>   function check_requirements() {
> @@ -95,7 +97,7 @@ function check_requirements() {
>       fi
>   }
>   
> -TEMP=$(getopt  -l "arch:,distro:,help" -o "a:d:h" -- "$@")
> +TEMP=$(getopt  -l "arch:,distro:,mirror:,help" -o "a:d:m:h" -- "$@")
>   if [ $? -ne 0 ]; then
>       usage
>   fi
> @@ -113,6 +115,10 @@ while true; do
>               distro="$2"
>               shift 2
>               ;;
> +        --mirror | -m)
> +            mirror="$2"
> +            shift 2
> +            ;;
>           --help | -h)
>               usage
>               exit
> @@ -162,7 +168,8 @@ debootstrap --include="$packages" \
>       --arch="${deb_arch}" \
>       "$@" \
>       "${distro}" \
> -    "$root"
> +    "$root" \
> +    "${mirror}"
>   
>   qemu=$(which $(qemu_static ${deb_arch}))
>
Eduard Zingerman March 29, 2024, 7:46 p.m. UTC | #3 
On Fri, 2024-03-29 at 18:10 +0800, Pu Lehui wrote:
[...]

> > Apparently jammy does not have binaries built for riscv64, or I'm failing to find correct mirror.
> > Could you please provide some instructions on how to prepare rootfs?
> 
> Hi Eduard, We need the mirror repository of ubuntu-ports, you could try 
> http://de.ports.ubuntu.com/.

Hi Pu, thank you this mirrorm it works.

Unfortunately my local setup is still not good enough.
I've installed cross-riscv64-gcc14 but it seems that a few more
libraries are necessary, as I get the following compilation errors:

  $ PLATFORM=riscv64 CROSS_COMPILE=riscv64-suse-linux- ./vmtest.sh -- ./test_verifier
  ... kernel compiles ok ...
  ../../../../scripts/sign-file.c:25:10: fatal error: openssl/opensslv.h: No such file or directory
     25 | #include <openssl/opensslv.h>
        |          ^~~~~~~~~~~~~~~~~~~~
  compilation terminated.
    CC      /home/eddy/work/bpf-next/tools/testing/selftests/bpf/tools/build/host/libbpf/sharedobjs/bpf.o
  In file included from nlattr.c:14:
  libbpf_internal.h:19:10: fatal error: libelf.h: No such file or directory
     19 | #include <libelf.h>
  ...

Looks like I won't be able to test this patch-set, unless you have
some writeup on how to create a riscv64 dev environment at hand.
Sorry for the noise.
Pu Lehui March 30, 2024, 10:12 a.m. UTC | #4 
On 2024/3/30 3:46, Eduard Zingerman wrote:
> On Fri, 2024-03-29 at 18:10 +0800, Pu Lehui wrote:
> [...]
> 
>>> Apparently jammy does not have binaries built for riscv64, or I'm failing to find correct mirror.
>>> Could you please provide some instructions on how to prepare rootfs?
>>
>> Hi Eduard, We need the mirror repository of ubuntu-ports, you could try
>> http://de.ports.ubuntu.com/.
> 
> Hi Pu, thank you this mirrorm it works.
> 
> Unfortunately my local setup is still not good enough.
> I've installed cross-riscv64-gcc14 but it seems that a few more
> libraries are necessary, as I get the following compilation errors: >
>    $ PLATFORM=riscv64 CROSS_COMPILE=riscv64-suse-linux- ./vmtest.sh -- ./test_verifier
>    ... kernel compiles ok ...
>    ../../../../scripts/sign-file.c:25:10: fatal error: openssl/opensslv.h: No such file or directory
>       25 | #include <openssl/opensslv.h>
>          |          ^~~~~~~~~~~~~~~~~~~~
>    compilation terminated.
>      CC      /home/eddy/work/bpf-next/tools/testing/selftests/bpf/tools/build/host/libbpf/sharedobjs/bpf.o
>    In file included from nlattr.c:14:
>    libbpf_internal.h:19:10: fatal error: libelf.h: No such file or directory
>       19 | #include <libelf.h>
>    ...
> 
> Looks like I won't be able to test this patch-set, unless you have
> some writeup on how to create a riscv64 dev environment at hand.
> Sorry for the noise

Yeah, environmental issues are indeed a developer's nightmare. I will 
try to do something for the newcomers of riscv64 bpf. At present, I have 
simply built a docker local vmtest environment [0] based on Bjorn's 
riscv-cross-builder. We can directly run vmtest within this environment. 
Hopefully it will help.

Link: https://github.com/pulehui/riscv-cross-builder/tree/vmtest [0]

PS: Since the current rootfs of riscv64 is not in the INDEX, I simply 
modified vmtest.sh to support local rootfs. And we can use it by:
```
PLATFORM=riscv64 CROSS_COMPILE=riscv64-linux-gnu- \
     tools/testing/selftests/bpf/vmtest.sh -l /rootfs -- \
         ./test_progs -d \
             \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \
                 | cut -d'#' -f1 \
                 | sed -e 's/^[[:space:]]*//' \
                       -e 's/[[:space:]]*$//' \
                 | tr -s '\n' ','\
             )\"
```

diff --git a/tools/testing/selftests/bpf/vmtest.sh 
b/tools/testing/selftests/bpf/vmtest.sh
index f6889de9b498..17aff708c416 100755
--- a/tools/testing/selftests/bpf/vmtest.sh
+++ b/tools/testing/selftests/bpf/vmtest.sh
@@ -148,6 +148,21 @@ download_rootfs()
  		zstd -d | sudo tar -C "$dir" -x
  }

+load_rootfs()
+{
+	local image_dir="$1"
+	local rootfsversion="$2"
+	local dir="$3"
+
+	if ! which zstd &> /dev/null; then
+		echo 'Could not find "zstd" on the system, please install zstd'
+		exit 1
+	fi
+
+	cat "${image_dir}/libbpf-vmtest-rootfs-$rootfsversion.tar.zst" |
+		zstd -d | sudo tar -C "$dir" -x
+}
+
  recompile_kernel()
  {
  	local kernel_checkout="$1"
@@ -234,6 +249,7 @@ EOF

  create_vm_image()
  {
+	local local_image_dir="$1"
  	local rootfs_img="${OUTPUT_DIR}/${ROOTFS_IMAGE}"
  	local mount_dir="${OUTPUT_DIR}/${MOUNT_DIR}"

@@ -245,7 +261,11 @@ create_vm_image()
  	mkfs.ext4 -q "${rootfs_img}"

  	mount_image
-	download_rootfs "$(newest_rootfs_version)" "${mount_dir}"
+	if [[ "${local_image_dir}" == "" ]]; then
+		download_rootfs "$(newest_rootfs_version)" "${mount_dir}"
+	else
+		load_rootfs "${local_image_dir}" "$(newest_rootfs_version)" 
"${mount_dir}"
+	fi
  	unmount_image
  }

@@ -363,12 +383,16 @@ main()
  	local update_image="no"
  	local exit_command="poweroff -f"
  	local debug_shell="no"
+	local local_image_dir=""

-	while getopts ':hskid:j:' opt; do
+	while getopts ':hskil:d:j:' opt; do
  		case ${opt} in
  		i)
  			update_image="yes"
  			;;
+		l)
+			local_image_dir="$OPTARG"
+			;;
  		d)
  			OUTPUT_DIR="$OPTARG"
  			;;
@@ -445,7 +469,7 @@ main()
  	fi

  	if [[ "${update_image}" == "yes" ]]; then
-		create_vm_image
+		create_vm_image "${local_image_dir}"
  	fi

  	update_selftests "${kernel_checkout}" "${make_command}"
Eduard Zingerman April 2, 2024, 11:40 p.m. UTC | #5 
On Sat, 2024-03-30 at 18:12 +0800, Pu Lehui wrote:
[...]

> > Looks like I won't be able to test this patch-set, unless you have
> > some writeup on how to create a riscv64 dev environment at hand.
> > Sorry for the noise
> 
> Yeah, environmental issues are indeed a developer's nightmare. I will 
> try to do something for the newcomers of riscv64 bpf. At present, I have 
> simply built a docker local vmtest environment [0] based on Bjorn's 
> riscv-cross-builder. We can directly run vmtest within this environment. 
> Hopefully it will help.
> 
> Link: https://github.com/pulehui/riscv-cross-builder/tree/vmtest [0]

Hi Pu,

Thank you for sharing the docker file, I've managed to run the tests
using it. In order to avoid creating files with root permissions I had
to add the following lines at the end of the Dockerfile:

+ RUN useradd --no-create-home --uid 1000 eddy
+ RUN passwd -d eddy
+ RUN echo 'eddy ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
+ # vmtest.sh does 'mount -o loop',
+ # ensure there is a loop device in the container
+ RUN mknod /dev/loop0 b 7 20

Where 'eddy' is my local user with UID 1000.
Probably this should be made more generic.
I used the following command to start the container:

docker run -ti -u 1000:1000 \
    --rm -v <path-to-kernel-dir>:/workspace \
    -v <path-to-rootfs-image-dir>:/rootfs \
    --privileged ubuntu-vmtest:latest /bin/bash

Also, I had to add '-d /rootfs/bpf_selftests' option for vmtest.sh in
order to avoid polluting user directory inside the container.
Maybe OUTPUT_DIR for vmtest.sh should be mounted as a separate volume.

I agree with Daniel, it would be great to document all of this
somewhere in the repo (or even scripted somehow).

Using the specified DENYLIST I get the following stats for test_progs:

  #3/2     arena_htab/arena_htab_asm:FAIL
  #3       arena_htab:FAIL
  #95      get_branch_snapshot:FAIL
  #172/1   perf_branches/perf_branches_hw:FAIL
  #172     perf_branches:FAIL
  #434/3   verifier_arena/basic_alloc3:FAIL
  #434     verifier_arena:FAIL
  Summary: 531/3581 PASSED, 64 SKIPPED, 4 FAILED

Tested-by: Eduard Zingerman <eddyz87@gmail.com>

> PS: Since the current rootfs of riscv64 is not in the INDEX, I simply 
> modified vmtest.sh to support local rootfs.

Could you please add this change to the patch-set?

[...]
Pu Lehui April 3, 2024, 10:31 a.m. UTC | #6 
On 2024/4/3 7:40, Eduard Zingerman wrote:
> On Sat, 2024-03-30 at 18:12 +0800, Pu Lehui wrote:
> [...]
> 
>>> Looks like I won't be able to test this patch-set, unless you have
>>> some writeup on how to create a riscv64 dev environment at hand.
>>> Sorry for the noise
>>
>> Yeah, environmental issues are indeed a developer's nightmare. I will
>> try to do something for the newcomers of riscv64 bpf. At present, I have
>> simply built a docker local vmtest environment [0] based on Bjorn's
>> riscv-cross-builder. We can directly run vmtest within this environment.
>> Hopefully it will help.
>>
>> Link: https://github.com/pulehui/riscv-cross-builder/tree/vmtest [0]
> 
> Hi Pu,
> 
> Thank you for sharing the docker file, I've managed to run the tests
> using it. In order to avoid creating files with root permissions I had
> to add the following lines at the end of the Dockerfile:
> 
> + RUN useradd --no-create-home --uid 1000 eddy
> + RUN passwd -d eddy
> + RUN echo 'eddy ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
> + # vmtest.sh does 'mount -o loop',
> + # ensure there is a loop device in the container
> + RUN mknod /dev/loop0 b 7 20
> 
> Where 'eddy' is my local user with UID 1000.
> Probably this should be made more generic.
> I used the following command to start the container:
> 
> docker run -ti -u 1000:1000 \
>      --rm -v <path-to-kernel-dir>:/workspace \
>      -v <path-to-rootfs-image-dir>:/rootfs \
>      --privileged ubuntu-vmtest:latest /bin/bash
> 
> Also, I had to add '-d /rootfs/bpf_selftests' option for vmtest.sh in
> order to avoid polluting user directory inside the container.
> Maybe OUTPUT_DIR for vmtest.sh should be mounted as a separate volume.
> 
> I agree with Daniel, it would be great to document all of this

Forgot to reply to this in my last email. It my pleasure to do this.

> somewhere in the repo (or even scripted somehow).
> 
> Using the specified DENYLIST I get the following stats for test_progs:
> 
>    #3/2     arena_htab/arena_htab_asm:FAIL
>    #3       arena_htab:FAIL

Puranjay has submitted to riscv bpf arena and will be merged soon. So I 
didn't add it to DENYLIST.riscv64.

https://lore.kernel.org/bpf/20240326224943.86912-1-puranjay12@gmail.com/

>    #95      get_branch_snapshot:FAIL
>    #172/1   perf_branches/perf_branches_hw:FAIL
>    #172     perf_branches:FAIL

riscv sbi pmu driver not support branch sampling yet.  The following 
patch should be used for better regression.

https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/commit/?id=ea6873118493

>    #434/3   verifier_arena/basic_alloc3:FAIL
>    #434     verifier_arena:FAIL
>    Summary: 531/3581 PASSED, 64 SKIPPED, 4 FAILED
> 
> Tested-by: Eduard Zingerman <eddyz87@gmail.com>
> 
>> PS: Since the current rootfs of riscv64 is not in the INDEX, I simply
>> modified vmtest.sh to support local rootfs.
> 
> Could you please add this change to the patch-set?

yep, will try to make it more convenient.

> 
> [...]