| Message ID | 20240711215244.19237-2-yichen.wang@bytedance.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | WIP: Use Intel DSA accelerator to offload zero page checking in multifd live migration. | expand |
> -----Original Message----- > From: Yichen Wang <yichen.wang@bytedance.com> > Sent: Friday, July 12, 2024 5:53 AM > To: Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau > <marcandre.lureau@redhat.com>; Daniel P. Berrangé <berrange@redhat.com>; > Thomas Huth <thuth@redhat.com>; Philippe Mathieu-Daudé > <philmd@linaro.org>; Peter Xu <peterx@redhat.com>; Fabiano Rosas > <farosas@suse.de>; Eric Blake <eblake@redhat.com>; Markus Armbruster > <armbru@redhat.com>; Michael S. Tsirkin <mst@redhat.com>; Cornelia Huck > <cohuck@redhat.com>; qemu-devel@nongnu.org > Cc: Hao Xiang <hao.xiang@linux.dev>; Liu, Yuan1 <yuan1.liu@intel.com>; > Kumar, Shivam <shivam.kumar1@nutanix.com>; Ho-Ren (Jack) Chuang > <horenchuang@bytedance.com>; Wang, Yichen <yichen.wang@bytedance.com> > Subject: [PATCH v5 01/13] meson: Introduce new instruction set enqcmd to > the build system. > > From: Hao Xiang <hao.xiang@linux.dev> > > Enable instruction set enqcmd in build. > > Signed-off-by: Hao Xiang <hao.xiang@linux.dev> > Signed-off-by: Yichen Wang <yichen.wang@bytedance.com> > --- > meson.build | 14 ++++++++++++++ > meson_options.txt | 2 ++ > scripts/meson-buildoptions.sh | 3 +++ > 3 files changed, 19 insertions(+) > > diff --git a/meson.build b/meson.build > index 6a93da48e1..af650cfabf 100644 > --- a/meson.build > +++ b/meson.build > @@ -2893,6 +2893,20 @@ config_host_data.set('CONFIG_AVX512BW_OPT', > get_option('avx512bw') \ > int main(int argc, char *argv[]) { return bar(argv[0]); } > '''), error_message: 'AVX512BW not available').allowed()) > > +config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \ > + .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot > enable ENQCMD') \ > + .require(cc.links(''' > + #include <stdint.h> > + #include <cpuid.h> > + #include <immintrin.h> > + static int __attribute__((target("enqcmd"))) bar(void *a) { > + uint64_t dst[8] = { 0 }; > + uint64_t src[8] = { 0 }; > + return _enqcmd(dst, src); > + } > + int main(int argc, char *argv[]) { return bar(argv[argc - 1]); } > + '''), error_message: 'ENQCMD not available').allowed()) > + How about using cpuid instruction to dynamically detect enqcmd and movdir64b instructions? My reasons are as follows 1. enqcmd/movdir64b and DSA devices are used together. DSA devices are dynamically detected, so enqcmd can also dynamically detect. Simple code for dynamically detect movdir64b and enqcmd bool check_dsa_instructions(void) { uint32_t eax, ebx, ecx, edx; bool movedirb_enabled; bool enqcmd_enabled; cpuid(0x07, 0x0, &eax, &ebx, &ecx, &edx); movedirb_enabled = (ecx >> 28) & 0x1; if (!movedirb_enabled) { return false; } enqcmd_enabled = (ecx >> 29) & 0x1; if (!enqcmd_enabled) { return false; } return true; } https://cdrdv2-public.intel.com/819680/architecture-instruction-set-extensions-programming-reference.pdf 2. The enqcmd/movdir64b are new instructions, I checked they are integrated into GCC10 However, users do not need gcc10 or higher to use two instructions. Simple code to implement enqcmd static inline int enqcmd(volatile void *reg, struct dsa_hw_desc *desc) { uint8_t retry; asm volatile (".byte 0xf2, 0x0f, 0x38, 0xf8, 0x02\t\n" "setz %0\t\n":"=r" (retry):"a"(reg), "d"(desc)); return (int)retry; } file:///C:/Users/yliu80/Downloads/353216-data-streaming-accelerator-user-guide-002.pdf > # For both AArch64 and AArch32, detect if builtins are available. > config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles(''' > #include <arm_neon.h> > diff --git a/meson_options.txt b/meson_options.txt > index 0269fa0f16..4ed820bb8d 100644 > --- a/meson_options.txt > +++ b/meson_options.txt > @@ -121,6 +121,8 @@ option('avx2', type: 'feature', value: 'auto', > description: 'AVX2 optimizations') > option('avx512bw', type: 'feature', value: 'auto', > description: 'AVX512BW optimizations') > +option('enqcmd', type: 'feature', value: 'disabled', > + description: 'ENQCMD optimizations') > option('keyring', type: 'feature', value: 'auto', > description: 'Linux keyring support') > option('libkeyutils', type: 'feature', value: 'auto', > diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh > index cfadb5ea86..280e117687 100644 > --- a/scripts/meson-buildoptions.sh > +++ b/scripts/meson-buildoptions.sh > @@ -95,6 +95,7 @@ meson_options_help() { > printf "%s\n" ' auth-pam PAM access control' > printf "%s\n" ' avx2 AVX2 optimizations' > printf "%s\n" ' avx512bw AVX512BW optimizations' > + printf "%s\n" ' enqcmd ENQCMD optimizations' > printf "%s\n" ' blkio libblkio block device driver' > printf "%s\n" ' bochs bochs image format support' > printf "%s\n" ' bpf eBPF support' > @@ -239,6 +240,8 @@ _meson_option_parse() { > --disable-avx2) printf "%s" -Davx2=disabled ;; > --enable-avx512bw) printf "%s" -Davx512bw=enabled ;; > --disable-avx512bw) printf "%s" -Davx512bw=disabled ;; > + --enable-enqcmd) printf "%s" -Denqcmd=enabled ;; > + --disable-enqcmd) printf "%s" -Denqcmd=disabled ;; > --enable-gcov) printf "%s" -Db_coverage=true ;; > --disable-gcov) printf "%s" -Db_coverage=false ;; > --enable-lto) printf "%s" -Db_lto=true ;; > -- > Yichen Wang
On Mon, Jul 15, 2024 at 8:02 AM Liu, Yuan1 <yuan1.liu@intel.com> wrote: > > > -----Original Message----- > > From: Yichen Wang <yichen.wang@bytedance.com> > > Sent: Friday, July 12, 2024 5:53 AM > > To: Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau > > <marcandre.lureau@redhat.com>; Daniel P. Berrangé <berrange@redhat.com>; > > Thomas Huth <thuth@redhat.com>; Philippe Mathieu-Daudé > > <philmd@linaro.org>; Peter Xu <peterx@redhat.com>; Fabiano Rosas > > <farosas@suse.de>; Eric Blake <eblake@redhat.com>; Markus Armbruster > > <armbru@redhat.com>; Michael S. Tsirkin <mst@redhat.com>; Cornelia Huck > > <cohuck@redhat.com>; qemu-devel@nongnu.org > > Cc: Hao Xiang <hao.xiang@linux.dev>; Liu, Yuan1 <yuan1.liu@intel.com>; > > Kumar, Shivam <shivam.kumar1@nutanix.com>; Ho-Ren (Jack) Chuang > > <horenchuang@bytedance.com>; Wang, Yichen <yichen.wang@bytedance.com> > > Subject: [PATCH v5 01/13] meson: Introduce new instruction set enqcmd to > > the build system. > > > > From: Hao Xiang <hao.xiang@linux.dev> > > > > Enable instruction set enqcmd in build. > > > > Signed-off-by: Hao Xiang <hao.xiang@linux.dev> > > Signed-off-by: Yichen Wang <yichen.wang@bytedance.com> > > --- > > meson.build | 14 ++++++++++++++ > > meson_options.txt | 2 ++ > > scripts/meson-buildoptions.sh | 3 +++ > > 3 files changed, 19 insertions(+) > > > > diff --git a/meson.build b/meson.build > > index 6a93da48e1..af650cfabf 100644 > > --- a/meson.build > > +++ b/meson.build > > @@ -2893,6 +2893,20 @@ config_host_data.set('CONFIG_AVX512BW_OPT', > > get_option('avx512bw') \ > > int main(int argc, char *argv[]) { return bar(argv[0]); } > > '''), error_message: 'AVX512BW not available').allowed()) > > > > +config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \ > > + .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot > > enable ENQCMD') \ > > + .require(cc.links(''' > > + #include <stdint.h> > > + #include <cpuid.h> > > + #include <immintrin.h> > > + static int __attribute__((target("enqcmd"))) bar(void *a) { > > + uint64_t dst[8] = { 0 }; > > + uint64_t src[8] = { 0 }; > > + return _enqcmd(dst, src); > > + } > > + int main(int argc, char *argv[]) { return bar(argv[argc - 1]); } > > + '''), error_message: 'ENQCMD not available').allowed()) > > + > > How about using cpuid instruction to dynamically detect enqcmd and movdir64b > instructions? > > My reasons are as follows > 1. enqcmd/movdir64b and DSA devices are used together. DSA devices are dynamically > detected, so enqcmd can also dynamically detect. > > Simple code for dynamically detect movdir64b and enqcmd > bool check_dsa_instructions(void) { > uint32_t eax, ebx, ecx, edx; > bool movedirb_enabled; > bool enqcmd_enabled; > > cpuid(0x07, 0x0, &eax, &ebx, &ecx, &edx); > movedirb_enabled = (ecx >> 28) & 0x1; > if (!movedirb_enabled) { > return false; > } > enqcmd_enabled = (ecx >> 29) & 0x1; > if (!enqcmd_enabled) { > return false; > } > return true; > } > https://cdrdv2-public.intel.com/819680/architecture-instruction-set-extensions-programming-reference.pdf > > 2. The enqcmd/movdir64b are new instructions, I checked they are integrated into GCC10 > However, users do not need gcc10 or higher to use two instructions. > Simple code to implement enqcmd > static inline int enqcmd(volatile void *reg, struct dsa_hw_desc *desc) > { > uint8_t retry; > asm volatile (".byte 0xf2, 0x0f, 0x38, 0xf8, 0x02\t\n" > "setz %0\t\n":"=r" (retry):"a"(reg), "d"(desc)); > return (int)retry; > } > file:///C:/Users/yliu80/Downloads/353216-data-streaming-accelerator-user-guide-002.pdf > This is for compile time detection. So if I am understanding correctly, we don't need this dynamic detection at meson build time, am I right? I actually already have similar code in dynamic detection at runtime, and I will refine that part with your suggestion above. > > # For both AArch64 and AArch32, detect if builtins are available. > > config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles(''' > > #include <arm_neon.h> > > diff --git a/meson_options.txt b/meson_options.txt > > index 0269fa0f16..4ed820bb8d 100644 > > --- a/meson_options.txt > > +++ b/meson_options.txt > > @@ -121,6 +121,8 @@ option('avx2', type: 'feature', value: 'auto', > > description: 'AVX2 optimizations') > > option('avx512bw', type: 'feature', value: 'auto', > > description: 'AVX512BW optimizations') > > +option('enqcmd', type: 'feature', value: 'disabled', > > + description: 'ENQCMD optimizations') > > option('keyring', type: 'feature', value: 'auto', > > description: 'Linux keyring support') > > option('libkeyutils', type: 'feature', value: 'auto', > > diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh > > index cfadb5ea86..280e117687 100644 > > --- a/scripts/meson-buildoptions.sh > > +++ b/scripts/meson-buildoptions.sh > > @@ -95,6 +95,7 @@ meson_options_help() { > > printf "%s\n" ' auth-pam PAM access control' > > printf "%s\n" ' avx2 AVX2 optimizations' > > printf "%s\n" ' avx512bw AVX512BW optimizations' > > + printf "%s\n" ' enqcmd ENQCMD optimizations' > > printf "%s\n" ' blkio libblkio block device driver' > > printf "%s\n" ' bochs bochs image format support' > > printf "%s\n" ' bpf eBPF support' > > @@ -239,6 +240,8 @@ _meson_option_parse() { > > --disable-avx2) printf "%s" -Davx2=disabled ;; > > --enable-avx512bw) printf "%s" -Davx512bw=enabled ;; > > --disable-avx512bw) printf "%s" -Davx512bw=disabled ;; > > + --enable-enqcmd) printf "%s" -Denqcmd=enabled ;; > > + --disable-enqcmd) printf "%s" -Denqcmd=disabled ;; > > --enable-gcov) printf "%s" -Db_coverage=true ;; > > --disable-gcov) printf "%s" -Db_coverage=false ;; > > --enable-lto) printf "%s" -Db_lto=true ;; > > -- > > Yichen Wang >
diff --git a/meson.build b/meson.build index 6a93da48e1..af650cfabf 100644 --- a/meson.build +++ b/meson.build @@ -2893,6 +2893,20 @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \ int main(int argc, char *argv[]) { return bar(argv[0]); } '''), error_message: 'AVX512BW not available').allowed()) +config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \ + .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable ENQCMD') \ + .require(cc.links(''' + #include <stdint.h> + #include <cpuid.h> + #include <immintrin.h> + static int __attribute__((target("enqcmd"))) bar(void *a) { + uint64_t dst[8] = { 0 }; + uint64_t src[8] = { 0 }; + return _enqcmd(dst, src); + } + int main(int argc, char *argv[]) { return bar(argv[argc - 1]); } + '''), error_message: 'ENQCMD not available').allowed()) + # For both AArch64 and AArch32, detect if builtins are available. config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles(''' #include <arm_neon.h> diff --git a/meson_options.txt b/meson_options.txt index 0269fa0f16..4ed820bb8d 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -121,6 +121,8 @@ option('avx2', type: 'feature', value: 'auto', description: 'AVX2 optimizations') option('avx512bw', type: 'feature', value: 'auto', description: 'AVX512BW optimizations') +option('enqcmd', type: 'feature', value: 'disabled', + description: 'ENQCMD optimizations') option('keyring', type: 'feature', value: 'auto', description: 'Linux keyring support') option('libkeyutils', type: 'feature', value: 'auto', diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh index cfadb5ea86..280e117687 100644 --- a/scripts/meson-buildoptions.sh +++ b/scripts/meson-buildoptions.sh @@ -95,6 +95,7 @@ meson_options_help() { printf "%s\n" ' auth-pam PAM access control' printf "%s\n" ' avx2 AVX2 optimizations' printf "%s\n" ' avx512bw AVX512BW optimizations' + printf "%s\n" ' enqcmd ENQCMD optimizations' printf "%s\n" ' blkio libblkio block device driver' printf "%s\n" ' bochs bochs image format support' printf "%s\n" ' bpf eBPF support' @@ -239,6 +240,8 @@ _meson_option_parse() { --disable-avx2) printf "%s" -Davx2=disabled ;; --enable-avx512bw) printf "%s" -Davx512bw=enabled ;; --disable-avx512bw) printf "%s" -Davx512bw=disabled ;; + --enable-enqcmd) printf "%s" -Denqcmd=enabled ;; + --disable-enqcmd) printf "%s" -Denqcmd=disabled ;; --enable-gcov) printf "%s" -Db_coverage=true ;; --disable-gcov) printf "%s" -Db_coverage=false ;; --enable-lto) printf "%s" -Db_lto=true ;;