mbox series

[v13,00/12] Build ACPI Heterogeneous Memory Attribute Table (HMAT)

Message ID 20191020111125.27659-1-tao3.xu@intel.com (mailing list archive)
Headers show
Series Build ACPI Heterogeneous Memory Attribute Table (HMAT) | expand

Message

Tao Xu Oct. 20, 2019, 11:11 a.m. UTC 
This series of patches will build Heterogeneous Memory Attribute Table (HMAT)
according to the command line. The ACPI HMAT describes the memory attributes,
such as memory side cache attributes and bandwidth and latency details,
related to the Memory Proximity Domain.
The software is expected to use HMAT information as hint for optimization.

In the linux kernel, the codes in drivers/acpi/hmat/hmat.c parse and report
the platform's HMAT tables.

The V12 patches link:
https://patchwork.kernel.org/cover/11153861/

Changelog:
v13:
    - Modify some text description
    - Drop "initiator_valid" field in struct NodeInfo
    - Reuse Garray to store the raw bandwidth and bandwidth data
    - Calculate common base unit using range bitmap
    - Add a patch to alculate hmat latency and bandwidth entry list
    - Drop the total_levels option and use readable cache size
    - Remove the unnecessary head file
    - Use decimal notation with appropriate suffix for cache size
v12:
    - Fix a bug that a memory-only node without initiator setting
      doesn't report error. (reported by Danmei Wei)
    - Fix a bug that if HMAT is enabled and without hmat-lb setting,
      QEMU will crash. (reported by Danmei Wei)
v11:
    - Move numa option patches forward.
    - Add num_initiator in Numa_state to record the number of
    initiators.
    - Simplify struct HMAT_LB_Info, use uint64_t array to store data.
    - Drop hmat_get_base().
    - Calculate base in build_hmat_lb().
v10:
    - Add qemu_strtotime_ps() to convert strings with time suffixes
    to numbers, and add some tests for it.
    - Add qapi buildin type time, and add some tests for it.
    - Add machine oprion properties "-machine hmat=on|off" for
	  enabling or disabling HMAT in QEMU.
v9:
    - change the CLI input way, make it more user firendly (Daniel Black)
    use latency=NUM[p|n|u]s and bandwidth=NUM[M|G|P](B/s) as input and drop
    the base-lat and base-bw input.
Liu Jingqi (5):
  numa: Extend CLI to provide memory latency and bandwidth information
  numa: Extend CLI to provide memory side cache information
  hmat acpi: Build Memory Proximity Domain Attributes Structure(s)
  hmat acpi: Build System Locality Latency and Bandwidth Information
    Structure(s)
  hmat acpi: Build Memory Side Cache Information Structure(s)

Tao Xu (7):
  util/cutils: Add qemu_strtotime_ps()
  tests/cutils: Add test for qemu_strtotime_ps()
  qapi: Add builtin type time
  tests: Add test for QAPI builtin type time
  numa: Extend CLI to provide initiator information for numa nodes
  numa: Calculate hmat latency and bandwidth entry list
  tests/bios-tables-test: add test cases for ACPI HMAT

 hw/acpi/Kconfig                    |   7 +-
 hw/acpi/Makefile.objs              |   1 +
 hw/acpi/hmat.c                     | 263 +++++++++++++++++++++++++++
 hw/acpi/hmat.h                     |  42 +++++
 hw/core/machine.c                  |  70 ++++++++
 hw/core/numa.c                     | 273 ++++++++++++++++++++++++++++-
 hw/i386/acpi-build.c               |   5 +
 include/qapi/visitor-impl.h        |   4 +
 include/qapi/visitor.h             |   9 +
 include/qemu/cutils.h              |   1 +
 include/sysemu/numa.h              | 104 +++++++++++
 qapi/machine.json                  | 179 ++++++++++++++++++-
 qapi/opts-visitor.c                |  22 +++
 qapi/qapi-visit-core.c             |  12 ++
 qapi/qobject-input-visitor.c       |  18 ++
 qapi/trace-events                  |   1 +
 qemu-options.hx                    |  96 +++++++++-
 scripts/qapi/common.py             |   1 +
 tests/bios-tables-test.c           |  44 +++++
 tests/test-cutils.c                | 199 +++++++++++++++++++++
 tests/test-keyval.c                | 125 +++++++++++++
 tests/test-qobject-input-visitor.c |  29 +++
 util/cutils.c                      |  82 +++++++++
 23 files changed, 1575 insertions(+), 12 deletions(-)
 create mode 100644 hw/acpi/hmat.c
 create mode 100644 hw/acpi/hmat.h

Comments

no-reply@patchew.org Oct. 20, 2019, 11:43 a.m. UTC | #1 
Patchew URL: https://patchew.org/QEMU/20191020111125.27659-1-tao3.xu@intel.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

Looking for expected file 'tests/data/acpi/pc/SRAT.acpihmat'
Looking for expected file 'tests/data/acpi/pc/SRAT'
**
ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
make: *** [check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
  TEST    iotest-qcow2: 025
  TEST    iotest-qcow2: 027
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=d317dcfdadb14357ab4e6e456e466c28', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-jvsh36ab/src/docker-src.2019-10-20-07.31.22.24116:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=d317dcfdadb14357ab4e6e456e466c28
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-jvsh36ab/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    12m1.295s
user    0m8.390s


The full log is available at
http://patchew.org/logs/20191020111125.27659-1-tao3.xu@intel.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 20, 2019, 12:13 p.m. UTC | #2 
Patchew URL: https://patchew.org/QEMU/20191020111125.27659-1-tao3.xu@intel.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

Looking for expected file 'tests/data/acpi/pc/SRAT.acpihmat'
Looking for expected file 'tests/data/acpi/pc/SRAT'
**
ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
make: *** [check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
  TEST    iotest-qcow2: 036
  TEST    iotest-qcow2: 037
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=1a622dbdc82341f49970cd92203d9bcd', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-6i8n4o9m/src/docker-src.2019-10-20-08.03.22.16115:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=1a622dbdc82341f49970cd92203d9bcd
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-6i8n4o9m/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    9m56.614s
user    0m6.591s


The full log is available at
http://patchew.org/logs/20191020111125.27659-1-tao3.xu@intel.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 20, 2019, 12:57 p.m. UTC | #3 
Patchew URL: https://patchew.org/QEMU/20191020111125.27659-1-tao3.xu@intel.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

Looking for expected file 'tests/data/acpi/pc/SRAT.acpihmat'
Looking for expected file 'tests/data/acpi/pc/SRAT'
**
ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
make: *** [check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
  TEST    iotest-qcow2: 035
  TEST    check-unit: tests/test-bufferiszero
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=333528b3def44bedb7868163d167de8a', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-83x7cpec/src/docker-src.2019-10-20-08.46.30.16537:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=333528b3def44bedb7868163d167de8a
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-83x7cpec/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    10m40.886s
user    0m8.703s


The full log is available at
http://patchew.org/logs/20191020111125.27659-1-tao3.xu@intel.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 20, 2019, 1:19 p.m. UTC | #4 
Patchew URL: https://patchew.org/QEMU/20191020111125.27659-1-tao3.xu@intel.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

Looking for expected file 'tests/data/acpi/pc/SRAT.acpihmat'
Looking for expected file 'tests/data/acpi/pc/SRAT'
**
ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/bios-tables-test.c:354:load_expected_aml: assertion failed: (exp_sdt.aml_file)
make: *** [check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
  TEST    iotest-qcow2: 035
  TEST    iotest-qcow2: 036
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=086575f112be4f06b5b6628ea1ada77d', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-ug9vqabf/src/docker-src.2019-10-20-09.09.36.21970:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=086575f112be4f06b5b6628ea1ada77d
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-ug9vqabf/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    10m16.781s
user    0m8.561s


The full log is available at
http://patchew.org/logs/20191020111125.27659-1-tao3.xu@intel.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Markus Armbruster Oct. 22, 2019, 11:22 a.m. UTC | #5 
I just stumbled over this series.  It touches the QAPI visitors and even
the generator, without cc'ing its maintainers.  Such changes require
review.  There's precious little time until the soft freeze now.  I'll
try, but no promises.

Please cc me and Michael Roth <mdroth@linux.vnet.ibm.com> on future
revisions.

In general, peruse output of scripts/get_maintainer.pl to find relevant
maintainers.

Tao Xu <tao3.xu@intel.com> writes:

> This series of patches will build Heterogeneous Memory Attribute Table (HMAT)
> according to the command line. The ACPI HMAT describes the memory attributes,
> such as memory side cache attributes and bandwidth and latency details,
> related to the Memory Proximity Domain.
> The software is expected to use HMAT information as hint for optimization.
>
> In the linux kernel, the codes in drivers/acpi/hmat/hmat.c parse and report
> the platform's HMAT tables.
>
> The V12 patches link:
> https://patchwork.kernel.org/cover/11153861/
>
> Changelog:
> v13:
>     - Modify some text description
>     - Drop "initiator_valid" field in struct NodeInfo
>     - Reuse Garray to store the raw bandwidth and bandwidth data
>     - Calculate common base unit using range bitmap
>     - Add a patch to alculate hmat latency and bandwidth entry list
>     - Drop the total_levels option and use readable cache size
>     - Remove the unnecessary head file
>     - Use decimal notation with appropriate suffix for cache size
> v12:
>     - Fix a bug that a memory-only node without initiator setting
>       doesn't report error. (reported by Danmei Wei)
>     - Fix a bug that if HMAT is enabled and without hmat-lb setting,
>       QEMU will crash. (reported by Danmei Wei)
> v11:
>     - Move numa option patches forward.
>     - Add num_initiator in Numa_state to record the number of
>     initiators.
>     - Simplify struct HMAT_LB_Info, use uint64_t array to store data.
>     - Drop hmat_get_base().
>     - Calculate base in build_hmat_lb().
> v10:
>     - Add qemu_strtotime_ps() to convert strings with time suffixes
>     to numbers, and add some tests for it.
>     - Add qapi buildin type time, and add some tests for it.
>     - Add machine oprion properties "-machine hmat=on|off" for
> 	  enabling or disabling HMAT in QEMU.

I guess this is where you started messing with QAPI.  Seven weeks ago.
The time crunch is of your own making, I'm afraid.

> v9:
>     - change the CLI input way, make it more user firendly (Daniel Black)
>     use latency=NUM[p|n|u]s and bandwidth=NUM[M|G|P](B/s) as input and drop
>     the base-lat and base-bw input.
> Liu Jingqi (5):
>   numa: Extend CLI to provide memory latency and bandwidth information
>   numa: Extend CLI to provide memory side cache information
>   hmat acpi: Build Memory Proximity Domain Attributes Structure(s)
>   hmat acpi: Build System Locality Latency and Bandwidth Information
>     Structure(s)
>   hmat acpi: Build Memory Side Cache Information Structure(s)
>
> Tao Xu (7):
>   util/cutils: Add qemu_strtotime_ps()
>   tests/cutils: Add test for qemu_strtotime_ps()
>   qapi: Add builtin type time
>   tests: Add test for QAPI builtin type time
>   numa: Extend CLI to provide initiator information for numa nodes
>   numa: Calculate hmat latency and bandwidth entry list
>   tests/bios-tables-test: add test cases for ACPI HMAT
>
>  hw/acpi/Kconfig                    |   7 +-
>  hw/acpi/Makefile.objs              |   1 +
>  hw/acpi/hmat.c                     | 263 +++++++++++++++++++++++++++
>  hw/acpi/hmat.h                     |  42 +++++
>  hw/core/machine.c                  |  70 ++++++++
>  hw/core/numa.c                     | 273 ++++++++++++++++++++++++++++-
>  hw/i386/acpi-build.c               |   5 +
>  include/qapi/visitor-impl.h        |   4 +
>  include/qapi/visitor.h             |   9 +
>  include/qemu/cutils.h              |   1 +
>  include/sysemu/numa.h              | 104 +++++++++++
>  qapi/machine.json                  | 179 ++++++++++++++++++-
>  qapi/opts-visitor.c                |  22 +++
>  qapi/qapi-visit-core.c             |  12 ++
>  qapi/qobject-input-visitor.c       |  18 ++
>  qapi/trace-events                  |   1 +
>  qemu-options.hx                    |  96 +++++++++-
>  scripts/qapi/common.py             |   1 +
>  tests/bios-tables-test.c           |  44 +++++
>  tests/test-cutils.c                | 199 +++++++++++++++++++++
>  tests/test-keyval.c                | 125 +++++++++++++
>  tests/test-qobject-input-visitor.c |  29 +++
>  util/cutils.c                      |  82 +++++++++
>  23 files changed, 1575 insertions(+), 12 deletions(-)
>  create mode 100644 hw/acpi/hmat.c
>  create mode 100644 hw/acpi/hmat.h
Tao Xu Oct. 23, 2019, 1:46 a.m. UTC | #6 
On 10/22/2019 7:22 PM, Markus Armbruster wrote:
> I just stumbled over this series.  It touches the QAPI visitors and even
> the generator, without cc'ing its maintainers.  Such changes require
> review.  There's precious little time until the soft freeze now.  I'll
> try, but no promises.
> 
> Please cc me and Michael Roth <mdroth@linux.vnet.ibm.com> on future
> revisions.
> 
> In general, peruse output of scripts/get_maintainer.pl to find relevant
> maintainers.

[...]
>> v10:
>>      - Add qemu_strtotime_ps() to convert strings with time suffixes
>>      to numbers, and add some tests for it.
>>      - Add qapi buildin type time, and add some tests for it.
>>      - Add machine oprion properties "-machine hmat=on|off" for
>> 	  enabling or disabling HMAT in QEMU.
> 
> I guess this is where you started messing with QAPI.  Seven weeks ago.
> The time crunch is of your own making, I'm afraid.
> 

I am sorry. I will cc all related maintainers next time.