From patchwork Fri Sep 20 08:38:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laszlo Ersek X-Patchwork-Id: 11154067 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 924C514E5 for ; Fri, 20 Sep 2019 08:40:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 732C52070C for ; Fri, 20 Sep 2019 08:40:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 732C52070C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:56422 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iBESe-00041k-Bw for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Sep 2019 04:40:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45636) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iBEQm-0001Z4-3d for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iBEQk-0002ie-Q6 for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46206) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iBEQk-0002iP-LL for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:18 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 01F48C057F31 for ; Fri, 20 Sep 2019 08:38:18 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-165.rdu2.redhat.com [10.10.120.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 43BE960BF1; Fri, 20 Sep 2019 08:38:17 +0000 (UTC) From: Laszlo Ersek To: qemu devel list Subject: [PATCH v2 1/2] edk2 build scripts: honor external BaseTools flags with uefi-test-tools Date: Fri, 20 Sep 2019 10:38:07 +0200 Message-Id: <20190920083808.21399-2-lersek@redhat.com> In-Reply-To: <20190920083808.21399-1-lersek@redhat.com> References: <20190920083808.21399-1-lersek@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 20 Sep 2019 08:38:18 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Unify the recipe for "build-edk2-tools" in "tests/uefi-test-tools/Makefile" with the recipe for "edk2-basetools" in "roms/Makefile". Cc: Philippe Mathieu-Daudé Signed-off-by: Laszlo Ersek Reviewed-by: Philippe Mathieu-Daudé Tested-by: Philippe Mathieu-Daudé --- Notes: v2: - new patch tests/uefi-test-tools/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile index 1d78bc14d51a..7e0177d7337e 100644 --- a/tests/uefi-test-tools/Makefile +++ b/tests/uefi-test-tools/Makefile @@ -99,7 +99,9 @@ Build/bios-tables-test.%.efi: build-edk2-tools +./build.sh $(edk2_dir) BiosTablesTest $* $@ build-edk2-tools: - $(MAKE) -C $(edk2_dir)/BaseTools + $(MAKE) -C $(edk2_dir)/BaseTools \ + EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ + EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)' clean: rm -rf Build Conf log From patchwork Fri Sep 20 08:38:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laszlo Ersek X-Patchwork-Id: 11154063 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 210BC14DB for ; Fri, 20 Sep 2019 08:40:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0271A2070C for ; Fri, 20 Sep 2019 08:40:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0271A2070C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:56414 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iBESX-0003yq-Vx for patchwork-qemu-devel@patchwork.kernel.org; Fri, 20 Sep 2019 04:40:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45649) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iBEQo-0001ZM-3l for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iBEQm-0002jd-AS for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:22 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41114) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iBEQm-0002j1-3O for qemu-devel@nongnu.org; Fri, 20 Sep 2019 04:38:20 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 61BAA3B558 for ; Fri, 20 Sep 2019 08:38:19 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-165.rdu2.redhat.com [10.10.120.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4F5D860BF1; Fri, 20 Sep 2019 08:38:18 +0000 (UTC) From: Laszlo Ersek To: qemu devel list Subject: [PATCH v2 2/2] edk2 build scripts: work around TianoCore#1607 without forcing Python 2 Date: Fri, 20 Sep 2019 10:38:08 +0200 Message-Id: <20190920083808.21399-3-lersek@redhat.com> In-Reply-To: <20190920083808.21399-1-lersek@redhat.com> References: <20190920083808.21399-1-lersek@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 20 Sep 2019 08:38:19 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , John Snow , Eduardo Habkost Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" It turns out that forcing python2 for running the edk2 "build" utility is neither necessary nor sufficient. Forcing python2 is not sufficient for two reasons: - QEMU is moving away from python2, with python2 nearing EOL, - according to my most recent testing, the lacking dependency information in the makefiles that are generated by edk2's "build" utility can cause parallel build failures even when "build" is executed by python2. And forcing python2 is not necessary because we can still return to the original idea of filtering out jobserver-related options from MAKEFLAGS. So do that. While at it, cut short edk2's auto-detection of the python3.* minor version, by setting PYTHON_COMMAND to "python3" (which we expect to be available wherever we intend to build edk2). With this patch, the guest UEFI binaries that are used as part of the BIOS tables test, and the OVMF and ArmVirtQemu platform firmwares, will be built strictly in a single job, regardless of an outermost "-jN" make option. Alas, there appears to be no reliable way to build edk2 in an (outer make, inner make) environment, with a jobserver enabled. Cc: Eduardo Habkost Cc: John Snow Cc: Philippe Mathieu-Daudé Reported-by: John Snow Signed-off-by: Laszlo Ersek Reviewed-by: Philippe Mathieu-Daudé Tested-by: Philippe Mathieu-Daudé --- Notes: v2: - set PYTHON_COMMAND to python3; update commit message [Phil] - do not pick up any feedback tags from v1 (v2 is not a trivial update) v1: - Tested on RHEL7 (where the outer "make" sets the old-style "--jobserver-fds" flag) and on Fedora 29 (where the outer "make" sets the new-style "--jobserver-auth" flag). - I've rebuilt all the edk2 binaries with this patch applied. Everything works fine. However, if you test this patch, you might notice that git reports all the build products as modified. That's because when using the python3 code in edk2 BaseTools, the generated makefiles differ greatly from the ones generated when running in python2 mode (e.g. due to different random seeds in python hashes / dictionaries). As a result, parts of the firmware volumes / firmware filesystems could appear in a different order than before. This is harmless, and doesn't necessitate checking in the rebuilt binaries. roms/Makefile | 1 + tests/uefi-test-tools/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 25 insertions(+), 4 deletions(-) diff --git a/roms/Makefile b/roms/Makefile index 6cf07d3b44ee..06339efa6fc4 100644 --- a/roms/Makefile +++ b/roms/Makefile @@ -141,6 +141,7 @@ build-efi-roms: build-pxe-roms # edk2-basetools: $(MAKE) -C edk2/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)' diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile index 7e0177d7337e..1dcddcdbbabf 100644 --- a/tests/uefi-test-tools/Makefile +++ b/tests/uefi-test-tools/Makefile @@ -100,6 +100,7 @@ Build/bios-tables-test.%.efi: build-edk2-tools build-edk2-tools: $(MAKE) -C $(edk2_dir)/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)' diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh index 4f46f8a6a217..d5391c763728 100755 --- a/roms/edk2-build.sh +++ b/roms/edk2-build.sh @@ -27,8 +27,7 @@ shift $num_args cd edk2 -# Work around . -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C @@ -43,6 +42,7 @@ fi # any), for the edk2 "build" utility. source ../edk2-funcs.sh edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target" diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh index a9fae7ee891b..3f4485b201f1 100644 --- a/roms/edk2-funcs.sh +++ b/roms/edk2-funcs.sh @@ -251,3 +251,20 @@ qemu_edk2_get_thread_count() printf '1\n' fi } + + +# Work around by +# filtering jobserver-related flags out of MAKEFLAGS. Print the result to the +# standard output. +# +# Parameters: +# $1: the value of the MAKEFLAGS variable +qemu_edk2_quirk_tianocore_1607() +{ + local makeflags="$1" + + printf %s "$makeflags" \ + | LC_ALL=C sed --regexp-extended \ + --expression='s/--jobserver-(auth|fds)=[0-9]+,[0-9]+//' \ + --expression='s/-j([0-9]+)?//' +} diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh index 8aa7935c43bb..3b78f3084069 100755 --- a/tests/uefi-test-tools/build.sh +++ b/tests/uefi-test-tools/build.sh @@ -29,8 +29,7 @@ export PACKAGES_PATH=$(realpath -- "$edk2_dir") export WORKSPACE=$PWD mkdir -p Conf -# Work around . -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C @@ -46,12 +45,15 @@ fi source "$edk2_dir/../edk2-funcs.sh" edk2_arch=$(qemu_edk2_get_arch "$emulation_target") edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") +edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target" # Build the UEFI binary mkdir -p log build \ --arch="$edk2_arch" \ + -n "$edk2_thread_count" \ --buildtarget=DEBUG \ --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ --tagname="$edk2_toolchain" \