From patchwork Tue Nov 19 16:13:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= X-Patchwork-Id: 13880236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A654AD4417D for ; Tue, 19 Nov 2024 16:14:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tDQqy-0002v9-E9; Tue, 19 Nov 2024 11:13:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tDQqx-0002uw-8m for qemu-devel@nongnu.org; Tue, 19 Nov 2024 11:13:23 -0500 Received: from mail-ej1-x632.google.com ([2a00:1450:4864:20::632]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tDQqs-0006jw-VI for qemu-devel@nongnu.org; Tue, 19 Nov 2024 11:13:22 -0500 Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-aa20c733e92so630327166b.0 for ; Tue, 19 Nov 2024 08:13:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1732032796; x=1732637596; darn=nongnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=NEO9y/2dXb2rglF5UQ2YE4C9JjX85T2tA6iOF/v6Xl4=; b=zj3cxHKDupIbnlbHpYkMREoRrYF58NpgAcNgXVRh6EMvo8rIz4+8ZlJyoojCfyosQS xJY1Xt4X67uBSPzR53An1klojMmBZyRb2WLzghdWUC4Nh37N6GmQ1PjAfGZeJY7rXzL8 wvbNW3b/2obUzXhOaoLuc+9Izt03wrDXE3gcrPXLoVrqtIc8ENcMHK4vyneisnKKxNor A14YM1dupRhhffvGKkHDAPU37D4T9kTliFljGl5icPM44h/1NbhgILvVmU46YZt8Oyq9 MBlUOqaMyv7+QNJHZr6QAv3NLPShkTaBZIInm9nQ4GsibB2HbTqzKHCFy23umSI2QLr3 v8pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732032796; x=1732637596; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NEO9y/2dXb2rglF5UQ2YE4C9JjX85T2tA6iOF/v6Xl4=; b=I7HtptX47gzmIbDxBWeiej7okDmtKCvGmLvYZjn7eAof9+SnsIzvv6RYA3LZ9z/8ys 3NhOgJ8HPcWX1iRS6/aRG5Uk4zlJsYuwx8gCA1bvKu5gdd5KpJg5Cc+zBB8J2gRnA9kz g+Zh7zgX9B4J3EiNLB5RzCL1/M6QoIzsaoGXf6iANgIyNB6H+s8c7eDK+L4bxEO4IwEx aJXFos1aIXH+uUHs0NX92+xQb2TTqO9erof3VjbbJJ3scLSYdzx8hqYhtNVwJ34Ydu5H wVZljdMg+hu46PiClhzqswCJOQVC+0MgpW+VK4VDORmpw2vhh0kxxUmxqxB9aUZTbpiv hPog== X-Gm-Message-State: AOJu0YwIVAYX3LjGvWXn8YZZbNOR81YsUJfLz7RDPMCVjDk6yBsh61c8 vNma8ce1VjNNqV18wRK/iFH7RJPPXGf7/lXCdtIMHXPEGps6m2OXJoY1r8ip+tChXPX5qThGOFL Msh8= X-Google-Smtp-Source: AGHT+IHxj4hukCSlum/LdZFSyiMKeYw9qRQcGapWL7qf4TUxWyQbpkZMK3PWuYuH2wdsn8vC3s2i9g== X-Received: by 2002:a17:907:80e:b0:a9a:eca:f7c4 with SMTP id a640c23a62f3a-aa48354cd95mr1346055566b.54.1732032795877; Tue, 19 Nov 2024 08:13:15 -0800 (PST) Received: from localhost.localdomain (sto95-h01-176-184-18-155.dsl.sta.abo.bbox.fr. [176.184.18.155]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa20dfffdb0sm672187866b.125.2024.11.19.08.13.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Nov 2024 08:13:15 -0800 (PST) From: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Paolo Bonzini , Peter Maydell , =?utf-8?q?Alex_Benn=C3=A9e?= , Richard Henderson , =?utf-8?q?Daniel_P_=2E_Be?= =?utf-8?q?rrang=C3=A9?= , Markus Armbruster , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= Subject: [RFC PATCH] docs: Add roadmap for heterogeneous emulation Date: Tue, 19 Nov 2024 17:13:12 +0100 Message-ID: <20241119161312.41346-1-philmd@linaro.org> X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::632; envelope-from=philmd@linaro.org; helo=mail-ej1-x632.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This document tries to document the steps required to: - Have a single binary to run system emulations - Emulate different architectures in the same process - Have QEMU assemble dynamic machines at runtime Signed-off-by: Philippe Mathieu-Daudé --- This is the document that was discussed at the KVM forum 2024 BoF. Since then I did some changes but I figured it'd be better to post the discussed doc first then comment the changes on the list. --- docs/devel/heterogeneous-emulation-roadmap.md | 892 ++++++++++++++++++ 1 file changed, 892 insertions(+) create mode 100644 docs/devel/heterogeneous-emulation-roadmap.md diff --git a/docs/devel/heterogeneous-emulation-roadmap.md b/docs/devel/heterogeneous-emulation-roadmap.md new file mode 100644 index 0000000000..6377b02559 --- /dev/null +++ b/docs/devel/heterogeneous-emulation-roadmap.md @@ -0,0 +1,892 @@ +# Toward a Heterogeneous QEMU + +## Introduction + +This document outlines a comprehensive roadmap for enhancing the +modularity, scalability, and multi-architecture support of QEMU. +The goal is to address several long-standing architectural issues +that hinder QEMU’s ability to emulate multiple architectures +concurrently, paving the way toward dynamic machine creation and +improved user experiences. + +QEMU’s original design focuses on the efficient emulation of a +single target architecture per binary, with a single TCG context +handling architecture-specific code generation and execution. +However, as the demand for more flexible and scalable emulation +grows, several limitations of this design have become evident. +These include reliance on global variables, target-specific static +definitions, tightly coupled APIs across different subsystems, and +the need for manual intervention when building or configuring +machine models. + +The proposed roadmap introduces a series of incremental changes +aimed at addressing these limitations and achieving the following +long-term goals: + +1. **Improved Modularity and Code Reusability**: Refactor QEMU's + subsystems to eliminate global variables and reduce the coupling + between hardware models and target-specific implementations. + This will involve creating clearer API boundaries and separating + hardware and target-specific code into distinct, reusable + modules. + +2. **Support for Multi-Architecture Emulation**: Enable QEMU to + emulate multiple target architectures concurrently within a + single binary by modularizing the TCG frontends and decoupling + architecture-specific code from the core emulation logic. + +3. **Simplification of QEMU's Core Components**: Streamline the + startup sequence, refactor the command-line interface (CLI), and + modularize the device models and subsystems to improve + maintainability and extensibility. This effort will also focus + on reducing redundancy and boilerplate code in the machine + creation process. + +4. **Human vs. Machine Configuration**: Recognize the divergent + needs between human users and management applications (machines) + in configuring QEMU. Propose a separation of concerns where a + new binary could focus on machine-oriented expressiveness while + providing a simpler, more intuitive frontend for human users. + +5. **Dynamic Machine Creation and Configuration**: Transition from + statically built machine models to a dynamic, data-driven + configuration system, where new machine types can be composed at + runtime using declarative templates. This will reduce the + complexity of adding new machines and allow for greater + flexibility in system configuration. + +The roadmap is divided into multiple phases, each addressing a +specific set of problems and dependencies in QEMU’s current +architecture. + +This document serves as a reference for contributors. It aims to +provide a clear path forward, along with the necessary steps and +dependencies required to implement the proposed changes. + +## Background and Motivation + +### Issues with Loadable Modules + +QEMU's support for loadable modules has deficiencies: + +- **Error Handling**: Modules that fail to load due to missing + dependencies can cause QEMU to exit unexpectedly, especially + during hot-plug operations. The module API needs to be + strengthened to propagate errors and must not exit the main + process. Proper error handling mechanisms are essential to + prevent crashes and improve stability. + +- **Platform Parity**: Currently, QEMU's support for loadable + modules is not consistent across all platforms. In particular, + DSO module loading needs to be implemented on Windows to achieve + feature parity with other operating systems. + +- **Introspection Challenges**: Functions that list or load modules + may not handle errors gracefully, leading to incomplete + information and unpredictable behavior. Users may not be aware of + whether a module is compiled-in or loadable, leading to + confusion. + +### Problems with Singletons and Global Symbols + +Global variables and singletons hinder the ability to: + +- **Support Multi-Architecture Binaries**: The use of global + variables (or singletons) causes conflicts when multiple + architectures are emulated concurrently. A frontend expects the + global variable to be accessed only by itself, but when using + multiple frontends concurrently, multiple frontends could access + the same variable, causing unexpected side effects. + +- **Modularize Code**: Singletons prevent instantiating multiple + instances of certain components, limiting flexibility. Symbol + conflicts and shared globals cause linking issues and + unpredictable behavior in heterogeneous machines. + +### Inconsistent QOM Object Life Cycle + +The lack of a unified life cycle model for QOM objects complicates +configuration and management: + +- **Different Life Cycles**: Devices, user-creatable objects, and + other components have varying life cycles and state transitions. + Without a consistent model, it's difficult to determine when and + how objects should be configured and realized, especially in a + dynamic environment. + +- **Configuration Challenges**: Managing object states and + transitions is difficult without a clear life cycle model. This + inconsistency leads to complex and error-prone configuration + processes. + +### Challenges with Initial Configuration + +QEMU's current configuration mechanisms present challenges for +both management applications and human users: + +- **Management Applications**: Require a stable and simple CLI to + bootstrap QMP (QEMU Machine Protocol) for monitoring and initial + configuration. However, they are forced to perform non-trivial + initial configurations via the CLI, which is not ideal. + +- **Human Users**: Prefer simple, consistent CLI and configuration + files. They struggle with the complexity and inconsistency of the + modern low-level configurations, leading to frustration and + errors. + +### Limitations of Static Machine Definitions + +Machines in QEMU are defined statically in C code, requiring +recompilation for any changes. This approach lacks flexibility and +makes it difficult to: + +- **Enable Dynamic Machine Creation**: Allow users and management + applications to define and configure machines at runtime without + modifying source code. + +### Need for Early Availability of QMP + +QMP becomes available late in the startup sequence, limiting its +usefulness for initial configuration. An integrated solution is +needed to make QMP available earlier, enabling: + +- **Dynamic Configuration via QMP**: Allow management applications + to perform arbitrary configurations before any machine components + are initialized. + +- **Simplified Bootstrapping**: Provide a minimal CLI that is + stable and sufficient to start QMP for further configuration. + +### Deficiencies in the Configuration Interface + +QOM's object configuration interface has limitations: + +- **Dynamic Properties**: Properties can be added or removed + dynamically, leading to a configuration interface that changes at + runtime. + +- **Introspection Limitations**: Weak and often undocumented type + information makes it challenging to design a robust and + user-friendly configuration interface. + +### Cross-Directory API Calls and Target-Specific Device Models + +- **Cross-Directory API Calls**: Problematic calls between `hw/` + and `target/` directories blur separation and complicate + modularization. Unfortunately, we have API calls from one + directory to another. For example, `hw/xtensa/pic_cpu.c` calls + target `xtensa_runstall()`, and `target/xtensa/exc_helper.c` + calls `check_interrupts()` in `hw/xtensa/pic_cpu.c`. + +- **Target-Specific Device Models in `hw/`**: Placing + target-specific models in `hw/` leads to dependencies that hinder + modularization. When a device model is target-specific (like the + ARM NVIC), it is pointless to expose it in `hw/`. Moving it to + `target/arm/hw/` could simplify the `hw/` <-> `target/` access + problem. + +### Device Models and Buses + +- **Singleton Buses**: The expectation of a single system bus + (sysbus) doesn't scale for heterogeneous machines requiring + multiple buses. Currently, a machine expects a single sysbus and + at most a single ISA bus. This doesn't scale since heterogeneous + machines might use multiple distinct buses. + +- **Bus Ownership Model**: The current ownership model doesn't + support buses shared by multiple controllers. Currently, a bus + model is "QOM owned" by a single device controller model. This + doesn't scale when a bus is shared by two controllers since we + can only have a single QOM owner. + +### Page Size and Target-Specific Definitions + +- **Static Definitions**: Target-specific static definitions need + to be converted to runtime variables to support multiple + architectures. For example, definitions such as + `TARGET_LONG_BITS`, `TARGET_PAGE_MASK`, `TARGET_PAGE_BITS`, + `TCG_GUEST_DEFAULT_MO`, `NB_MMU_MODES`, + `TARGET_PHYS_ADDR_SPACE_BITS`, and + `TARGET_VIRT_ADDR_SPACE_BITS` need to be changed to runtime ones. + +- **Variable Page Sizes**: The current use of a single + `TargetPageBits` structure is insufficient for variable page + sizes across architectures. We cannot use a single + `TargetPageBits` structure anymore for variable page sizes; we + need to make this structure per vCPU. + +### Global Headers and `NEED_CPU_H` Definition + +- **Header Contamination**: Too many global headers are + 'contaminated' with the `NEED_CPU_H` definition, which forces the + header to become target specific. This complicates modularization + and code reuse. + +## Identified Problems + +### Problem 1: Loadable Modules and Error Handling + +**Current Challenges:** + +The current implementation of loadable modules in QEMU has several +deficiencies, particularly in error handling. When a module fails +to load due to missing dependencies, QEMU may exit unexpectedly, +which can be especially problematic during hot-plug operations. +Additionally, the introspection mechanisms may not list all +available types if modules fail to load silently. + +**Proposed Solutions:** + +1. **Enhance Error Handling for Loadable Modules:** + + - Improve the module loading API to handle failures gracefully + without crashing the main process. This includes strengthening + the module API to propagate errors properly and avoid + unexpected exits. + - Ensure that introspection commands like `qom-list-types` + provide accurate information, even in the presence of loadable + modules. + - Modify the module API to propagate errors properly and avoid + unexpected exits. + +2. **Implement DSO Module Loading on Windows:** + + - Add feature parity for DSO module support on Windows OS to + ensure consistent behavior across platforms. + +3. **Testing and Stability:** + + - Implement thorough testing to ensure that the changes do not + introduce new issues or regressions, particularly in the + context of module loading and hot-plugging. + +### Problem 2: Singletons and Global Symbols + +**Current Challenges:** + +Global variables and singletons in QEMU's codebase pose challenges +for multi-architecture support. These elements can lead to +conflicts and unpredictable behavior, especially when trying to +link multiple targets into a single binary or manage heterogeneous +machines. + +**Proposed Solutions:** + +1. **Enumerate and Isolate Global Variables and Singletons:** + + - Identify all global variables and singletons that need to be + isolated. Refactor the codebase to eliminate singletons and + reduce reliance on global symbols, particularly in areas where + multiple architectures are involved. + - This change is essential for enabling concurrent + multi-architecture emulation and supporting dynamic machine + creation. + +2. **Modify Function Prototypes and Structures:** + + - Change methods and structures that are currently + target-specific to be target-agnostic. For example, functions + like `tcg_gen_code()` and structures like `TranslationBlock` + and `TCGContext` should become target-agnostic. + +### Problem 3: Target-Specific Static Definitions + +**Current Challenges:** + +Target-specific static definitions prevent the code from being +flexible enough to handle multiple architectures simultaneously. +Definitions like `TARGET_LONG_BITS`, `TARGET_PAGE_MASK`, and others +are statically defined and cannot vary at runtime. + +**Proposed Solutions:** + +1. **Convert Static Definitions to Runtime Variables:** + + - Change target-specific static definitions to runtime variables + or structure fields. For instance: + + - Convert `TCG_GUEST_DEFAULT_MO` definition to an + `ArchAccelClass` field. + - Convert `TARGET_PHYS_ADDR_SPACE_BITS` and + `TARGET_VIRT_ADDR_SPACE_BITS` definitions to `CPUClass` + fields. + - Convert `TARGET_PAGE_BITS_MIN` definition to a `CPUClass` + field. + - Convert `TARGET_LONG_BITS` definition to a `CPUClass` + helper function. + - Convert `TARGET_PAGE_*` definitions to runtime variables. + +2. **Support Variable Page Sizes:** + + - Convert all MTTCG-enabled targets to use + `TARGET_PAGE_BITS_VARY`. Since multiple vCPUs can share the + same accelerator and page sizes, better would be to have a set + of compatible vCPUs share the same accelerator context. + +### Problem 4: TCG Frontend and Code Generation + +**Current Challenges:** + +Functions like `tcg_gen_code()` are too target-specific and need to +be made target-agnostic. The TCG frontend is built multiple times +for different endianness and word sizes, leading to redundancy. + +**Proposed Solutions:** + +1. **Make TCG Code Target-Agnostic:** + + - Refactor `tb_gen_code()` and `tcg_gen_code()` functions to be + target-agnostic, keeping `gen_intermediate_code()` + target-specific. + - Ensure that TCG frontends leverage the `MemOp` argument to be + built once, reducing redundancy. + +2. **Modularize TCG Frontends:** + + - Build the TCG frontend for each architecture as a library + (e.g., `libtcg-$ARCH.so`). Such a library should be + exclusively composed of compilation units in `target/$ARCH`, + possibly including `target/$arch/hw/` specific hardware models. + - Modularize the set of TCG frontends for all targets. + +### Problem 5: Cross-Directory API Calls and Target-Specific Device Models + +**Current Challenges:** + +Problematic API calls between `hw/` and `target/` directories blur +separation and complicate modularization. Target-specific device +models placed in `hw/` lead to dependencies that hinder +modularization. + +**Proposed Solutions:** + +1. **Enforce API Separation:** + + - Remove non-QDev API calls from `target/` to `hw/` (and + vice-versa), restricting to QOM, QDev, IRQ, and Clock APIs. + - Enforce a clearer API separation to avoid direct calls and + enable both `hw/` and `target/` components to be built as + modules. + +2. **Move Target-Specific Device Models:** + + - When a device model is target-specific, move it from `hw/` to + `target/$ARCH/hw/`. This simplifies the `hw/` <-> `target/` + access problem and reduces dependencies. + +### Problem 6: Device Models and Buses + +**Current Challenges:** + +The current bus models and APIs assume singleton buses and device +ownership by a single controller, which doesn't scale for +heterogeneous machines that require multiple buses and shared +ownership. + +**Proposed Solutions:** + +1. **Rework Bus Models and Ownership:** + + - Remove device ownership on multi-controller buses, making the + single owner the `MachineState`. This allows buses to be + accessed by multiple controller models without ownership + conflicts. + +2. **Make Bus Parameters Explicit:** + + - Modify APIs to make the bus parameter explicit, so devices can + be accessed on any bus. This removes the reliance on singleton + buses and allows for multiple distinct buses in heterogeneous + machines. + +3. **Remove Singleton Buses:** + + - Remove the `ISA_BUS` singleton and replace ISA bus-dependent + devices with stubs to simplify proof of concept. + +### Problem 7: Lack of a Clear QOM Object Life Cycle + +**Current Challenges:** + +QOM lacks a unified and well-defined life cycle model, leading to +inconsistencies and complexity in managing object states during +configuration. + +**Proposed Solutions:** + +1. **Clarify QDev Methods and Lifecycle:** + + - Clarify which QDev methods belong to the QOM layer, such as + 'realize'. Define a unified life cycle model that can be + applied across all object types in QOM, ensuring predictable + transitions between states. + +2. **Add a "Wiring" Phase:** + + - Add an extra "wiring" phase in the QDev state machine to link + parts together. This is essential when composing complex + device models from smaller parts and when devices need to + reference each other before realization. + +3. **Merge SysBus Methods into QDev:** + + - Merge most of the `SysBus` methods into the more generic QDev + layer: named GPIO and MemoryRegion. This unification simplifies + the device model and reduces redundancy. + +### Problem 8: Deficiencies in QOM's Object Configuration Interface + +**Current Challenges:** + +The current QOM object configuration interface is dynamic and often +unintuitive, with properties being added or removed during runtime. +This leads to a complex and sometimes error-prone configuration +process, particularly when trying to achieve a declarative +configuration style. + +**Proposed Solutions:** + +1. **Integrate Configuration into QAPI:** + + - Use static QAPI-based QOM properties to provide a more + standardized, introspectable, and documented approach to + object configuration. Restrict dynamic read-write properties + to debugging and introspection. + +2. **Make Config Properties Read-Only After Realization:** + + - Ensure that configuration properties are read-only after the + device is realized. This simplifies the configuration + interface and enforces consistency. + +3. **Differentiate Property Types:** + + - Distinguish between static properties (set during creation) + and dynamic properties (modifiable at runtime), simplifying + the configuration process. + +### Problem 9: Initial Configuration & CLI Challenges + +**Current Challenges:** + +The current CLI is cumbersome and unintuitive, making it difficult +for both human users and management applications to configure +machines efficiently. Human users often prefer simpler, legacy- +style configurations but are sometimes forced to engage with low- +level, detailed configurations when dealing with specific features, +leading to a frustrating user experience. Management applications, +on the other hand, require a more expressive and detailed +configuration system, further complicating the situation. + +**Proposed Solutions:** + +1. **Split Human vs. Machine Data:** + + - Propose a clean separation between human-friendly and machine- + oriented configurations. Extract or factor out the CLI API + (human-facing), keeping the minimum needed to start a QMP + interface (machine-facing). + +2. **Simplify the QEMU Startup Sequence:** + + - Rework the QEMU startup code to greatly simplify it. This + involves simplifying CLI limitations (e.g. where command line + order matters) and eventually moving towards data-driven machine + configurations. + +3. **Make All Devices User-Creatable:** + + - Eventually, all devices will become user-creatable, allowing + for more flexible and dynamic machine configurations. Audit + the effect on the final Realize phase in + `qdev_machine_creation_done()`. Restrict UserCreatable API + to backends. + +## Roadmap + +The roadmap is structured into phases, each building upon the +previous and allowing for parallel work where feasible. Time +estimates are provided for each task, assuming a dedicated +development team. + +### Phase 0: Preparatory Work (Estimated Duration: 2 Months) + +**Objective**: Establish a foundation for multi-architecture +support and improve module handling. + +1. **Enhance Module Loading API** (3 Weeks) + + - Modify the module loading API to handle failures gracefully + without crashing the main process. This includes strengthening + the module API to propagate errors properly and avoid + unexpected exits. + +2. **Implement DSO Module Loading on Windows** (2 Weeks) + + - Add feature parity for DSO module support on Windows OS. This + ensures that QEMU's module loading capabilities are consistent + across platforms, enabling broader adoption and testing of + modular components on Windows systems. + +3. **Deprecate Non-QOM Code** (2 Weeks) + + - Introduce runtime warnings for the use of non-QOM code to + encourage migration to QOM-compliant implementations. + Explicitly deprecate non-QOM code, guiding developers towards + the modern object model. + +### Phase 1: Enable Heterogeneous Emulation (Estimated Duration: 6 Months) + +**Objective**: Modify the core infrastructure to support multiple +architectures concurrently. + +1. **Enumerate and Isolate Global Variables and Singletons** (6 + Weeks) + + - Identify all global variables and singletons that have to be + isolated. Refactor the codebase to eliminate or encapsulate + them to prevent conflicts when multiple architectures are + emulated concurrently. + +2. **Resolve Function Name Clashes** (2 Weeks) + + - Implement dispatch mechanisms for target-specific functions. + For functions that have the same name across targets, create + generic function names that dispatch to the proper target + handler using the vCPU to discriminate which architecture + handler to call. + +3. **Rework QMP Handlers for Multi-Architecture** (3 Weeks) + + - Modify QMP target-specific handlers to handle multiple + architectures gracefully. Alter QMP introspection as needed, + so previously not implemented methods are now dispatched and, + if not available for a particular target, return appropriate + error messages like "Not Available". + +4. **Make TCG Code Target-Agnostic** (6 Weeks) + + - Refactor `tb_gen_code()` and `tcg_gen_code()` functions to be + target-agnostic, keeping `gen_intermediate_code()` + target-specific. Ensure that `TranslationBlock` and + `TCGContext` structures become target-agnostic. + +5. **Convert Static Definitions to Runtime Variables** (6 Weeks) + + - Change target-specific static definitions to runtime variables + or structure fields. For instance: + + - Convert `TCG_GUEST_DEFAULT_MO` definition to an + `ArchAccelClass` field. + - Convert `TARGET_PHYS_ADDR_SPACE_BITS` and + `TARGET_VIRT_ADDR_SPACE_BITS` definitions to `CPUClass` + fields. + - Convert `TARGET_PAGE_BITS_MIN` definition to a `CPUClass` + field. + - Convert `TARGET_LONG_BITS` definition to a `CPUClass` + helper function. + - Convert `TARGET_PAGE_*` definitions to runtime variables. + +6. **Support Variable Page Sizes** (3 Weeks) + + - Convert all MTTCG-enabled targets to use + `TARGET_PAGE_BITS_VARY`. Since multiple vCPUs can share the + same accelerator and page sizes, create a set of compatible + vCPUs that share the same accelerator context. + +7. **Compile `accel/tcg/` System Once** (2 Weeks) + + - Modify the build system to compile the TCG accelerator system + only once, making it target-agnostic and shared across + architectures. + +8. **Modularize TCG Frontends** (4 Weeks) + + - Build TCG frontends as separate modules (e.g., + `libtcg-$ARCH.so`). Such a library should be exclusively + composed of compilation units in `target/$ARCH`. Modularize + the set of TCG frontends for all targets. + +9. **Introduce `AccelCpuCluster` Concept** (5 Weeks) + + - Implement `AccelCpuCluster` to group similar vCPUs that share + the same accelerator (TCG) state. This allows for efficient + handling of vCPUs with common properties. + +10. **Make `SoftFloat` context configuratble at runtime** (4 Weeks) + + - Convert per-target static definitions of SoftFloat context to + runtime ones, possibly allowing re-use between vCPUs. + +11. **Convert Non-MTTCG Targets** (4 Weeks) + + - Extend the previous steps to include non-MTTCG targets, + ensuring that all targets can benefit from the new + architecture. + +12. **Make SoftFloat code target-agnostic** (2 Weeks) + + - Convert per-target static configurations to a runtime configurable + context. + +### Phase 2: Hardware Cleanups and API Separation (Estimated Duration: 3 Months) + +**Objective**: Refine hardware models and APIs to support +modularization and multi-architecture support. + +1. **Reduce `NEED_CPU_H` in Headers** (2 Weeks) + + - Split headers containing `NEED_CPU_H` into target-specific + and target-agnostic ones. Use the `-common.h` suffix for + target-agnostic headers and `-target.h` for target-specific + ones. + +2. **Eliminate Randomness in QOM Paths** (2 Weeks) + + - Remove auto-incremented IDs and random numbers in device QOM + paths to keep the composition tree reproducible. Ensure that + devices have stable and predictable QOM paths. + +3. **Clarify QOM/QDev Methods and Lifecycle** (4 Weeks) + + - Clarify which QDev methods belong to the QOM layer, such as + 'realize'. Define a unified life cycle model for QOM objects. + This includes adding an extra "wiring" phase in the QDev state + machine to link parts together. + +4. **Move Target-Specific Code from `hw/` to `target/hw/`** (3 + Weeks) + + - When a device model is target-specific, move it from `hw/` to + `target/$ARCH/hw/`. This simplifies the `hw/` <-> `target/` + access problem and reduces dependencies. + +5. **Enforce API Separation Between `hw/` and `target/`** (4 Weeks) + + - Remove non-QDev API calls from `target/` to `hw/` (and + vice-versa), restricting to QOM, QDev, IRQ, and Clock APIs. + Enforce a clearer API separation to avoid direct calls and + enable both `hw/` and `target/` components to be built as + modules. + +6. **Restrict Use of Global CPU Variables in `hw/`** (4 Weeks) + + - Prohibit the use of global CPU variables like `current_cpu` + and `first_cpu` in hardware code (`hw/`). Instead, use + `CpuCluster[]` or pass explicit references. This prevents + casting errors and architecture-specific assumptions in common + code. + +7. **Merge SysBus Methods into QDev** (3 Weeks) + + - Merge most of the `SysBus` methods into the more generic QDev + layer, starting with GPIO. This unification simplifies the + device model and reduces redundancy. + +8. **Remove `ISA_BUS` Singleton** (4 Weeks) + + - Remove the `ISA_BUS` singleton. Replace ISA bus-dependent + devices with stubs or rework them to not rely on the singleton, + simplifying the bus model. + +9. **Optimize QOM Cast Macros** (2 Weeks) + + - Reduce and optimize the use of QOM cast macros in non-API + code and callbacks to improve performance and maintainability. + +10. **Consider Power/Clock/Reset Tree API** (1 Week) + + - Brainstorm whether a Power/Clock/Reset tree API is needed, + which could be helpful later with device composition. + +### Phase 3: Unify Binaries and Simplify Machine Modes (Estimated Duration: 2 Months) + +**Objective**: Move towards a single QEMU binary supporting +multiple architectures and modes. + +1. **Rework Bus Models and Ownership** (4 Weeks) + + - Remove device ownership on multi-controller buses, making the + single owner the `MachineState`. Modify bus APIs to make the + bus parameter explicit, allowing devices to be accessed on any + bus. + +2. **Eliminate `qdev_get_machine()` Usage** (5 Weeks) + + - Refactor code to remove reliance on `qdev_get_machine()`. + Fields in `MachineState` that are target-specific (e.g., + `dtb`, `kernel`, `initrd`, `NumaState`, `CpuTopology`) should + become per-vCPU or per cluster. + +3. **Add Machine Property for Mode Selection** (2 Weeks) + + - Add a machine property to specify whether to start in 32-bit + or 64-bit mode and forward the property down to devices. This + allows for flexible configuration of machines that can operate + in different modes. + +4. **Unify Word Size Support** (4 Weeks) + + - Modify the build system to produce a single binary supporting + both 32-bit and 64-bit architectures, unifying word size + support. + +5. **Unify Endianness Support** (4 Weeks) + + - Similarly, unify endianness support to produce a single binary + capable of handling both big-endian and little-endian + architectures. + +6. **Investigate GDB Stub Restrictions** (2 Weeks) + + - Investigate any restrictions in the GDB stub that may prevent + multi-architecture support or affect debugging capabilities. + +7. **Disallow Unattached QOM Devices** (2 Weeks) + + - Do not allow unattached QOM devices. Ensure that all devices + are part of the composition tree, eliminating the + `/machine/unattached/` orphanage problem. + +8. **Remove Target-Specific QMP and Monitor Commands** (2 Weeks) + + - Remove target-specific QMP and monitor commands to provide a + consistent interface across architectures. + +### Phase 4: Prepare for Declarative Machine Configuration (Estimated Duration: 3 Months) + +**Objective**: Lay the groundwork for dynamic, data-driven machine +configurations. + +1. **Static QAPI-Based QOM Properties** (8 Weeks) + + - Integrate the configuration interface into the QAPI schema + using static QAPI-based QOM properties. Restrict dynamic + read-write properties to debugging and introspection. Make + configuration properties read-only after realization. + +2. **Clarify Machine Phases Enum** (2 Weeks) + + - Clarify and extend the Machine Phases enum. Determine if a + `PHASE_MACHINE_HARDWARE_WIRED` is needed before + `PHASE_MACHINE_INITIALIZED` to be able to wire cyclic IRQs. + +3. **Simplify QEMU Startup Sequence** (12 Weeks) + + - Rework the QEMU startup code to greatly simplify it. Extract + or factor out the CLI API (human-facing), keeping the minimum + needed to start a QMP interface (machine-facing). + +4. **Enhance QOM Object Registration and Filtering** (3 Weeks) + + - Register all QOM objects at startup with appropriate + filtering. Implement functionality to list devices filtered + per machine. Handle cases where devices are only available in + certain modes or architectures. + +5. **Merge `UserCreatable` with QOM** (2 Weeks) + + - Integrate the `UserCreatable` class into QOM for streamlined + object creation. If eventually all devices can be created by + QMP when building a machine, all devices thus inherit the + `UserCreatable` class. + +6. **Early Availability of QMP** (4 Weeks) + + - Make QMP available earlier in the startup sequence for initial + configuration. Provide a minimal CLI that is stable and + sufficient to start QMP for further configuration. + +### Phase 5: Implement Dynamic Machines (Estimated Duration: 4 Months) + +**Objective**: Enable the creation of machines dynamically at +runtime using a declarative approach. + +1. **Develop Declarative Language or DSL** (12 Weeks) + + - Create a Domain-Specific Language (DSL) or use existing + formats to describe machine configurations. Devices are + expressed as DSL, enabling composable machines. + +2. **Refactor Machine Creation Process** (8 Weeks) + + - Rewrite the machine initialization code to consume declarative + configurations. Transition from statically built machine + models to a dynamic, data-driven configuration system. + +3. **Simplify and Modularize Startup Code** (4 Weeks) + + - Streamline the startup sequence to accommodate dynamic + machine creation. Simplify the QEMU startup sequence, focusing + on modularity and flexibility. + +4. **Provide Simplified Configuration Profiles** (3 Weeks) + + - Introduce predefined machine profiles for common use cases + (e.g., `q35-minimal.cfg`, `q35-recommended.cfg`, + `q35-simple.cfg`). Allow users to select a configuration that + best fits their scenario without needing to fine-tune every + parameter. + +5. **Address Migration Compatibility** (4 Weeks) + + - Ensure that data-driven machine definitions support migration + compatibility. Handle the "sensible defaults" problem by + providing default settings for RAM size, CPU model, etc. + +6. **Acknowledge and Plan for New Challenges** (Ongoing) + + - Recognize and address new challenges arising from the + transition to data-driven machines, such as managing the QOM + object life cycle, handling complex device compositions, and + ensuring performance and stability. + +### Phase 6: Enhance User Experience for Human Users (Estimated Duration: 2 Months) + +**Objective**: Provide a user-friendly interface and configuration +profiles for human users. + +1. **Develop Human-Focused Frontend** (4 Weeks) + + - Create a frontend that abstracts complexity and provides a + simple CLI for human users. Recognize the divergent needs + between human users and management applications, and offer a + solution that caters to both. + +2. **Implement Simplified Configuration Profiles** (3 Weeks) + + - Offer predefined profiles (e.g., minimal, recommended, + simple) for different use cases. Allow users to easily select + configurations without dealing with low-level details. + +3. **Documentation and User Guides** (2 Weeks) + + - Provide comprehensive documentation to assist users in + transitioning to the new interface. Include examples and + guides for common tasks. + +4. **Feedback and Iteration** (Ongoing) + + - Gather user feedback and iteratively improve the frontend and + profiles. Ensure that the solution meets the needs of the + community. + +### Total Estimated Duration: Approximately 20 Months + +**Note**: These time estimates are approximate and may vary based +on the size of the development team, complexity of tasks, and +unforeseen challenges. + +## Conclusion + +By addressing the identified problems through this structured +roadmap, QEMU can evolve into a more modular, flexible, and user- +friendly platform capable of concurrent multi-architecture +emulation. The proposed changes will enable dynamic machine +creation, improve code maintainability, and enhance the user +experience for both management applications and human users. + +Implementing this roadmap requires coordinated effort and +collaboration among the development community. While new challenges +will undoubtedly arise, the long-term benefits of a more adaptable +and future-proof QEMU far outweigh the initial complexities.