From patchwork Wed Sep 11 17:59:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 11141677 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C8E5924 for ; Wed, 11 Sep 2019 18:01:31 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CF43C2081B for ; Wed, 11 Sep 2019 18:01:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="D8oXZg4J" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF43C2081B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i86u4-0000Fi-EN; Wed, 11 Sep 2019 17:59:40 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i86u3-0000Fa-Gc for xen-devel@lists.xenproject.org; Wed, 11 Sep 2019 17:59:39 +0000 X-Inumbo-ID: eb87bd42-d4bd-11e9-b76c-bc764e2007e4 Received: from mail-lf1-x142.google.com (unknown [2a00:1450:4864:20::142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id eb87bd42-d4bd-11e9-b76c-bc764e2007e4; Wed, 11 Sep 2019 17:59:38 +0000 (UTC) Received: by mail-lf1-x142.google.com with SMTP id l11so17188586lfk.6 for ; Wed, 11 Sep 2019 10:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=5C95kf9c0fiXXe9eHC4b3N15laYfsWrjL1Mm0/xpy9o=; b=D8oXZg4JRXEH+3wLUmFuEvsO+H3SkWsssJ799qsoqqzKlN+pDBDHy8x60MTObIYoM+ u4+z7PY6GSaWSl7D0aDDkR60K6ktksA4ul1idlUnEr4MU4KHA+A4Wy7XP+ceByMkv1vk XXIC1P/CQCvhJpt1Yk1dZqEzfqslFZxnPbRnQPTbADgOpv+XcHSsKgnHOMpSNDlO91r0 yEdJr2z4VELHsTqJuxhwvNE87CpTaPUWVuycM968cdrAAkxJ62xVMsXVpyoqbOTyr66n x2hvRwx5w7EKjFJgwcUShRPvNm0sq0HNNvums8wtzDHGCovFkiuZkKZsn8OIt15rnK1C J4Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=5C95kf9c0fiXXe9eHC4b3N15laYfsWrjL1Mm0/xpy9o=; b=LWCyGUrlsdnQOQaXbKgnHdCt6Zic0JDh6KmD9NSRzxUdLHihZ0Giyyo4y3OMQ8XquF 6VJLK18VJuZy5asR7hkyiLe04kUpmoEXLLzKAG4EyJneRvP59SbjSJsMki9smGN4WSVo f9c1k+3tb/hK0QqF2FA7FBE5qB++HFBQDf4wRKRna9Ca0rHCgK8uUtnMOI9J1eu/teQq Dtunnu/JKiq3Q8VeECZCcUJgpDt/5MmMv4/7qnPnjOOsZHA0M27vc6ZrDMHnU+e1qVbX 5HU9QsmHqf9oVjFDohpkc2XQLEPlqRMm4B4hyNF4wOR6hctvq10btGB7z08hI7SG3gY0 Z6Aw== X-Gm-Message-State: APjAAAVsPvYXz1ToWJlZpgoFhI8PeE4WRuQo02bvTrfPwLGAiSxSPUBI Kpqq9w0qanz7ZchzDNvszB3/LBy9 X-Google-Smtp-Source: APXvYqw//SwD3HVJns5qGXd44qI/1SMIOvkaY54K++FAAdkxlJq9zlFIXW2d1zwXvAjEErzKcKyJNw== X-Received: by 2002:ac2:44b9:: with SMTP id c25mr26248817lfm.112.1568224776981; Wed, 11 Sep 2019 10:59:36 -0700 (PDT) Received: from otyshchenko.kyiv.epam.com (ll-74.141.223.85.sovam.net.ua. [85.223.141.74]) by smtp.gmail.com with ESMTPSA id u1sm5294335lfi.83.2019.09.11.10.59.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 11 Sep 2019 10:59:36 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Date: Wed, 11 Sep 2019 20:59:30 +0300 Message-Id: <1568224770-25402-1-git-send-email-olekstysh@gmail.com> X-Mailer: git-send-email 2.7.4 Subject: [Xen-devel] [RFC PATCH V3] xen/arm: Restrict "p2m_ipa_bits" according to the IOMMU requirements X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Oleksandr Tyshchenko , julien.grall@arm.com, sstabellini@kernel.org, Volodymyr_Babchuk@epam.com MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" From: Oleksandr Tyshchenko There is a strict requirement for the IOMMU which wants to share the P2M table with the CPU. The IOMMU's Stage-2 input size must be equal to the P2M IPA size. It is not a problem when the IOMMU can support all values the CPU supports. In that case, the IOMMU driver would just use any "p2m_ipa_bits" value as is. But, there are cases when not. In order to make P2M sharing possible on the platforms which IOMMUs have a limitation in maximum Stage-2 input size introduce the following logic. First initialize the IOMMU subsystem and gather requirements regarding the maximum IPA bits supported by each IOMMU device to figure out the minimum value among them. In the P2M code, take into the account the IOMMU requirements and choose suitable "pa_range" according to the restricted "p2m_ipa_bits". Signed-off-by: Oleksandr Tyshchenko CC: Julien Grall --- Still RFC: 1. Patch assumes that IPMMU support is already in. 2. Not checked for the SMMU. Changes RFC V2 [2] -> RFC V3: - Check in setup_virt_paging() that the "restricted" P2M IPA size is at least 40-bit - Modify logic in setup_virt_paging() a bit to make it "IOMMU-agnostic" - Clarify comments in code, add some explanations - Avoid using the term "IOMMU" in P2M code where possible Changes RFC V1 [1] -> RFC V2 [2]: - Don't update p2m_ipa_bits by the IOMMU drivers directly, introduce p2m_restrict_ipa_bits() - Clarify patch subject/description - Add more comments to code - Check for equivalent "pabits" in setup_virt_paging() - Remove ASSERTs from the SMMU and IPMMU drivers [1] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg02078.html [2] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg02237.html --- xen/arch/arm/p2m.c | 41 ++++++++++++++++++++++++++++---- xen/arch/arm/setup.c | 9 +++++-- xen/drivers/passthrough/arm/ipmmu-vmsa.c | 18 ++------------ xen/drivers/passthrough/arm/smmu.c | 11 +++------ xen/include/asm-arm/p2m.h | 9 +++++++ 5 files changed, 58 insertions(+), 30 deletions(-) diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c index 2374e92..d5e2539 100644 --- a/xen/arch/arm/p2m.c +++ b/xen/arch/arm/p2m.c @@ -34,7 +34,11 @@ static unsigned int __read_mostly max_vmid = MAX_VMID_8_BIT; #define P2M_ROOT_PAGES (1<mm64.pa_range < pa_range ) - pa_range = info->mm64.pa_range; + + /* + * Restrict "p2m_ipa_bits" if needed. As P2M table is always configured + * with IPA bits == PA bits, compare against "pabits". + */ + if ( pa_range_info[info->mm64.pa_range].pabits < p2m_ipa_bits ) + p2m_ipa_bits = pa_range_info[info->mm64.pa_range].pabits; /* Set a flag if the current cpu does not support 16 bit VMIDs. */ if ( info->mm64.vmid_bits != MM64_VMID_16_BITS_SUPPORT ) @@ -1988,6 +2011,16 @@ void __init setup_virt_paging(void) if ( !vmid_8_bit ) max_vmid = MAX_VMID_16_BIT; + /* Choose suitable "pa_range" according to the resulted "p2m_ipa_bits". */ + for ( i = 0; i < ARRAY_SIZE(pa_range_info); i++ ) + { + if ( p2m_ipa_bits == pa_range_info[i].pabits ) + { + pa_range = i; + break; + } + } + /* pa_range is 4 bits, but the defined encodings are only 3 bits */ if ( pa_range >= ARRAY_SIZE(pa_range_info) || !pa_range_info[pa_range].pabits ) panic("Unknown encoding of ID_AA64MMFR0_EL1.PARange %x\n", pa_range); diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index 51a6677..0e628bc 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -936,12 +936,17 @@ void __init start_xen(unsigned long boot_phys_offset, printk("Brought up %ld CPUs\n", (long)num_online_cpus()); /* TODO: smp_cpus_done(); */ - setup_virt_paging(); - + /* + * The IOMMU subsystem must be initialized before P2M as we need + * to gather requirements regarding the maximum IPA bits supported by + * each IOMMU device. + */ rc = iommu_setup(); if ( !iommu_enabled && rc != -ENODEV ) panic("Couldn't configure correctly all the IOMMUs."); + setup_virt_paging(); + do_initcalls(); /* diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c index ea29e91..efbda39 100644 --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c @@ -844,22 +844,8 @@ static int ipmmu_probe(struct dt_device_node *node) goto out; } - /* - * As 4-level translation table is not supported in IPMMU, we need - * to check IPA size used for P2M table beforehand to be sure it is - * 3-level and the IPMMU will be able to use it. - * - * TODO: First initialize the IOMMU and gather the requirements and - * then initialize the P2M. In the P2M code, take into the account - * the IOMMU requirements and restrict "pa_range" if necessary. - */ - if ( IPMMU_MAX_P2M_IPA_BITS < p2m_ipa_bits ) - { - printk(XENLOG_ERR "ipmmu: P2M IPA size is not supported (P2M=%u IPMMU=%u)!\n", - p2m_ipa_bits, IPMMU_MAX_P2M_IPA_BITS); - ret = -ENODEV; - goto out; - } + /* Set maximum Stage-2 input size supported by the IPMMU. */ + p2m_restrict_ipa_bits(IPMMU_MAX_P2M_IPA_BITS); irq = platform_get_irq(node, 0); if ( irq < 0 ) diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c index 8ae986a..701fe9c 100644 --- a/xen/drivers/passthrough/arm/smmu.c +++ b/xen/drivers/passthrough/arm/smmu.c @@ -2198,14 +2198,9 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu) size = arm_smmu_id_size_to_bits((id >> ID2_IAS_SHIFT) & ID2_IAS_MASK); smmu->s1_output_size = min_t(unsigned long, PHYS_MASK_SHIFT, size); - /* Xen: Stage-2 input size has to match p2m_ipa_bits. */ - if (size < p2m_ipa_bits) { - dev_err(smmu->dev, - "P2M IPA size not supported (P2M=%u SMMU=%lu)!\n", - p2m_ipa_bits, size); - return -ENODEV; - } - smmu->s2_input_size = p2m_ipa_bits; + /* Xen: Set maximum Stage-2 input size supported by the SMMU. */ + p2m_restrict_ipa_bits(size); + smmu->s2_input_size = size; #if 0 /* Stage-2 input size limited due to pgd allocation (PTRS_PER_PGD) */ #ifdef CONFIG_64BIT diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h index f970c53..0a377ea 100644 --- a/xen/include/asm-arm/p2m.h +++ b/xen/include/asm-arm/p2m.h @@ -165,6 +165,15 @@ void p2m_altp2m_check(struct vcpu *v, uint16_t idx) /* Not supported on ARM. */ } +/* + * Helper to restrict "p2m_ipa_bits" according the external entity + * (e.g. IOMMU) requirements. + * + * Each corresponding driver should report the maximum IPA bits + * (Stage-2 input size) it can support. + */ +void p2m_restrict_ipa_bits(unsigned int ipa_bits); + /* Second stage paging setup, to be called on all CPUs */ void setup_virt_paging(void);