From patchwork Thu May 13 14:18:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 12255873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7E78C433B4 for ; Thu, 13 May 2021 14:20:56 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5C06F610A0 for ; Thu, 13 May 2021 14:20:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C06F610A0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=a5G0OhwEplopEIDPvJh5aAPJQgbJ10MhRCaQottJqtw=; b=AHuUaKfMK4spqb9HNS4uv316ap /9Kz97G2g85+vAKuLdd52pEf+r18D+5uc9md8/GwJCdWiiAkdQan5PCUil89YqEjNP/wVWzKdBMyN 4RTHrlvUyIK7pATSdfaiAVz0j6N/7FB4NXdFA1K5rLUZHdLN9Yr0u4gBS54i+ZDV83dL9APWtOWn4 TuxXX5dhvJiwFDPegmmYe1L2/RFq64ihWEHlno2UdCvWaRnODQJuqXGqJXUZAYwXIPmtknLKbVf0m oTuFIf5jOuY6GqZ97BfvC9EjRchHuDHgWWtvd1MhdCtKu9qfmhrRpAU/PK+Xkw0si7lzpZP+hKcOp pi5Ams5Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lhCAj-005ikK-7V; Thu, 13 May 2021 14:18:41 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lhCAg-005ik7-Ci for linux-arm-kernel@desiato.infradead.org; Thu, 13 May 2021 14:18:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:Reply-To:Content-ID :Content-Description:In-Reply-To:References; bh=jt1sez6gv8kg6+baNglAhmD6c6XH+eUZPh3hfftngT0=; b=lvtOBr6KWC8u9CFQGVZ/4ZO4pg 3Voq5X2UngTe3Og8uQNVPqP7qBaSLlrMaq7Oq9/13/yrSfqcUXLjsbxHDbzbt2t0PlI5xjE6qv/St IviTmdAHOXytHF67s4oarY/OcV/Tr8f9qxjkdFlMc73O/303xE0QeKdtRtu62dX/x+ycZfE+8frFd 2+DeBtLFM2UWG9RJTvqUkRB7N/ct/nlMg/pHldoH3959WT7fcIF2GO21Nqs0xbgoN0fp+tkJuYZM9 3YGKHZ1z7E1y8RS2xBESb5DJm3xGIZINbTjLylB8uCxaDXDQZ5XjhWOolIjnvw7/gOxvyhpaI0KmG lP4Asgig==; Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lhCAd-00BJIe-IQ for linux-arm-kernel@lists.infradead.org; Thu, 13 May 2021 14:18:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:Reply-To:Content-ID :Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To: Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=jt1sez6gv8kg6+baNglAhmD6c6XH+eUZPh3hfftngT0=; b=Wlo37D/AoLAJR8/p/2CNAiTFAI NIuB3iDejAZkP4w3BqP08cgrNnqxYuyC/85V9ilyGqdRlggKkqDvrRpoqQKp7YyOtMHjtOGgtw5I4 TtD0cTqGkSqdhdiJ+f0TcSNGO0E4DivMR27Y2dBBJ5dfeE6qh8PvH5+BcdDlocCUDZiuIDDqZoTvj n+ZyipFli3Mjuk5tyAql8LQA+uhrCDORRDKVh3zjeKgNeTOLMI5l2vFQ9wy8R99iOOTLUPuBEZUt6 dnzMR3IW3TcpEC17JJzFSb6UWKD2Z2855rTzzlLPGA4f9w2iLdgtDGoMt/0jFBk62r39/9wQoirva NYMrXRWA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:56674 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lhCAV-0006In-ET; Thu, 13 May 2021 15:18:27 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.92) (envelope-from ) id 1lhCAV-0002yb-50; Thu, 13 May 2021 15:18:27 +0100 From: Russell King To: Bjorn Helgaas Cc: Arnd Bergmann , linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org Subject: [PATCH] PCI: dynamically map ECAM regions MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 13 May 2021 15:18:27 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210513_071835_639988_FC2F55C8 X-CRM114-Status: GOOD ( 21.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempting to boot 32-bit ARM kernels under QEMU's 3.x virt models fails when we have more than 512M of RAM in the model as we run out of vmalloc space for the PCI ECAM regions. This failure will be silent when running libvirt, as the console in that situation is a PCI device. In this configuration, the kernel maps the whole ECAM, which QEMU sets up for 256 buses, even when maybe only seven buses are in use. Each bus uses 1M of ECAM space, and ioremap() adds an additional guard page between allocations. The kernel vmap allocator will align these regions to 512K, resulting in each mapping eating 1.5M of vmalloc space. This means we need 384M of vmalloc space just to map all of these, which is very wasteful of resources. Fix this by only mapping the ECAM for buses we are going to be using. In my setups, this is around seven buses in most guests, which is 10.5M of vmalloc space - way smaller than the 384M that would otherwise be required. This also means that the kernel can boot without forcing extra RAM into highmem with the vmalloc= argument, or decreasing the virtual RAM available to the guest. Suggested-by: Arnd Bergmann Signed-off-by: Russell King Reviewed-by: Arnd Bergmann --- drivers/pci/ecam.c | 54 ++++++++++++++++++++++++++++++++++------ include/linux/pci-ecam.h | 1 + 2 files changed, 47 insertions(+), 8 deletions(-) diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c index d2a1920bb055..1c40d2506aef 100644 --- a/drivers/pci/ecam.c +++ b/drivers/pci/ecam.c @@ -32,7 +32,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev, struct pci_config_window *cfg; unsigned int bus_range, bus_range_max, bsz; struct resource *conflict; - int i, err; + int err; if (busr->start > busr->end) return ERR_PTR(-EINVAL); @@ -50,6 +50,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev, cfg->busr.start = busr->start; cfg->busr.end = busr->end; cfg->busr.flags = IORESOURCE_BUS; + cfg->bus_shift = bus_shift; bus_range = resource_size(&cfg->busr); bus_range_max = resource_size(cfgres) >> bus_shift; if (bus_range > bus_range_max) { @@ -77,13 +78,6 @@ struct pci_config_window *pci_ecam_create(struct device *dev, cfg->winp = kcalloc(bus_range, sizeof(*cfg->winp), GFP_KERNEL); if (!cfg->winp) goto err_exit_malloc; - for (i = 0; i < bus_range; i++) { - cfg->winp[i] = - pci_remap_cfgspace(cfgres->start + i * bsz, - bsz); - if (!cfg->winp[i]) - goto err_exit_iomap; - } } else { cfg->win = pci_remap_cfgspace(cfgres->start, bus_range * bsz); if (!cfg->win) @@ -129,6 +123,44 @@ void pci_ecam_free(struct pci_config_window *cfg) } EXPORT_SYMBOL_GPL(pci_ecam_free); +static int pci_ecam_add_bus(struct pci_bus *bus) +{ + struct pci_config_window *cfg = bus->sysdata; + unsigned int bsz = 1 << cfg->bus_shift; + unsigned int busn = bus->number; + phys_addr_t start; + + if (!per_bus_mapping) + return 0; + + if (busn < cfg->busr.start || busn > cfg->busr.end) + return -EINVAL; + + busn -= cfg->busr.start; + start = cfg->res.start + busn * bsz; + + cfg->winp[busn] = pci_remap_cfgspace(start, bsz); + if (!cfg->winp[busn]) + return -ENOMEM; + + return 0; +} + +static void pci_ecam_remove_bus(struct pci_bus *bus) +{ + struct pci_config_window *cfg = bus->sysdata; + unsigned int busn = bus->number; + + if (!per_bus_mapping || busn < cfg->busr.start || busn > cfg->busr.end) + return; + + busn -= cfg->busr.start; + if (cfg->winp[busn]) { + iounmap(cfg->winp[busn]); + cfg->winp[busn] = NULL; + } +} + /* * Function to implement the pci_ops ->map_bus method */ @@ -167,6 +199,8 @@ EXPORT_SYMBOL_GPL(pci_ecam_map_bus); /* ECAM ops */ const struct pci_ecam_ops pci_generic_ecam_ops = { .pci_ops = { + .add_bus = pci_ecam_add_bus, + .remove_bus = pci_ecam_remove_bus, .map_bus = pci_ecam_map_bus, .read = pci_generic_config_read, .write = pci_generic_config_write, @@ -178,6 +212,8 @@ EXPORT_SYMBOL_GPL(pci_generic_ecam_ops); /* ECAM ops for 32-bit access only (non-compliant) */ const struct pci_ecam_ops pci_32b_ops = { .pci_ops = { + .add_bus = pci_ecam_add_bus, + .remove_bus = pci_ecam_remove_bus, .map_bus = pci_ecam_map_bus, .read = pci_generic_config_read32, .write = pci_generic_config_write32, @@ -187,6 +223,8 @@ const struct pci_ecam_ops pci_32b_ops = { /* ECAM ops for 32-bit read only (non-compliant) */ const struct pci_ecam_ops pci_32b_read_ops = { .pci_ops = { + .add_bus = pci_ecam_add_bus, + .remove_bus = pci_ecam_remove_bus, .map_bus = pci_ecam_map_bus, .read = pci_generic_config_read32, .write = pci_generic_config_write, diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h index 65d3d83015c3..944da75ff25c 100644 --- a/include/linux/pci-ecam.h +++ b/include/linux/pci-ecam.h @@ -55,6 +55,7 @@ struct pci_ecam_ops { struct pci_config_window { struct resource res; struct resource busr; + unsigned int bus_shift; void *priv; const struct pci_ecam_ops *ops; union {