From patchwork Thu May 7 23:39:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 11535133 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5785F1668 for ; Thu, 7 May 2020 23:55:35 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 31D5120735 for ; Thu, 7 May 2020 23:55:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 31D5120735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 7E8491181F21A; Thu, 7 May 2020 16:53:34 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=134.134.136.100; helo=mga07.intel.com; envelope-from=dan.j.williams@intel.com; receiver= Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 125AC1181F205 for ; Thu, 7 May 2020 16:53:31 -0700 (PDT) IronPort-SDR: vLDGQKuOLXDHtWghw28wm8jKRAaEZifiwBf4Uulak9GWluS4uO9q/e0M2vHnfOmKnlzJz7XCLY CXy15NL+OsTQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 May 2020 16:55:31 -0700 IronPort-SDR: ka+TMz8D+wZX+awfx1R69MtBaIv58Pi9zH33FF2popoTwztxUaRb2E+qUdIlSDpbThN0/cL/rW f8d3NyYtFNIA== X-IronPort-AV: E=Sophos;i="5.73,365,1583222400"; d="scan'208";a="285189036" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 May 2020 16:55:31 -0700 Subject: [PATCH v2] ACPI: Drop rcu usage for MMIO mappings From: Dan Williams To: rafael.j.wysocki@intel.com Date: Thu, 07 May 2020 16:39:20 -0700 Message-ID: <158889473309.2292982.18007035454673387731.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Message-ID-Hash: 7YS6VA53YSHVIAVWM7CVUQCL72NTNTJT X-Message-ID-Hash: 7YS6VA53YSHVIAVWM7CVUQCL72NTNTJT X-MailFrom: dan.j.williams@intel.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: stable@vger.kernel.org, Len Brown , Borislav Petkov , James Morse , Erik Kaneda , Myron Stowe , "Rafael J. Wysocki" , Andy Shevchenko , linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-nvdimm@lists.01.org X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: Recently a performance problem was reported for a process invoking a non-trival ASL program. The method call in this case ends up repetitively triggering a call path like: acpi_ex_store acpi_ex_store_object_to_node acpi_ex_write_data_to_field acpi_ex_insert_into_field acpi_ex_write_with_update_rule acpi_ex_field_datum_io acpi_ex_access_region acpi_ev_address_space_dispatch acpi_ex_system_memory_space_handler acpi_os_map_cleanup.part.14 _synchronize_rcu_expedited.constprop.89 schedule The end result of frequent synchronize_rcu_expedited() invocation is tiny sub-millisecond spurts of execution where the scheduler freely migrates this apparently sleepy task. The overhead of frequent scheduler invocation multiplies the execution time by a factor of 2-3X. For example, performance improves from 16 minutes to 7 minutes for a firmware update procedure across 24 devices. Perhaps the rcu usage was intended to allow for not taking a sleeping lock in the acpi_os_{read,write}_memory() path which ostensibly could be called from an APEI NMI error interrupt? Neither rcu_read_lock() nor ioremap() are interrupt safe, so add a WARN_ONCE() to validate that rcu was not serving as a mechanism to avoid direct calls to ioremap(). Even the original implementation had a spin_lock_irqsave(), but that is not NMI safe. APEI itself already has some concept of avoiding ioremap() from interrupt context (see erst_exec_move_data()), if the new warning triggers it means that APEI either needs more instrumentation like that to pre-emptively fail, or more infrastructure to arrange for pre-mapping the resources it needs in NMI context. Cc: Fixes: 620242ae8c3d ("ACPI: Maintain a list of ACPI memory mapped I/O remappings") Cc: Len Brown Cc: Borislav Petkov Cc: Ira Weiny Cc: James Morse Cc: Erik Kaneda Cc: Myron Stowe Cc: "Rafael J. Wysocki" Cc: Andy Shevchenko Signed-off-by: Dan Williams Reported-by: kernel test robot --- Changes since v1 [1]: - Actually cc: the most important list for ACPI changes (Rafael) - Cleanup unnecessary variable initialization (Andy) Link: https://lore.kernel.org/linux-nvdimm/158880834905.2183490.15616329469420234017.stgit@dwillia2-desk3.amr.corp.intel.com/ drivers/acpi/osl.c | 117 +++++++++++++++++++++++++--------------------------- 1 file changed, 57 insertions(+), 60 deletions(-) diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c index 762c5d50b8fe..a44b75aac5d0 100644 --- a/drivers/acpi/osl.c +++ b/drivers/acpi/osl.c @@ -214,13 +214,13 @@ acpi_physical_address __init acpi_os_get_root_pointer(void) return pa; } -/* Must be called with 'acpi_ioremap_lock' or RCU read lock held. */ static struct acpi_ioremap * acpi_map_lookup(acpi_physical_address phys, acpi_size size) { struct acpi_ioremap *map; - list_for_each_entry_rcu(map, &acpi_ioremaps, list, acpi_ioremap_lock_held()) + lockdep_assert_held(&acpi_ioremap_lock); + list_for_each_entry(map, &acpi_ioremaps, list) if (map->phys <= phys && phys + size <= map->phys + map->size) return map; @@ -228,7 +228,6 @@ acpi_map_lookup(acpi_physical_address phys, acpi_size size) return NULL; } -/* Must be called with 'acpi_ioremap_lock' or RCU read lock held. */ static void __iomem * acpi_map_vaddr_lookup(acpi_physical_address phys, unsigned int size) { @@ -263,7 +262,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size) { struct acpi_ioremap *map; - list_for_each_entry_rcu(map, &acpi_ioremaps, list, acpi_ioremap_lock_held()) + lockdep_assert_held(&acpi_ioremap_lock); + list_for_each_entry(map, &acpi_ioremaps, list) if (map->virt <= virt && virt + size <= map->virt + map->size) return map; @@ -360,7 +360,7 @@ void __iomem __ref map->size = pg_sz; map->refcount = 1; - list_add_tail_rcu(&map->list, &acpi_ioremaps); + list_add_tail(&map->list, &acpi_ioremaps); out: mutex_unlock(&acpi_ioremap_lock); @@ -374,20 +374,13 @@ void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size) } EXPORT_SYMBOL_GPL(acpi_os_map_memory); -/* Must be called with mutex_lock(&acpi_ioremap_lock) */ -static unsigned long acpi_os_drop_map_ref(struct acpi_ioremap *map) -{ - unsigned long refcount = --map->refcount; - - if (!refcount) - list_del_rcu(&map->list); - return refcount; -} - -static void acpi_os_map_cleanup(struct acpi_ioremap *map) +static void acpi_os_drop_map_ref(struct acpi_ioremap *map) { - synchronize_rcu_expedited(); + lockdep_assert_held(&acpi_ioremap_lock); + if (--map->refcount > 0) + return; acpi_unmap(map->phys, map->virt); + list_del(&map->list); kfree(map); } @@ -408,7 +401,6 @@ static void acpi_os_map_cleanup(struct acpi_ioremap *map) void __ref acpi_os_unmap_iomem(void __iomem *virt, acpi_size size) { struct acpi_ioremap *map; - unsigned long refcount; if (!acpi_permanent_mmap) { __acpi_unmap_table(virt, size); @@ -422,11 +414,8 @@ void __ref acpi_os_unmap_iomem(void __iomem *virt, acpi_size size) WARN(true, PREFIX "%s: bad address %p\n", __func__, virt); return; } - refcount = acpi_os_drop_map_ref(map); + acpi_os_drop_map_ref(map); mutex_unlock(&acpi_ioremap_lock); - - if (!refcount) - acpi_os_map_cleanup(map); } EXPORT_SYMBOL_GPL(acpi_os_unmap_iomem); @@ -461,7 +450,6 @@ void acpi_os_unmap_generic_address(struct acpi_generic_address *gas) { u64 addr; struct acpi_ioremap *map; - unsigned long refcount; if (gas->space_id != ACPI_ADR_SPACE_SYSTEM_MEMORY) return; @@ -477,11 +465,8 @@ void acpi_os_unmap_generic_address(struct acpi_generic_address *gas) mutex_unlock(&acpi_ioremap_lock); return; } - refcount = acpi_os_drop_map_ref(map); + acpi_os_drop_map_ref(map); mutex_unlock(&acpi_ioremap_lock); - - if (!refcount) - acpi_os_map_cleanup(map); } EXPORT_SYMBOL(acpi_os_unmap_generic_address); @@ -700,55 +685,71 @@ int acpi_os_read_iomem(void __iomem *virt_addr, u64 *value, u32 width) return 0; } +static void __iomem *acpi_os_rw_map(acpi_physical_address phys_addr, + unsigned int size, bool *did_fallback) +{ + void __iomem *virt_addr; + + if (WARN_ONCE(in_interrupt(), "ioremap in interrupt context\n")) + return NULL; + + /* Try to use a cached mapping and fallback otherwise */ + *did_fallback = false; + mutex_lock(&acpi_ioremap_lock); + virt_addr = acpi_map_vaddr_lookup(phys_addr, size); + if (virt_addr) + return virt_addr; + mutex_unlock(&acpi_ioremap_lock); + + virt_addr = acpi_os_ioremap(phys_addr, size); + *did_fallback = true; + + return virt_addr; +} + +static void acpi_os_rw_unmap(void __iomem *virt_addr, bool did_fallback) +{ + if (did_fallback) { + /* in the fallback case no lock is held */ + iounmap(virt_addr); + return; + } + + mutex_unlock(&acpi_ioremap_lock); +} + acpi_status acpi_os_read_memory(acpi_physical_address phys_addr, u64 *value, u32 width) { - void __iomem *virt_addr; unsigned int size = width / 8; - bool unmap = false; + bool did_fallback = false; + void __iomem *virt_addr; u64 dummy; int error; - rcu_read_lock(); - virt_addr = acpi_map_vaddr_lookup(phys_addr, size); - if (!virt_addr) { - rcu_read_unlock(); - virt_addr = acpi_os_ioremap(phys_addr, size); - if (!virt_addr) - return AE_BAD_ADDRESS; - unmap = true; - } - + virt_addr = acpi_os_rw_map(phys_addr, size, &did_fallback); + if (!virt_addr) + return AE_BAD_ADDRESS; if (!value) value = &dummy; error = acpi_os_read_iomem(virt_addr, value, width); BUG_ON(error); - if (unmap) - iounmap(virt_addr); - else - rcu_read_unlock(); - + acpi_os_rw_unmap(virt_addr, did_fallback); return AE_OK; } acpi_status acpi_os_write_memory(acpi_physical_address phys_addr, u64 value, u32 width) { - void __iomem *virt_addr; unsigned int size = width / 8; - bool unmap = false; + bool did_fallback = false; + void __iomem *virt_addr; - rcu_read_lock(); - virt_addr = acpi_map_vaddr_lookup(phys_addr, size); - if (!virt_addr) { - rcu_read_unlock(); - virt_addr = acpi_os_ioremap(phys_addr, size); - if (!virt_addr) - return AE_BAD_ADDRESS; - unmap = true; - } + virt_addr = acpi_os_rw_map(phys_addr, size, &did_fallback); + if (!virt_addr) + return AE_BAD_ADDRESS; switch (width) { case 8: @@ -767,11 +768,7 @@ acpi_os_write_memory(acpi_physical_address phys_addr, u64 value, u32 width) BUG(); } - if (unmap) - iounmap(virt_addr); - else - rcu_read_unlock(); - + acpi_os_rw_unmap(virt_addr, did_fallback); return AE_OK; } From patchwork Wed Jun 10 12:21:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 11597727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 097DB90 for ; Wed, 10 Jun 2020 12:23:42 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E59AB2086A for ; Wed, 10 Jun 2020 12:23:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E59AB2086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id A7D23100A302C; Wed, 10 Jun 2020 05:23:41 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=79.96.170.134; helo=cloudserver094114.home.pl; envelope-from=rjw@rjwysocki.net; receiver= Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0EBAF100A457E for ; Wed, 10 Jun 2020 05:23:38 -0700 (PDT) Received: from 89-64-83-71.dynamic.chello.pl (89.64.83.71) (HELO kreacher.localnet) by serwer1319399.home.pl (79.96.170.134) with SMTP (IdeaSmtpServer 0.83.415) id c86131f5cc9a7435; Wed, 10 Jun 2020 14:23:37 +0200 From: "Rafael J. Wysocki" To: Dan Williams Subject: [RFT][PATCH 2/3] ACPICA: Remove unused memory mappings on interpreter exit Date: Wed, 10 Jun 2020 14:21:48 +0200 Message-ID: <3974162.pZLctmZ5Iv@kreacher> In-Reply-To: <318372766.6LKUBsbRXE@kreacher> References: <158889473309.2292982.18007035454673387731.stgit@dwillia2-desk3.amr.corp.intel.com> <318372766.6LKUBsbRXE@kreacher> MIME-Version: 1.0 Message-ID-Hash: JSKEI3GOM2VHJV4TBFGJE3M4U54E3XSM X-Message-ID-Hash: JSKEI3GOM2VHJV4TBFGJE3M4U54E3XSM X-MailFrom: rjw@rjwysocki.net X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation CC: Erik Kaneda , rafael.j.wysocki@intel.com, Len Brown , Borislav Petkov , James Morse , Myron Stowe , Andy Shevchenko , linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-nvdimm@lists.01.org, Bob Moore X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: From: "Rafael J. Wysocki" For transient memory opregions that are created dynamically under the namespace and interpreter mutexes and go away quickly, there still is the problem that removing their memory mappings may take significant time and so doing that while holding the mutexes should be avoided. For example, unmapping a chunk of memory associated with a memory opregion in Linux involves running synchronize_rcu_expedited() which really should not be done with the namespace mutex held. To address that problem, notice that the unused memory mappings left behind by the "dynamic" opregions that went away need not be unmapped right away when the opregion is deactivated. Instead, they may be unmapped when exiting the interpreter, after the namespace and interpreter mutexes have been dropped (there's one more place dealing with opregions in the debug code that can be treated analogously). Accordingly, change acpi_ev_system_memory_region_setup() to put the unused mappings into a global list instead of unmapping them right away and add acpi_ev_system_release_memory_mappings() to be called when leaving the interpreter in order to unmap the unused memory mappings in the global list (which is protected by the namespace mutex). Signed-off-by: Rafael J. Wysocki --- drivers/acpi/acpica/acevents.h | 2 ++ drivers/acpi/acpica/dbtest.c | 3 ++ drivers/acpi/acpica/evrgnini.c | 51 ++++++++++++++++++++++++++++++++-- drivers/acpi/acpica/exutils.c | 3 ++ drivers/acpi/acpica/utxface.c | 23 +++++++++++++++ include/acpi/acpixf.h | 1 + 6 files changed, 80 insertions(+), 3 deletions(-) diff --git a/drivers/acpi/acpica/acevents.h b/drivers/acpi/acpica/acevents.h index 79f292687bd6..463eb9124765 100644 --- a/drivers/acpi/acpica/acevents.h +++ b/drivers/acpi/acpica/acevents.h @@ -197,6 +197,8 @@ acpi_ev_execute_reg_method(union acpi_operand_object *region_obj, u32 function); /* * evregini - Region initialization and setup */ +void acpi_ev_system_release_memory_mappings(void); + acpi_status acpi_ev_system_memory_region_setup(acpi_handle handle, u32 function, diff --git a/drivers/acpi/acpica/dbtest.c b/drivers/acpi/acpica/dbtest.c index 6db44a5ac786..7dac6dae5c48 100644 --- a/drivers/acpi/acpica/dbtest.c +++ b/drivers/acpi/acpica/dbtest.c @@ -8,6 +8,7 @@ #include #include "accommon.h" #include "acdebug.h" +#include "acevents.h" #include "acnamesp.h" #include "acpredef.h" #include "acinterp.h" @@ -768,6 +769,8 @@ acpi_db_test_field_unit_type(union acpi_operand_object *obj_desc) acpi_ut_release_mutex(ACPI_MTX_NAMESPACE); acpi_ut_release_mutex(ACPI_MTX_INTERPRETER); + acpi_ev_system_release_memory_mappings(); + bit_length = obj_desc->common_field.bit_length; byte_length = ACPI_ROUND_BITS_UP_TO_BYTES(bit_length); diff --git a/drivers/acpi/acpica/evrgnini.c b/drivers/acpi/acpica/evrgnini.c index 48a5e6eaf9b9..946c4eef054d 100644 --- a/drivers/acpi/acpica/evrgnini.c +++ b/drivers/acpi/acpica/evrgnini.c @@ -16,6 +16,52 @@ #define _COMPONENT ACPI_EVENTS ACPI_MODULE_NAME("evrgnini") +#ifdef ACPI_OS_MAP_MEMORY_FAST_PATH +static struct acpi_mem_mapping *unused_memory_mappings; + +/******************************************************************************* + * + * FUNCTION: acpi_ev_system_release_memory_mappings + * + * PARAMETERS: None + * + * RETURN: None + * + * DESCRIPTION: Release all of the unused memory mappings in the queue + * under the interpreter mutex. + * + ******************************************************************************/ +void acpi_ev_system_release_memory_mappings(void) +{ + struct acpi_mem_mapping *mapping; + + ACPI_FUNCTION_TRACE(acpi_ev_system_release_memory_mappings); + + acpi_ut_acquire_mutex(ACPI_MTX_NAMESPACE); + + while (unused_memory_mappings) { + mapping = unused_memory_mappings; + unused_memory_mappings = mapping->next; + + acpi_ut_release_mutex(ACPI_MTX_NAMESPACE); + + acpi_os_unmap_memory(mapping->logical_address, mapping->length); + ACPI_FREE(mapping); + + acpi_ut_acquire_mutex(ACPI_MTX_NAMESPACE); + } + + acpi_ut_release_mutex(ACPI_MTX_NAMESPACE); + + return_VOID; +} +#else /* !ACPI_OS_MAP_MEMORY_FAST_PATH */ +void acpi_ev_system_release_memory_mappings(void) +{ + return_VOID; +} +#endif /* !ACPI_OS_MAP_MEMORY_FAST_PATH */ + /******************************************************************************* * * FUNCTION: acpi_ev_system_memory_region_setup @@ -60,9 +106,8 @@ acpi_ev_system_memory_region_setup(acpi_handle handle, while (local_region_context->first_mapping) { mapping = local_region_context->first_mapping; local_region_context->first_mapping = mapping->next; - acpi_os_unmap_memory(mapping->logical_address, - mapping->length); - ACPI_FREE(mapping); + mapping->next = unused_memory_mappings; + unused_memory_mappings = mapping; } #endif } diff --git a/drivers/acpi/acpica/exutils.c b/drivers/acpi/acpica/exutils.c index 8fefa6feac2f..516d67664392 100644 --- a/drivers/acpi/acpica/exutils.c +++ b/drivers/acpi/acpica/exutils.c @@ -25,6 +25,7 @@ #include #include "accommon.h" +#include "acevents.h" #include "acinterp.h" #include "amlcode.h" @@ -106,6 +107,8 @@ void acpi_ex_exit_interpreter(void) "Could not release AML Interpreter mutex")); } + acpi_ev_system_release_memory_mappings(); + return_VOID; } diff --git a/drivers/acpi/acpica/utxface.c b/drivers/acpi/acpica/utxface.c index ca7c9f0144ef..d972696be846 100644 --- a/drivers/acpi/acpica/utxface.c +++ b/drivers/acpi/acpica/utxface.c @@ -11,6 +11,7 @@ #include #include "accommon.h" +#include "acevents.h" #include "acdebug.h" #define _COMPONENT ACPI_UTILITIES @@ -244,6 +245,28 @@ acpi_status acpi_purge_cached_objects(void) ACPI_EXPORT_SYMBOL(acpi_purge_cached_objects) +/***************************************************************************** + * + * FUNCTION: acpi_release_unused_memory_mappings + * + * PARAMETERS: None + * + * RETURN: None + * + * DESCRIPTION: Remove memory mappings that are not used any more. + * + ****************************************************************************/ +void acpi_release_unused_memory_mappings(void) +{ + ACPI_FUNCTION_TRACE(acpi_release_unused_memory_mappings); + + acpi_ev_system_release_memory_mappings(); + + return_VOID; +} + +ACPI_EXPORT_SYMBOL(acpi_release_unused_memory_mappings) + /***************************************************************************** * * FUNCTION: acpi_install_interface diff --git a/include/acpi/acpixf.h b/include/acpi/acpixf.h index 1dc8d262035b..8d2cc02257ed 100644 --- a/include/acpi/acpixf.h +++ b/include/acpi/acpixf.h @@ -449,6 +449,7 @@ ACPI_EXTERNAL_RETURN_STATUS(acpi_status acpi_size length, struct acpi_pld_info **return_buffer)) +ACPI_EXTERNAL_RETURN_VOID(void acpi_release_unused_memory_mappings(void)) /* * ACPI table load/unload interfaces From patchwork Wed Jun 10 12:22:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 11597725 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 192DA90 for ; Wed, 10 Jun 2020 12:23:41 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 00F1C2074B for ; Wed, 10 Jun 2020 12:23:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00F1C2074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 92CAD100A3CF3; Wed, 10 Jun 2020 05:23:40 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=79.96.170.134; helo=cloudserver094114.home.pl; envelope-from=rjw@rjwysocki.net; receiver= Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id EAE61100A457D for ; Wed, 10 Jun 2020 05:23:38 -0700 (PDT) Received: from 89-64-83-71.dynamic.chello.pl (89.64.83.71) (HELO kreacher.localnet) by serwer1319399.home.pl (79.96.170.134) with SMTP (IdeaSmtpServer 0.83.415) id 085cf89194671cbf; Wed, 10 Jun 2020 14:23:35 +0200 From: "Rafael J. Wysocki" To: Dan Williams Subject: [RFT][PATCH 3/3] ACPI: OSL: Define ACPI_OS_MAP_MEMORY_FAST_PATH() Date: Wed, 10 Jun 2020 14:22:50 +0200 Message-ID: <6458983.dlBdKaB8z0@kreacher> In-Reply-To: <318372766.6LKUBsbRXE@kreacher> References: <158889473309.2292982.18007035454673387731.stgit@dwillia2-desk3.amr.corp.intel.com> <318372766.6LKUBsbRXE@kreacher> MIME-Version: 1.0 Message-ID-Hash: DT25AQOOWAIGSFMAECDWNVDMIFABJAS7 X-Message-ID-Hash: DT25AQOOWAIGSFMAECDWNVDMIFABJAS7 X-MailFrom: rjw@rjwysocki.net X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation CC: Erik Kaneda , rafael.j.wysocki@intel.com, Len Brown , Borislav Petkov , James Morse , Myron Stowe , Andy Shevchenko , linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-nvdimm@lists.01.org, Bob Moore X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: From: "Rafael J. Wysocki" Define the ACPI_OS_MAP_MEMORY_FAST_PATH() macro to allow acpi_ex_system_memory_space_handler() to avoid memory unmapping overhead by deferring the unmap operations to the point when the AML interpreter is exited after removing the operation region that held the memory mappings which are not used any more. That macro, when called on a knwon-existing memory mapping, causes the reference counter of that mapping in the OS layer to be incremented and returns a pointer representing the virtual address of the start of the mapped memory area without really mapping it, so the first subsequent unmap operation on it will only decrement the reference counter. Signed-off-by: Rafael J. Wysocki --- drivers/acpi/osl.c | 67 +++++++++++++++++++++++-------- include/acpi/platform/aclinuxex.h | 4 ++ 2 files changed, 55 insertions(+), 16 deletions(-) diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c index 762c5d50b8fe..b75f3a17776f 100644 --- a/drivers/acpi/osl.c +++ b/drivers/acpi/osl.c @@ -302,21 +302,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr) iounmap(vaddr); } -/** - * acpi_os_map_iomem - Get a virtual address for a given physical address range. - * @phys: Start of the physical address range to map. - * @size: Size of the physical address range to map. - * - * Look up the given physical address range in the list of existing ACPI memory - * mappings. If found, get a reference to it and return a pointer to it (its - * virtual address). If not found, map it, add it to that list and return a - * pointer to it. - * - * During early init (when acpi_permanent_mmap has not been set yet) this - * routine simply calls __acpi_map_table() to get the job done. - */ -void __iomem __ref -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size) +static void __iomem __ref *__acpi_os_map_iomem(acpi_physical_address phys, + acpi_size size, bool fast_path) { struct acpi_ioremap *map; void __iomem *virt; @@ -328,8 +315,12 @@ void __iomem __ref return NULL; } - if (!acpi_permanent_mmap) + if (!acpi_permanent_mmap) { + if (WARN_ON(fast_path)) + return NULL; + return __acpi_map_table((unsigned long)phys, size); + } mutex_lock(&acpi_ioremap_lock); /* Check if there's a suitable mapping already. */ @@ -339,6 +330,11 @@ void __iomem __ref goto out; } + if (fast_path) { + mutex_unlock(&acpi_ioremap_lock); + return NULL; + } + map = kzalloc(sizeof(*map), GFP_KERNEL); if (!map) { mutex_unlock(&acpi_ioremap_lock); @@ -366,6 +362,25 @@ void __iomem __ref mutex_unlock(&acpi_ioremap_lock); return map->virt + (phys - map->phys); } + +/** + * acpi_os_map_iomem - Get a virtual address for a given physical address range. + * @phys: Start of the physical address range to map. + * @size: Size of the physical address range to map. + * + * Look up the given physical address range in the list of existing ACPI memory + * mappings. If found, get a reference to it and return a pointer representing + * its virtual address. If not found, map it, add it to that list and return a + * pointer representing its virtual address. + * + * During early init (when acpi_permanent_mmap has not been set yet) call + * __acpi_map_table() to obtain the mapping. + */ +void __iomem __ref *acpi_os_map_iomem(acpi_physical_address phys, + acpi_size size) +{ + return __acpi_os_map_iomem(phys, size, false); +} EXPORT_SYMBOL_GPL(acpi_os_map_iomem); void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size) @@ -374,6 +389,24 @@ void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size) } EXPORT_SYMBOL_GPL(acpi_os_map_memory); +/** + * acpi_os_map_memory_fast_path - Fast-path physical-to-virtual address mapping. + * @phys: Start of the physical address range to map. + * @size: Size of the physical address range to map. + * + * Look up the given physical address range in the list of existing ACPI memory + * mappings. If found, get a reference to it and return a pointer representing + * its virtual address. If not found, return NULL. + * + * During early init (when acpi_permanent_mmap has not been set yet) log a + * warning and return NULL. + */ +void __ref *acpi_os_map_memory_fast_path(acpi_physical_address phys, + acpi_size size) +{ + return __acpi_os_map_iomem(phys, size, true); +} + /* Must be called with mutex_lock(&acpi_ioremap_lock) */ static unsigned long acpi_os_drop_map_ref(struct acpi_ioremap *map) { @@ -1571,6 +1604,8 @@ acpi_status acpi_release_memory(acpi_handle handle, struct resource *res, return acpi_walk_namespace(ACPI_TYPE_REGION, handle, level, acpi_deactivate_mem_region, NULL, res, NULL); + + acpi_release_unused_memory_mappings(); } EXPORT_SYMBOL_GPL(acpi_release_memory); diff --git a/include/acpi/platform/aclinuxex.h b/include/acpi/platform/aclinuxex.h index 04f88f2de781..1d8be4ac9ef9 100644 --- a/include/acpi/platform/aclinuxex.h +++ b/include/acpi/platform/aclinuxex.h @@ -139,6 +139,10 @@ static inline void acpi_os_terminate_debugger(void) * OSL interfaces added by Linux */ +void *acpi_os_map_memory_fast_path(acpi_physical_address where, acpi_size length); + +#define ACPI_OS_MAP_MEMORY_FAST_PATH(a, s) acpi_os_map_memory_fast_path(a, s) + #endif /* __KERNEL__ */ #endif /* __ACLINUXEX_H__ */ From patchwork Mon Jun 22 14:02:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 11617863 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D18CF13A0 for ; Mon, 22 Jun 2020 14:03:22 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B9B86206E2 for ; Mon, 22 Jun 2020 14:03:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9B86206E2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 6E43D10FC729F; Mon, 22 Jun 2020 07:03:22 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=79.96.170.134; helo=cloudserver094114.home.pl; envelope-from=rjw@rjwysocki.net; receiver= Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id B292F10FC61BF for ; Mon, 22 Jun 2020 07:03:19 -0700 (PDT) Received: from 89-64-85-91.dynamic.chello.pl (89.64.85.91) (HELO kreacher.localnet) by serwer1319399.home.pl (79.96.170.134) with SMTP (IdeaSmtpServer 0.83.415) id add3d0c7dbc26dbb; Mon, 22 Jun 2020 16:03:16 +0200 From: "Rafael J. Wysocki" To: Dan Williams , Erik Kaneda Subject: [RFT][PATCH v2 4/4] ACPI: OSL: Implement acpi_os_map_memory_fast_path() Date: Mon, 22 Jun 2020 16:02:44 +0200 Message-ID: <39838855.e8c3ya2Sh3@kreacher> In-Reply-To: <2713141.s8EVnczdoM@kreacher> References: <158889473309.2292982.18007035454673387731.stgit@dwillia2-desk3.amr.corp.intel.com> <2713141.s8EVnczdoM@kreacher> MIME-Version: 1.0 Message-ID-Hash: BU3VTR5TICBGQIZGARF7SEMCT4BIFTRW X-Message-ID-Hash: BU3VTR5TICBGQIZGARF7SEMCT4BIFTRW X-MailFrom: rjw@rjwysocki.net X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation CC: rafael.j.wysocki@intel.com, Len Brown , Borislav Petkov , James Morse , Myron Stowe , Andy Shevchenko , linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-nvdimm@lists.01.org, Bob Moore X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: From: "Rafael J. Wysocki" Add acpi_os_map_memory_fast_path() and set ACPI_USE_FAST_PATH_MAPPING to allow acpi_ex_system_memory_space_handler() to avoid unnecessary memory mapping and unmapping overhead by retaining all memory mappings created by it until the memory opregions associated with them go away. Signed-off-by: Rafael J. Wysocki --- drivers/acpi/osl.c | 65 +++++++++++++++++++++++-------- include/acpi/platform/aclinuxex.h | 4 ++ 2 files changed, 53 insertions(+), 16 deletions(-) diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c index 28863d908fa8..89554ec9a178 100644 --- a/drivers/acpi/osl.c +++ b/drivers/acpi/osl.c @@ -306,21 +306,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr) iounmap(vaddr); } -/** - * acpi_os_map_iomem - Get a virtual address for a given physical address range. - * @phys: Start of the physical address range to map. - * @size: Size of the physical address range to map. - * - * Look up the given physical address range in the list of existing ACPI memory - * mappings. If found, get a reference to it and return a pointer to it (its - * virtual address). If not found, map it, add it to that list and return a - * pointer to it. - * - * During early init (when acpi_permanent_mmap has not been set yet) this - * routine simply calls __acpi_map_table() to get the job done. - */ -void __iomem __ref -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size) +static void __iomem __ref *__acpi_os_map_iomem(acpi_physical_address phys, + acpi_size size, bool fast_path) { struct acpi_ioremap *map; void __iomem *virt; @@ -332,8 +319,12 @@ void __iomem __ref return NULL; } - if (!acpi_permanent_mmap) + if (!acpi_permanent_mmap) { + if (WARN_ON(fast_path)) + return NULL; + return __acpi_map_table((unsigned long)phys, size); + } mutex_lock(&acpi_ioremap_lock); /* Check if there's a suitable mapping already. */ @@ -343,6 +334,11 @@ void __iomem __ref goto out; } + if (fast_path) { + mutex_unlock(&acpi_ioremap_lock); + return NULL; + } + map = kzalloc(sizeof(*map), GFP_KERNEL); if (!map) { mutex_unlock(&acpi_ioremap_lock); @@ -370,6 +366,25 @@ void __iomem __ref mutex_unlock(&acpi_ioremap_lock); return map->virt + (phys - map->phys); } + +/** + * acpi_os_map_iomem - Get a virtual address for a given physical address range. + * @phys: Start of the physical address range to map. + * @size: Size of the physical address range to map. + * + * Look up the given physical address range in the list of existing ACPI memory + * mappings. If found, get a reference to it and return a pointer representing + * its virtual address. If not found, map it, add it to that list and return a + * pointer representing its virtual address. + * + * During early init (when acpi_permanent_mmap has not been set yet) call + * __acpi_map_table() to obtain the mapping. + */ +void __iomem __ref *acpi_os_map_iomem(acpi_physical_address phys, + acpi_size size) +{ + return __acpi_os_map_iomem(phys, size, false); +} EXPORT_SYMBOL_GPL(acpi_os_map_iomem); void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size) @@ -378,6 +393,24 @@ void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size) } EXPORT_SYMBOL_GPL(acpi_os_map_memory); +/** + * acpi_os_map_memory_fast_path - Fast-path physical-to-virtual address mapping. + * @phys: Start of the physical address range to map. + * @size: Size of the physical address range to map. + * + * Look up the given physical address range in the list of existing ACPI memory + * mappings. If found, get a reference to it and return a pointer representing + * its virtual address. If not found, return NULL. + * + * During early init (when acpi_permanent_mmap has not been set yet) log a + * warning and return NULL. + */ +void __ref *acpi_os_map_memory_fast_path(acpi_physical_address phys, + acpi_size size) +{ + return __acpi_os_map_iomem(phys, size, true); +} + /* Must be called with mutex_lock(&acpi_ioremap_lock) */ static bool acpi_os_drop_map_ref(struct acpi_ioremap *map, bool defer) { diff --git a/include/acpi/platform/aclinuxex.h b/include/acpi/platform/aclinuxex.h index e13f364d6c69..89c387449425 100644 --- a/include/acpi/platform/aclinuxex.h +++ b/include/acpi/platform/aclinuxex.h @@ -143,6 +143,10 @@ void acpi_os_release_unused_mappings(void); #define ACPI_USE_DEFERRED_UNMAPPING +void *acpi_os_map_memory_fast_path(acpi_physical_address where, acpi_size length); + +#define ACPI_USE_FAST_PATH_MAPPING + #endif /* __KERNEL__ */ #endif /* __ACLINUXEX_H__ */