From patchwork Thu Sep 20 22:29:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608813 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3315415A6 for ; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24A372DF19 for ; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 188D12E002; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85DB22DF19 for ; Thu, 20 Sep 2018 22:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26B728E0006; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F4BE8E0001; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E3F68E0006; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id C420D8E0001 for ; Thu, 20 Sep 2018 18:29:48 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id 90-v6so4980158pla.18 for ; Thu, 20 Sep 2018 15:29:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=PLvctc37G0Nx2a3mlCkOPpDTxZJGpZA6T4lg+7tnz6M=; b=h42bxWOFj3ki6jyudUTnG22StdGzLtw1Gol+fF/5Q7D4IcreWgje/rhvsy5FOK9Iad pWPSMLb4HmPaA85vbSrIqjzJ8DMwIp+wFAORaBrtJLpWR7q4TlYZj1/CyHt29io7twPN yvc7OMPWPZXBUCyEgZvipSkqnAGbX0YheugbXAeWz2j7nQ+txC3Y0h5K6kjaig5Igv55 P2ZjsJJAfHCtTmN/1NzTOdNpz9U1e8jYJM9wm5gRGWWIT5Fpkrw97HTIukzo+78JE9qj 2DIB/mV9wWMpFWosigNuvWBDZ/FvsQaB7mf6PIEz3mayFjoQsnCB/2H3MXg0HY9Q3U28 coqg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CapGHA50j8myM0GX8E5Wd3LxtAgmZ1ry7pgPa3j6TIe9+yZR3a iXiH2eSlkDO6R+0ko+1Q+qfQxkmzDRF+8eFNWzcycli0l5xUUJgT0g01h+pafES00KBLHzZ5q5t /baiPJop5egQ7bfRBWRrr+LkcrktD/JPPNioE3ZlQGLqDOkDyC6Wi5XK2+y4dtGnIJA== X-Received: by 2002:a63:904a:: with SMTP id a71-v6mr38313564pge.339.1537482588449; Thu, 20 Sep 2018 15:29:48 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYyL7jh/p50+EHyLq9x/EYWLvbu9JAWLzqK7PyIeYNgRxigMwFJYc+mc0K2QxDPi9MMha7a X-Received: by 2002:a63:904a:: with SMTP id a71-v6mr38313513pge.339.1537482587160; Thu, 20 Sep 2018 15:29:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482587; cv=none; d=google.com; s=arc-20160816; b=n8Y7O8jbU2TLv3YhDl5donJ3Ci85nx6X29dUSpPYyE0gT6qR7dUT8ObS9eBEbKoQEz A8MKOqDmubY/pz7rmZunszAfnK4vRnCmEs1KuS9bUyvce1ErFw+pDEfSUleicOVkJhQL uAO2N84T24owsTOHS5ihkqsQ5ZtRlgndM4RMuGyUAC+eKdF6jucryJ6QkdAiqgLttqY7 riRDKp/Wazlal3oKUlnDJIlVQIpv+VFMBBPDXijoSaQsq3V80o3PapVCGcA63Q7+d3f9 bA5rVSTHaSyMHk0/jH+A2ehozQOvzsGd0CNLDXWseIzls4ckFiS74oAj91u9KhAnwmba 5lDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=PLvctc37G0Nx2a3mlCkOPpDTxZJGpZA6T4lg+7tnz6M=; b=c+LASBHUMClZya1tmabXFQcANY97uXrbi7wAYc1dI2qXdGaRY4gjWNbI66rjVcOxzk 9SmOJbB6EUkr5D07rlxpslo7nkM5U967z0ppz3P4e9otjv8QCaW2WQE83jX9dVyrv0/i LapEVTabXLV9hyWsO68Xwdat/bSJLN78oxWB8UOWL9EjAd1hwg0uiMdQbnkUqNY4+T0i oEhTfEYhBy2967M188StgKL+gvI/DApGiodqeDRNqWY94YgYbia+qCoBD9p3qJtFpguT 4o/B4fCU+ud/Lfl8l+PERuYnttOtft0EMX81Ujxw2DcMBFQ7tzSBKrC+i1ZtRQdFISur 6SmA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id t9-v6si23641437pgo.68.2018.09.20.15.29.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:29:47 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:29:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85286849" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:29:45 -0700 Subject: [PATCH v4 4/5] async: Add support for queueing on specific node From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:29:45 -0700 Message-ID: <20180920222938.19464.34102.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces two new variants of the async_schedule_ functions that allow scheduling on a specific node. These functions are async_schedule_on and async_schedule_on_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. The original functions were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE. The main motivation behind this is to address the need to be able to schedule NVDIMM init work on specific NUMA nodes in order to improve performance of memory initialization. One additional change I made is I dropped the "extern" from the function prototypes in the async.h kernel header since they aren't needed. Signed-off-by: Alexander Duyck --- include/linux/async.h | 20 +++++++++++++++++--- kernel/async.c | 36 +++++++++++++++++++++++++----------- 2 files changed, 42 insertions(+), 14 deletions(-) diff --git a/include/linux/async.h b/include/linux/async.h index 6b0226bdaadc..9878b99cbb01 100644 --- a/include/linux/async.h +++ b/include/linux/async.h @@ -14,6 +14,7 @@ #include #include +#include typedef u64 async_cookie_t; typedef void (*async_func_t) (void *data, async_cookie_t cookie); @@ -37,9 +38,22 @@ struct async_domain { struct async_domain _name = { .pending = LIST_HEAD_INIT(_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule(async_func_t func, void *data); -extern async_cookie_t async_schedule_domain(async_func_t func, void *data, - struct async_domain *domain); +async_cookie_t async_schedule_on(async_func_t func, void *data, int node); +async_cookie_t async_schedule_on_domain(async_func_t func, void *data, int node, + struct async_domain *domain); + +static inline async_cookie_t async_schedule(async_func_t func, void *data) +{ + return async_schedule_on(func, data, NUMA_NO_NODE); +} + +static inline async_cookie_t +async_schedule_domain(async_func_t func, void *data, + struct async_domain *domain) +{ + return async_schedule_on_domain(func, data, NUMA_NO_NODE, domain); +} + void async_unregister_domain(struct async_domain *domain); extern void async_synchronize_full(void); extern void async_synchronize_full_domain(struct async_domain *domain); diff --git a/kernel/async.c b/kernel/async.c index a893d6170944..1d7ce81c1949 100644 --- a/kernel/async.c +++ b/kernel/async.c @@ -56,6 +56,7 @@ synchronization with the async_synchronize_full() function, before returning #include #include #include +#include #include "workqueue_internal.h" @@ -149,8 +150,11 @@ static void async_run_entry_fn(struct work_struct *work) wake_up(&async_done); } -static async_cookie_t __async_schedule(async_func_t func, void *data, struct async_domain *domain) +static async_cookie_t __async_schedule(async_func_t func, void *data, + struct async_domain *domain, + int node) { + int cpu = WORK_CPU_UNBOUND; struct async_entry *entry; unsigned long flags; async_cookie_t newcookie; @@ -194,30 +198,40 @@ static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy /* mark that this task has queued an async job, used by module init */ current->flags |= PF_USED_ASYNC; + /* guarantee cpu_online_mask doesn't change during scheduling */ + get_online_cpus(); + + if (node >= 0 && node < MAX_NUMNODES && node_online(node)) + cpu = cpumask_any_and(cpumask_of_node(node), cpu_online_mask); + /* schedule for execution */ - queue_work(system_unbound_wq, &entry->work); + queue_work_on(cpu, system_unbound_wq, &entry->work); + + put_online_cpus(); return newcookie; } /** - * async_schedule - schedule a function for asynchronous execution + * async_schedule_on - schedule a function for asynchronous execution * @func: function to execute asynchronously * @data: data pointer to pass to the function + * @node: NUMA node to complete the work on * * Returns an async_cookie_t that may be used for checkpointing later. * Note: This function may be called from atomic or non-atomic contexts. */ -async_cookie_t async_schedule(async_func_t func, void *data) +async_cookie_t async_schedule_on(async_func_t func, void *data, int node) { - return __async_schedule(func, data, &async_dfl_domain); + return __async_schedule(func, data, &async_dfl_domain, node); } -EXPORT_SYMBOL_GPL(async_schedule); +EXPORT_SYMBOL_GPL(async_schedule_on); /** - * async_schedule_domain - schedule a function for asynchronous execution within a certain domain + * async_schedule_on_domain - schedule a function for asynchronous execution within a certain domain * @func: function to execute asynchronously * @data: data pointer to pass to the function + * @node: NUMA node to complete the work on * @domain: the domain * * Returns an async_cookie_t that may be used for checkpointing later. @@ -226,12 +240,12 @@ async_cookie_t async_schedule(async_func_t func, void *data) * synchronization domain is specified via @domain. Note: This function * may be called from atomic or non-atomic contexts. */ -async_cookie_t async_schedule_domain(async_func_t func, void *data, - struct async_domain *domain) +async_cookie_t async_schedule_on_domain(async_func_t func, void *data, int node, + struct async_domain *domain) { - return __async_schedule(func, data, domain); + return __async_schedule(func, data, domain, node); } -EXPORT_SYMBOL_GPL(async_schedule_domain); +EXPORT_SYMBOL_GPL(async_schedule_on_domain); /** * async_synchronize_full - synchronize all asynchronous function calls