From patchwork Thu Mar 21 20:01:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10864249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0E8613B5 for ; Thu, 21 Mar 2019 20:03:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C51102A43A for ; Thu, 21 Mar 2019 20:03:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B94552A44A; Thu, 21 Mar 2019 20:03:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 208D42A43A for ; Thu, 21 Mar 2019 20:03:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 833D86B0006; Thu, 21 Mar 2019 16:02:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7BBFE6B0007; Thu, 21 Mar 2019 16:02:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60D066B0008; Thu, 21 Mar 2019 16:02:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 2653C6B0006 for ; Thu, 21 Mar 2019 16:02:59 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id h69so1011305pfd.21 for ; Thu, 21 Mar 2019 13:02:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ZXflYn009/8re0hSvilHhjOsKnJ5N944EbzSGROrxW8=; b=XSVJ7LFUKYA/yWGO5bWWZYRyKej3dmSsHN1haKTbbjpU3BcgefJbn9pQ2udl5XzZ+l 10IVmAqWC0tks0xXm8Nf93A2qRxlQqnhkp6YxXJUNRREFoaL8z2BB8kZMP/DJLPUICLT GTplQZOGPhB/q5nxle2zkP0YvMhRKlOSxMukGpJY3oH2RZZ+Fa0g1hi2xn5KpsjqmZZi g6v53ZWMODwmQlI8PIp5VO79zN2k3gqtbqrJqtP+xXsG2MWO6cwiQCRJK+mWH6ZGmBhI Eutqs+/k13fD2Y9EH4+v+hBA6WQd3fjaw7UoT83pxci/s1LHUULBYmVsm0rCLR4BieJa 3uUA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXv3hdko6JLVZ+j+W/bZpLoRWGl9xsE3gZzTF8YeNu/x/rfWIFs bmC5YLmLcovkQDFsNTF7f0PI0C3hzxZqrE/HCwU83TAXKBSNzSHU9k/y9WmZQxGc0CmHBAnULT4 YHLj25AZSkJJFKBO8gWsRRjqBHCxPxrpfHGSAd/Rlv+sr9MBJ5cgb4dVfI5oMhJcjBQ== X-Received: by 2002:a17:902:848e:: with SMTP id c14mr5398989plo.339.1553198578677; Thu, 21 Mar 2019 13:02:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqzPTkoq4Hw5/6XvqxuGt43MX/ISDDB/hFtQ3QTTgWMOHj8vYISeE1IiDSBBt7e+WzC00Ms0 X-Received: by 2002:a17:902:848e:: with SMTP id c14mr5398857plo.339.1553198577161; Thu, 21 Mar 2019 13:02:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553198577; cv=none; d=google.com; s=arc-20160816; b=U9WasszvzveInC8tWnM4aNxXC16PQSPbvoEVMm5HUMqGLEPAvR6fQTTQMr5FLN9Tsu /Jj2Tz/ufaQBcf1OWS1yFxeXw9pEiInhN7yemKMsrqY9rZelOFb4k1vFm2MjxVImVIQE pKPerpjKGhCsC5txgV1ym5cOlZGnpuFkoKeY6mHK1K970PexZrZZ2qBhNM0S4DYp9Io6 DzgmiNuxSNq7GNbL8r4zkDfC9EKYx0uVgwYHrtkDeZUTONIdrXPtHryM7PuAscUhIf3j oTL5037eM2EpCnRZmKUsUAPVuWz+d3t8bFn9YKeZFgbgd2mt8KKCwZcJGwyffDVmHRJO DUXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ZXflYn009/8re0hSvilHhjOsKnJ5N944EbzSGROrxW8=; b=zZDdSLi35a3M/pAK0S89ua2foos9naayPSrEy1ceNp+xosaFQ/45RKXR5Z3OPDd+MO 6h0ceKhhbXI3MaxonFFZJ0v1c40eiysBJfrMGWBCKWGQEAZKbJ7mGD1Br2csEyR4J2C3 in3Hn4xLsPeO5eu3/21VwMnzl7K8uifnWuB7BQR/VGTQewu/GLE+VJ0w4bLkI3L4Va36 +u5maIS/uF7x7kQiyRcKAhzLSjhpatKfa1dFjWUY6Zu0rKfALFE4gMSDVQnZmAaHobHQ TME8WbSzGp7CsR7ZbRC6U3gR5sOqWugEK8bVP1uQ/gvG2aAhGP8gUG/wt0IkkYu23zml lxtg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id r25si4703408pfd.91.2019.03.21.13.02.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Mar 2019 13:02:57 -0700 (PDT) Received-SPF: pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 13:02:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="309246233" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2019 13:02:56 -0700 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org Cc: Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 1/5] node: Define and export memory migration path Date: Thu, 21 Mar 2019 14:01:53 -0600 Message-Id: <20190321200157.29678-2-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20190321200157.29678-1-keith.busch@intel.com> References: <20190321200157.29678-1-keith.busch@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Prepare for the kernel to auto-migrate pages to other memory nodes with a user defined node migration table. A user may create a single target for each NUMA node to enable the kernel to do NUMA page migrations instead of simply reclaiming colder pages. A node with no target is a "terminal node", so reclaim acts normally there. The migration target does not fundamentally _need_ to be a single node, but this implementation starts there to limit complexity. If you consider the migration path as a graph, cycles (loops) in the graph are disallowed. This avoids wasting resources by constantly migrating (A->B, B->A, A->B ...). The expectation is that cycles will never be allowed. Signed-off-by: Keith Busch --- Documentation/ABI/stable/sysfs-devices-node | 11 ++++- drivers/base/node.c | 73 +++++++++++++++++++++++++++++ include/linux/node.h | 6 +++ 3 files changed, 89 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node index 3e90e1f3bf0a..7439e1845e5d 100644 --- a/Documentation/ABI/stable/sysfs-devices-node +++ b/Documentation/ABI/stable/sysfs-devices-node @@ -90,4 +90,13 @@ Date: December 2009 Contact: Lee Schermerhorn Description: The node's huge page size control/query attributes. - See Documentation/admin-guide/mm/hugetlbpage.rst \ No newline at end of file + See Documentation/admin-guide/mm/hugetlbpage.rst + +What: /sys/devices/system/node/nodeX/migration_path +Data March 2019 +Contact: Linux Memory Management list +Description: + Defines which node the kernel should attempt to migrate this + node's pages to when this node requires memory reclaim. A + negative value means this is a terminal node and memory can not + be reclaimed through kernel managed migration. diff --git a/drivers/base/node.c b/drivers/base/node.c index 86d6cd92ce3d..20a90905555f 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -59,6 +59,10 @@ static inline ssize_t node_read_cpulist(struct device *dev, static DEVICE_ATTR(cpumap, S_IRUGO, node_read_cpumask, NULL); static DEVICE_ATTR(cpulist, S_IRUGO, node_read_cpulist, NULL); +#define TERMINAL_NODE -1 +static int node_migration[MAX_NUMNODES] = {[0 ... MAX_NUMNODES - 1] = TERMINAL_NODE}; +static DEFINE_SPINLOCK(node_migration_lock); + #define K(x) ((x) << (PAGE_SHIFT - 10)) static ssize_t node_read_meminfo(struct device *dev, struct device_attribute *attr, char *buf) @@ -233,6 +237,74 @@ static ssize_t node_read_distance(struct device *dev, } static DEVICE_ATTR(distance, S_IRUGO, node_read_distance, NULL); +static ssize_t migration_path_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + return sprintf(buf, "%d\n", node_migration[dev->id]); +} + +static ssize_t migration_path_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + int i, err, nid = dev->id; + nodemask_t visited = NODE_MASK_NONE; + long next; + + err = kstrtol(buf, 0, &next); + if (err) + return -EINVAL; + + if (next < 0) { + spin_lock(&node_migration_lock); + WRITE_ONCE(node_migration[nid], TERMINAL_NODE); + spin_unlock(&node_migration_lock); + return count; + } + if (next > MAX_NUMNODES || !node_online(next)) + return -EINVAL; + + /* + * Follow the entire migration path from 'nid' through the point where + * we hit a TERMINAL_NODE. + * + * Don't allow looped migration cycles in the path. + */ + node_set(nid, visited); + spin_lock(&node_migration_lock); + for (i = next; node_migration[i] != TERMINAL_NODE; + i = node_migration[i]) { + /* Fail if we have visited this node already */ + if (node_test_and_set(i, visited)) { + spin_unlock(&node_migration_lock); + return -EINVAL; + } + } + WRITE_ONCE(node_migration[nid], next); + spin_unlock(&node_migration_lock); + + return count; +} +static DEVICE_ATTR_RW(migration_path); + +/** + * next_migration_node() - Get the next node in the migration path + * @current_node: The starting node to lookup the next node + * + * @returns: node id for next memory node in the migration path hierarchy from + * @current_node; -1 if @current_node is terminal or its migration + * node is not online. + */ +int next_migration_node(int current_node) +{ + int nid = READ_ONCE(node_migration[current_node]); + + if (nid >= 0 && node_online(nid)) + return nid; + return TERMINAL_NODE; +} + static struct attribute *node_dev_attrs[] = { &dev_attr_cpumap.attr, &dev_attr_cpulist.attr, @@ -240,6 +312,7 @@ static struct attribute *node_dev_attrs[] = { &dev_attr_numastat.attr, &dev_attr_distance.attr, &dev_attr_vmstat.attr, + &dev_attr_migration_path.attr, NULL }; ATTRIBUTE_GROUPS(node_dev); diff --git a/include/linux/node.h b/include/linux/node.h index 257bb3d6d014..af46c7a8b94f 100644 --- a/include/linux/node.h +++ b/include/linux/node.h @@ -67,6 +67,7 @@ static inline int register_one_node(int nid) return error; } +extern int next_migration_node(int current_node); extern void unregister_one_node(int nid); extern int register_cpu_under_node(unsigned int cpu, unsigned int nid); extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid); @@ -115,6 +116,11 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg, node_registration_func_t unreg) { } + +static inline int next_migration_node(int current_node) +{ + return -1; +} #endif #define to_node(device) container_of(device, struct node, dev) From patchwork Thu Mar 21 20:01:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10864251 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7594C13B5 for ; Thu, 21 Mar 2019 20:03:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5CC422A43A for ; Thu, 21 Mar 2019 20:03:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 513752A44A; Thu, 21 Mar 2019 20:03:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B043D2A43A for ; Thu, 21 Mar 2019 20:03:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 490F16B000D; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 32AF36B0010; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F38796B000D; Thu, 21 Mar 2019 16:03:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id B7FCF6B0007 for ; Thu, 21 Mar 2019 16:03:00 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id z14so1763845pgv.0 for ; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=gmhbQpGGOf5KS7QcODtDEeapu1XYScq6rBODur+OmoM=; b=uUMs2B4/cDoby+ZupTROjDhUDFXuP8vISOZM/t+NIT33NEWvxWXBGneN56BYHuwkVG zTuvpLHsPDIDojsdLTu82epwtyjPaxxhc7e4g1PQhiU6x7to7q3dD+qPdwdAppcf5sBR m53TqksLg1PzkokniwPI9qDcQakMNFx1IGsB4zTwSISuGK4VLhRkmGkJffKDSFlXAyQa jIQ1izM9/vvQ297liQF3IAJswHN9/4qTn41RlIjvurP6/7JwSgYINIBccl1x/LnNyTlO zx/Wm1lrgSTjfJFZA1ylPBhV/fUl19+dlfjjsqNTXS6Z8dxvtSjHjNhKkZqDBz63m6YL vGJg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWJk5jvybW/hMA90IbwLqSdfDOixQGJ/0r/B7oZFuOayJQwN2jI wQm9nCTXDDyEX2zJk+dL+uGroNg0C4HMysj1QeUehEhqDlKtU57hqKmWshN2zcE+FYYitclr2+s 34vwUG+pLbvIKy5nmma2ZwGCZstUOcrgCUGImmO7Gg7BbYRMsvFdwN0bJbXq5dBlZ+g== X-Received: by 2002:a17:902:d24:: with SMTP id 33mr5392012plu.246.1553198579227; Thu, 21 Mar 2019 13:02:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqzKFAI6GsuS1bcrFFlzrrHgG0GxEGUUf462weOn1rqOA0FWnFN3Hi2u2E1uGgnvpyAdCHO4 X-Received: by 2002:a17:902:d24:: with SMTP id 33mr5391885plu.246.1553198577720; Thu, 21 Mar 2019 13:02:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553198577; cv=none; d=google.com; s=arc-20160816; b=09p6GImtIgYGv/1iR1saT1px6IAtQe06LxkUcmsf9ARibOn6+QaSWhBWTvxUsByfL8 CshdkGSkVwn+p8By8cZaniGK4pewj3JvZPibtVKINPdAhQrGUsLJHAg+RQKGOnDQ/n+r 5y6xbI3T7zRrQD7UGU7c+zV8BlPm/pg22nIKpYGmcw2mAr9j8NI8/iQ8zbvuKOWi2ULR MQCS+tBUA6Y+7MLP4B13iRyKgrlZIGOJZEGDUoidvBIAW/P6AvDGICCX7zuqgF+p7ZON ZRo3lQFTKFrED8E6YEG0V8oQ06ioeIj3uirinrS60qUBkjIHA8AK5GZaA+v9nzsEFz96 OwLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=gmhbQpGGOf5KS7QcODtDEeapu1XYScq6rBODur+OmoM=; b=XO8y8SYIK9G61HKMGNSpcNnYk1vGL3qAHs/ik8aBL7V8vjXUQRi94ZtybgAWAxRTOC AYN1MM3j9t1va/5GUd58ukWj3LXaCulz5RBptOv5MvjAGts7UIFq2/hgMSzFdMc1KlXy 8DMx7LYM3P2mWZhKOFvXeoMKVEqcBVxZgjvNGJe7fvkCdtueSpiQktj9ZLrkZPx6Ig4f o1BmRk3qFy3CUGXgaavWqL5MC/Vxk0qVAYUeRScfv0ERkGiMItomyhHZVhOfv5Wp4qt+ C61u3icyF58WnZb9X891nOaqlLWUPp47VinkecHncXMffPythVQsSy2v4G1GLrB4hPOT 4tpQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id r25si4703408pfd.91.2019.03.21.13.02.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Mar 2019 13:02:57 -0700 (PDT) Received-SPF: pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 13:02:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="309246237" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2019 13:02:56 -0700 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org Cc: Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 2/5] mm: Split handling old page for migration Date: Thu, 21 Mar 2019 14:01:54 -0600 Message-Id: <20190321200157.29678-3-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20190321200157.29678-1-keith.busch@intel.com> References: <20190321200157.29678-1-keith.busch@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Refactor unmap_and_move() handling for the new page into a separate function from locking and preparing the old page. No functional change here: this is just making it easier to reuse this part of the page migration from contexts that already locked the old page. Signed-off-by: Keith Busch --- mm/migrate.c | 115 +++++++++++++++++++++++++++++++---------------------------- 1 file changed, 61 insertions(+), 54 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ac6f4939bb59..705b320d4b35 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1000,57 +1000,14 @@ static int move_to_new_page(struct page *newpage, struct page *page, return rc; } -static int __unmap_and_move(struct page *page, struct page *newpage, - int force, enum migrate_mode mode) +static int __unmap_and_move_locked(struct page *page, struct page *newpage, + enum migrate_mode mode) { int rc = -EAGAIN; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(page); - if (!trylock_page(page)) { - if (!force || mode == MIGRATE_ASYNC) - goto out; - - /* - * It's not safe for direct compaction to call lock_page. - * For example, during page readahead pages are added locked - * to the LRU. Later, when the IO completes the pages are - * marked uptodate and unlocked. However, the queueing - * could be merging multiple pages for one bio (e.g. - * mpage_readpages). If an allocation happens for the - * second or third page, the process can end up locking - * the same page twice and deadlocking. Rather than - * trying to be clever about what pages can be locked, - * avoid the use of lock_page for direct compaction - * altogether. - */ - if (current->flags & PF_MEMALLOC) - goto out; - - lock_page(page); - } - - if (PageWriteback(page)) { - /* - * Only in the case of a full synchronous migration is it - * necessary to wait for PageWriteback. In the async case, - * the retry loop is too short and in the sync-light case, - * the overhead of stalling is too much - */ - switch (mode) { - case MIGRATE_SYNC: - case MIGRATE_SYNC_NO_COPY: - break; - default: - rc = -EBUSY; - goto out_unlock; - } - if (!force) - goto out_unlock; - wait_on_page_writeback(page); - } - /* * By try_to_unmap(), page->mapcount goes down to 0 here. In this case, * we cannot notice that anon_vma is freed while we migrates a page. @@ -1077,11 +1034,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!trylock_page(newpage))) - goto out_unlock; + goto out; if (unlikely(!is_lru)) { rc = move_to_new_page(newpage, page, mode); - goto out_unlock_both; + goto out_unlock; } /* @@ -1100,7 +1057,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, VM_BUG_ON_PAGE(PageAnon(page), page); if (page_has_private(page)) { try_to_free_buffers(page); - goto out_unlock_both; + goto out_unlock; } } else if (page_mapped(page)) { /* Establish migration ptes */ @@ -1110,22 +1067,19 @@ static int __unmap_and_move(struct page *page, struct page *newpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); page_was_mapped = 1; } - if (!page_mapped(page)) rc = move_to_new_page(newpage, page, mode); if (page_was_mapped) remove_migration_ptes(page, rc == MIGRATEPAGE_SUCCESS ? newpage : page, false); - -out_unlock_both: - unlock_page(newpage); out_unlock: + unlock_page(newpage); /* Drop an anon_vma reference if we took one */ +out: if (anon_vma) put_anon_vma(anon_vma); - unlock_page(page); -out: + /* * If migration is successful, decrease refcount of the newpage * which will not free the page because new page owner increased @@ -1141,7 +1095,60 @@ static int __unmap_and_move(struct page *page, struct page *newpage, else putback_lru_page(newpage); } + return rc; +} + +static int __unmap_and_move(struct page *page, struct page *newpage, + int force, enum migrate_mode mode) +{ + int rc = -EAGAIN; + + if (!trylock_page(page)) { + if (!force || mode == MIGRATE_ASYNC) + goto out; + + /* + * It's not safe for direct compaction to call lock_page. + * For example, during page readahead pages are added locked + * to the LRU. Later, when the IO completes the pages are + * marked uptodate and unlocked. However, the queueing + * could be merging multiple pages for one bio (e.g. + * mpage_readpages). If an allocation happens for the + * second or third page, the process can end up locking + * the same page twice and deadlocking. Rather than + * trying to be clever about what pages can be locked, + * avoid the use of lock_page for direct compaction + * altogether. + */ + if (current->flags & PF_MEMALLOC) + goto out; + + lock_page(page); + } + if (PageWriteback(page)) { + /* + * Only in the case of a full synchronous migration is it + * necessary to wait for PageWriteback. In the async case, + * the retry loop is too short and in the sync-light case, + * the overhead of stalling is too much + */ + switch (mode) { + case MIGRATE_SYNC: + case MIGRATE_SYNC_NO_COPY: + break; + default: + rc = -EBUSY; + goto out_unlock; + } + if (!force) + goto out_unlock; + wait_on_page_writeback(page); + } + rc = __unmap_and_move_locked(page, newpage, mode); +out_unlock: + unlock_page(page); +out: return rc; } From patchwork Thu Mar 21 20:01:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10864255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 454131575 for ; Thu, 21 Mar 2019 20:03:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CB032A43A for ; Thu, 21 Mar 2019 20:03:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 210C62A44A; Thu, 21 Mar 2019 20:03:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EDA62A43A for ; Thu, 21 Mar 2019 20:03:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7E736B0010; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C9ECE6B0008; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76EF56B0008; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id C26CC6B0008 for ; Thu, 21 Mar 2019 16:03:00 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id j10so6477341pff.5 for ; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Of1nlBcwSQDxF7g8uZVRddiVCwLGPv/XHyS3dYVKxIk=; b=SuNvraQra0owJG8q8S5mHH9W0eBJKHhM52XPTfOGN7CRxoc8tSgpeVQBKGWszNgajT RooL97dSVVNErm83vLPhmnkG05dkumAD/5SmmwHbom7mLsoNbd6Yk+0IhdGOzj5xDn25 C8DuOOWxzsE7D6DWnOf5o0dTkPPli3DwOSTbxQQJpAjWitSLf/P0TYoflEkumcEG2E/L R4r1tQ9eifIGVccUMjRxbvIOim9ACrx85Vx8BJXDl1z51j39yc4dEeapY6suv5Uid9a1 DPDAwYYnvZclZl1y3GUHRYMChFS9vvf63CL6wZFMVc/bXFiU1xndi9bTjIoMke0rLzik rMXw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUYBdcfG6nmvZHw/+V8j/RuDAWTO4Ude3fZa/3942P1tj9nCmM9 5CPcx0U71jWC1PY8F4HgifvqPlRGO39nFx6BkLbFAWonwD2FdZVY/6Tr0MWB+6dp6d3s2TGhMv4 UCaRF/s6s9RqhmosmRbbZnfWg/eK3+g6XqVV7hCsD3PC+fs4yQKTf9QGqu7yPEfbkAg== X-Received: by 2002:a17:902:2ac3:: with SMTP id j61mr5428506plb.112.1553198579531; Thu, 21 Mar 2019 13:02:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQDNvH7MvJHTuqBpSSF+VOMaS2qWW9zFO8AzwmGHwY4zosZpBRaUQI6mA/o6C33bsIKFoo X-Received: by 2002:a17:902:2ac3:: with SMTP id j61mr5428431plb.112.1553198578451; Thu, 21 Mar 2019 13:02:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553198578; cv=none; d=google.com; s=arc-20160816; b=wQbxBlac9FO7qBnAnP1YqLKHbL7yz/D7N1XeDG0O3hHDedJB9gZUtGE8I/YbFDBqlz e/T2VaPxl0dgsospVZoJbBAg2EV3/yMKNQ8kOvG8VgA90zHTAq1REyjN8HffjlnbbjSE 4k/cUcIsWC0hbXoEZMYRPiCYqoKKWY/NGfFSdZgIYeqDEo+oQPcJPquiNeZmV+iBOmP4 vdQ0NkxHKDpRntu+V5J8E8sste27Ic8SP9X8yWQDKLeit5aRW1ipTuVWptr7Pp6TcuQB jkWrEUySZiIpUmwYhGECYsrUKWWLqAJrHtKoSPkvGEGC4rsIkfGWAcGyQMef2mL4j0Lk ysJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Of1nlBcwSQDxF7g8uZVRddiVCwLGPv/XHyS3dYVKxIk=; b=xQsz6Ny9zefVdsafmssWBn3z6FPJ25huUp/rjqxenwssYwB9Pkynf46jl76N1fITzp Ag5XXyydHgWmjQVcLw1kK5pQ4HGTBAxaPtFsIchVrO9wFVzdwWc5wEyQlswPuU6cftPa 5hKaahwGxun4mToV6ohpOsTgyxWPwD6Uza+gESmYUgUH1EWtZ6n3ohKKzmrtFNwFU23f PbSbrepNi7rZAA6rYW+pwtVlmq1janq0UjT9nERVNM5aLy0jIfxPR5YlOAMR/jYqQ0yn sIVJZYQ1jQos0UwoAdDTU42Pfy7ZUlcVFyovQVUxjAfU1a84dABqnGQJjYkWsgecpKif wy1w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id r25si4703408pfd.91.2019.03.21.13.02.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Mar 2019 13:02:58 -0700 (PDT) Received-SPF: pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 13:02:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="309246241" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2019 13:02:57 -0700 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org Cc: Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 3/5] mm: Attempt to migrate page in lieu of discard Date: Thu, 21 Mar 2019 14:01:55 -0600 Message-Id: <20190321200157.29678-4-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20190321200157.29678-1-keith.busch@intel.com> References: <20190321200157.29678-1-keith.busch@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a memory node has a preferred migration path to demote cold pages, attempt to move those inactive pages to that migration node before reclaiming. This will better utilize available memory, provide a faster tier than swapping or discarding, and allow such pages to be reused immediately without IO to retrieve the data. Some places we would like to see this used: 1. Persistent memory being as a slower, cheaper DRAM replacement 2. Remote memory-only "expansion" NUMA nodes 3. Resolving memory imbalances where one NUMA node is seeing more allocation activity than another. This helps keep more recent allocations closer to the CPUs on the node doing the allocating. Signed-off-by: Keith Busch --- include/linux/migrate.h | 6 ++++++ include/trace/events/migrate.h | 3 ++- mm/debug.c | 1 + mm/migrate.c | 45 ++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c | 15 ++++++++++++++ 5 files changed, 69 insertions(+), 1 deletion(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index e13d9bf2f9a5..a004cb1b2dbb 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -25,6 +25,7 @@ enum migrate_reason { MR_MEMPOLICY_MBIND, MR_NUMA_MISPLACED, MR_CONTIG_RANGE, + MR_DEMOTION, MR_TYPES }; @@ -79,6 +80,7 @@ extern int migrate_huge_page_move_mapping(struct address_space *mapping, extern int migrate_page_move_mapping(struct address_space *mapping, struct page *newpage, struct page *page, enum migrate_mode mode, int extra_count); +extern bool migrate_demote_mapping(struct page *page); #else static inline void putback_movable_pages(struct list_head *l) {} @@ -105,6 +107,10 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping, return -ENOSYS; } +static inline bool migrate_demote_mapping(struct page *page) +{ + return false; +} #endif /* CONFIG_MIGRATION */ #ifdef CONFIG_COMPACTION diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index 705b33d1e395..d25de0cc8714 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -20,7 +20,8 @@ EM( MR_SYSCALL, "syscall_or_cpuset") \ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ - EMe(MR_CONTIG_RANGE, "contig_range") + EM(MR_CONTIG_RANGE, "contig_range") \ + EMe(MR_DEMOTION, "demotion") /* * First define the enums in the above macros to be exported to userspace diff --git a/mm/debug.c b/mm/debug.c index c0b31b6c3877..53d499f65199 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -25,6 +25,7 @@ const char *migrate_reason_names[MR_TYPES] = { "mempolicy_mbind", "numa_misplaced", "cma", + "demotion", }; const struct trace_print_flags pageflag_names[] = { diff --git a/mm/migrate.c b/mm/migrate.c index 705b320d4b35..83fad87361bf 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1152,6 +1152,51 @@ static int __unmap_and_move(struct page *page, struct page *newpage, return rc; } +/** + * migrate_demote_mapping() - Migrate this page and its mappings to its + * demotion node. + * @page: An isolated, non-compound page that should move to + * its current node's migration path. + * + * @returns: True if migrate demotion was successful, false otherwise + */ +bool migrate_demote_mapping(struct page *page) +{ + int rc, next_nid = next_migration_node(page_to_nid(page)); + struct page *newpage; + + /* + * The flags are set to allocate only on the desired node in the + * migration path, and to fail fast if not immediately available. We + * are already in the memory reclaim path, we don't want heroic + * efforts to get a page. + */ + gfp_t mask = GFP_NOWAIT | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC | __GFP_THISNODE; + + VM_BUG_ON_PAGE(PageCompound(page), page); + VM_BUG_ON_PAGE(PageLRU(page), page); + + if (next_nid < 0) + return false; + + newpage = alloc_pages_node(next_nid, mask, 0); + if (!newpage) + return false; + + /* + * MIGRATE_ASYNC is the most light weight and never blocks. + */ + rc = __unmap_and_move_locked(page, newpage, MIGRATE_ASYNC); + if (rc != MIGRATEPAGE_SUCCESS) { + __free_pages(newpage, 0); + return false; + } + + set_page_owner_migrate_reason(newpage, MR_DEMOTION); + return true; +} + /* * gcc 4.7 and 4.8 on arm get an ICEs when inlining unmap_and_move(). Work * around it. diff --git a/mm/vmscan.c b/mm/vmscan.c index a5ad0b35ab8e..0a95804e946a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1261,6 +1261,21 @@ static unsigned long shrink_page_list(struct list_head *page_list, ; /* try to reclaim the page below */ } + if (!PageCompound(page)) { + if (migrate_demote_mapping(page)) { + unlock_page(page); + if (likely(put_page_testzero(page))) + goto free_it; + + /* + * Speculative reference will free this page, + * so leave it off the LRU. + */ + nr_reclaimed++; + continue; + } + } + /* * Anonymous process memory has backing store? * Try to allocate it some swap space here. From patchwork Thu Mar 21 20:01:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10864253 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67B6713B5 for ; Thu, 21 Mar 2019 20:03:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F97E2A43A for ; Thu, 21 Mar 2019 20:03:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4433D2A44A; Thu, 21 Mar 2019 20:03:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC0B82A43A for ; Thu, 21 Mar 2019 20:03:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA9B36B000E; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B9486B0010; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 410796B000C; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id CB73B6B000A for ; Thu, 21 Mar 2019 16:03:00 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id 18so6444056pgx.11 for ; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=awb2dSQuAJCZWSdScxKVVSAlrOniCFGJRyt4vv2Mf74=; b=iQX5x9DRzsQ/j+cKJMrMgfzteQSfDCFzVdwWPdQkWkVGrqM/reu+2ppXyW+4sZ4BF6 Hjb6pktqrwCcSWYckYXsZ29UYYmHhaYSdsZSNFHARJGCnmC+E/Mo9R/XBMC7cxggL2nY C4hTXt2EKEGTSyxlyCaKd+XUu23gssy9PAdH0pYC87ksCAqKc+YrWh3bfYVj14lSrgpX 2OrFSyMUsuhck0zPnjd32GT29ScCFg0RdgkzW5mwygelDqILeseT+MXVPUjDP5WcSYFP 2mPGPAt/+p6FGmtGE+DyAEaIFEMh6QjxcsAYYDfk8JIHp01XEHsh6r4ok+segbb2fcbE VI9w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXE5UjlSmvGTp1qWlPrycxFwiKBWVOuSKs2I1BZePTMFEjAJGf5 PqpLEqkQ9SaNVb5Ir4f7O/o/Yu5TFzxFcozpbvvL16xNN7O4UEzNjliq1WjxiUclc1HwW8xTqzi DARurMvuAUqQS+lTTmaMzRVntqzKnR4wZdXsm6rzVT3kzSIrrMVpirrPcc+wQWxjZ2Q== X-Received: by 2002:a17:902:1c9:: with SMTP id b67mr5290944plb.176.1553198580441; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqxsYeC/j3zS9Bdz2YboVHgjTSPjX0VAl1LdctPQXWT28PCmUcEECQwhwaKwqorc2h+3MFHG X-Received: by 2002:a17:902:1c9:: with SMTP id b67mr5290836plb.176.1553198579028; Thu, 21 Mar 2019 13:02:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553198579; cv=none; d=google.com; s=arc-20160816; b=SMuo+dJpmWtmqZ1F/uuh/ngtJFwl3HPHKeOO3hda4aqdb1Xp1QNiU3siy9A7kUmudC vbZ34WNF77srq/aCMhV1y1EMhrgarNMrXOT8Nfve3gkuhIe+XkwdqyWyIuMOH4itmSvc R0IsD2zE2Z6co05F846RakPZCKqo+68NHIF/okWy4/yppB+MJmiEOSyML+UXkP3ge2pr A7VnBZ+4ePSHOVbVftjpFA+H5RcMQjbsF2DHAuaLlEadh25HkAsRhGPbXSNWvCOuPgS8 d31oe9XK2J0AWCSH8pqttLe0UDQzkmJWmLb+8NCBOjMHyc+HtGldYpJdwRkv6OgoEiBk VP/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=awb2dSQuAJCZWSdScxKVVSAlrOniCFGJRyt4vv2Mf74=; b=ZSCEQW0yoYfzxrTtWeVLF81qEn7E0iGrTZjb54WR4RsDlAncTrSOqXmFX1o48SmtGw i1YfdsXYtvX1zBIeL18pqljtsVKphOGCYYfyn4puilmv7yQyAQOeNgq2kIFewshwNqx4 9q36yGa16qg+PJmluTz1jG6KS207dlCAnnuXe1nDDa75yLc7FIP073Of5OOWdpZMtJPg NtxIztmCOebKaWxZzIse/sAf51mf/lNLyhH9KdD4gxMNsETXiCc8hGbwPqwX8zH+/Igq n/J06lTVqiqzpSbBs9KZbNXJraPbtjK4ZYoDL8EJuQEfjMUmkInccaxcQ7a16I6+zYtb TnvA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id r25si4703408pfd.91.2019.03.21.13.02.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Mar 2019 13:02:59 -0700 (PDT) Received-SPF: pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 13:02:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="309246245" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2019 13:02:58 -0700 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org Cc: Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 4/5] mm: Consider anonymous pages without swap Date: Thu, 21 Mar 2019 14:01:56 -0600 Message-Id: <20190321200157.29678-5-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20190321200157.29678-1-keith.busch@intel.com> References: <20190321200157.29678-1-keith.busch@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Age and reclaim anonymous pages from nodes that have an online migration node even if swap is not enabled. Signed-off-by: Keith Busch --- include/linux/swap.h | 20 ++++++++++++++++++++ mm/vmscan.c | 10 +++++----- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 4bfb5c4ac108..91b405a3b44f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -680,5 +680,25 @@ static inline bool mem_cgroup_swap_full(struct page *page) } #endif +static inline bool reclaim_anon_pages(struct mem_cgroup *memcg, + int node_id) +{ + /* Always age anon pages when we have swap */ + if (memcg == NULL) { + if (get_nr_swap_pages() > 0) + return true; + } else { + if (mem_cgroup_get_nr_swap_pages(memcg) > 0) + return true; + } + + /* Also age anon pages if we can auto-migrate them */ + if (next_migration_node(node_id) >= 0) + return true; + + /* No way to reclaim anon pages */ + return false; +} + #endif /* __KERNEL__*/ #endif /* _LINUX_SWAP_H */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 0a95804e946a..226c4c838947 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -327,7 +327,7 @@ unsigned long zone_reclaimable_pages(struct zone *zone) nr = zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_FILE) + zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_FILE); - if (get_nr_swap_pages() > 0) + if (reclaim_anon_pages(NULL, zone_to_nid(zone))) nr += zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_ANON) + zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_ANON); @@ -2206,7 +2206,7 @@ static bool inactive_list_is_low(struct lruvec *lruvec, bool file, * If we don't have swap space, anonymous page deactivation * is pointless. */ - if (!file && !total_swap_pages) + if (!file && !reclaim_anon_pages(NULL, pgdat->node_id)) return false; inactive = lruvec_lru_size(lruvec, inactive_lru, sc->reclaim_idx); @@ -2287,7 +2287,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, enum lru_list lru; /* If we have no swap space, do not bother scanning anon pages. */ - if (!sc->may_swap || mem_cgroup_get_nr_swap_pages(memcg) <= 0) { + if (!sc->may_swap || !reclaim_anon_pages(memcg, pgdat->node_id)) { scan_balance = SCAN_FILE; goto out; } @@ -2650,7 +2650,7 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, */ pages_for_compaction = compact_gap(sc->order); inactive_lru_pages = node_page_state(pgdat, NR_INACTIVE_FILE); - if (get_nr_swap_pages() > 0) + if (!reclaim_anon_pages(NULL, pgdat->node_id)) inactive_lru_pages += node_page_state(pgdat, NR_INACTIVE_ANON); if (sc->nr_reclaimed < pages_for_compaction && inactive_lru_pages > pages_for_compaction) @@ -3347,7 +3347,7 @@ static void age_active_anon(struct pglist_data *pgdat, { struct mem_cgroup *memcg; - if (!total_swap_pages) + if (!reclaim_anon_pages(NULL, pgdat->node_id)) return; memcg = mem_cgroup_iter(NULL, NULL, NULL); From patchwork Thu Mar 21 20:01:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10864257 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E33E413B5 for ; Thu, 21 Mar 2019 20:03:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9AAD2A43A for ; Thu, 21 Mar 2019 20:03:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BDBF52A44A; Thu, 21 Mar 2019 20:03:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 553562A43A for ; Thu, 21 Mar 2019 20:03:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1969D6B0008; Thu, 21 Mar 2019 16:03:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E7A766B0007; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA12C6B000C; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 080136B0007 for ; Thu, 21 Mar 2019 16:03:01 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id v3so6452858pgk.9 for ; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+jDHrInzqAkEuroi35KDeQz8ygOnozOul64UgglecQs=; b=p2q4hVJCPG27gIBmTUQGvU0hdXzDKDewmAPvcrav+JXWMk1gCx6L9AyU5QbgBDSaCm CPJZuEE3xUsswDpCZfb0fjMuP6l/ShHZ6rcF6keUYO1TKpHqwJFKxKecj+37XPZyazrV s9DpVJIS65zKPQI8JyycSQLF5eSn/EFt1TPWD0oDKvCbVvS0/YtKq/D+60dxGeIFHrci mPb2kZ2Ipq9cm2fBkrPFNSOXHJTpgwqEygbcuUxjEZcFz6ueqUMQMh+Y9v0k2pLUblZM PBgeavVv5CshhiBv2/PPdXMnpRKjHy5W97AA7x0DtjiTZy2kcNFdbehwfhgpw9XseK/x Gahw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXg2lh2aqYp/V2Q2XwZDL+3JF00ptdg8naOU5/+3M/Htvu8fwkx qWjTArJY0rzy1AYX3koHiQTqDS8HB1YIARsU+cjZ9zNcmm1FIwyJs6vHEZNEMktdPW2Djw642GY Jy4Ug1WxcKyqsTLSpI9dv/4bZuaWrq2QQaUgpvmxqSo9Gh7+MTNzTU5g82IwkjelIwg== X-Received: by 2002:a62:1249:: with SMTP id a70mr5181240pfj.160.1553198580640; Thu, 21 Mar 2019 13:03:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqzbduOdmCnUo3PAoxsUYOXWXOclHVugZFMyTW+e2YndqO4EtuI0N642gfwyQlUIdPcVKRB5 X-Received: by 2002:a62:1249:: with SMTP id a70mr5181182pfj.160.1553198579844; Thu, 21 Mar 2019 13:02:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553198579; cv=none; d=google.com; s=arc-20160816; b=lROQEgeN/C/Td0VpFG7iHWDDOyL29F8BHshBy51ULpEX8uOZbPXIKjJe5Zghjoqiwj Q4Qzdn0Adf6VxdCrfMhehBNAQcFxFZV9WI9Ui3TF4bBIHz0I0H5stQk2V3r2x6NI60Wl I85YM8d+hotshRiDLKed1vdqI8JqWikE0a4G2wz2HRT+Nrl4PAhDWCNRooMhSsatlSvG 5p3hXjZs1E9rKKgMZj/tfsXFRecVA2dr98xavGlUtjPcs9Dvj2fvgij6WItHO/GGkYEZ oWN66AG6/JmtZKQj26rJRf/ZlURp9xLKYMS/+Wa5SFu6L24xFk3SdWJHXTBd8q4w6aPe pfmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+jDHrInzqAkEuroi35KDeQz8ygOnozOul64UgglecQs=; b=pi7RT5Itc5tVRikvs0zw0l4MpG9GjgQAlBIreAoky62Ugx+RKbF/f8PygOLZTYD9oS 2rFv/KhfDWJx9fNBC0HixvczVHZmvLtKR2sc9wnm/5+4T5va8YQExjt+YlnqY3177buW zQhLh8422wF6jX1rrHZX1wi9J4e3AnxEGQRBL9W2V6bghrRmAvl65XFWhgIXdErIiQlj GM66386duAKUwR0xECOkEjW7Pc1jOvsuwiHzTETHGCWJjtr2ZTxJeF6AhHC6l1elVHSs 0M35AoiOM+TS2ncfHKA80uMidXyqcV0RBvOpXo12dij3QCKUFfzM8Jpwbg2cGYQRm+yL IkXQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id r25si4703408pfd.91.2019.03.21.13.02.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Mar 2019 13:02:59 -0700 (PDT) Received-SPF: pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of keith.busch@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=keith.busch@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 13:02:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,254,1549958400"; d="scan'208";a="309246248" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2019 13:02:58 -0700 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org Cc: Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 5/5] mm/migrate: Add page movement trace event Date: Thu, 21 Mar 2019 14:01:57 -0600 Message-Id: <20190321200157.29678-6-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20190321200157.29678-1-keith.busch@intel.com> References: <20190321200157.29678-1-keith.busch@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Trace the source and destination node of a page migration to help debug memory usage. Signed-off-by: Keith Busch --- include/trace/events/migrate.h | 26 ++++++++++++++++++++++++++ mm/migrate.c | 1 + 2 files changed, 27 insertions(+) diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index d25de0cc8714..3d4b7131e547 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -6,6 +6,7 @@ #define _TRACE_MIGRATE_H #include +#include #define MIGRATE_MODE \ EM( MIGRATE_ASYNC, "MIGRATE_ASYNC") \ @@ -71,6 +72,31 @@ TRACE_EVENT(mm_migrate_pages, __print_symbolic(__entry->mode, MIGRATE_MODE), __print_symbolic(__entry->reason, MIGRATE_REASON)) ); + +TRACE_EVENT(mm_migrate_move_page, + + TP_PROTO(struct page *from, struct page *to, int status), + + TP_ARGS(from, to, status), + + TP_STRUCT__entry( + __field(struct page *, from) + __field(struct page *, to) + __field(int, status) + ), + + TP_fast_assign( + __entry->from = from; + __entry->to = to; + __entry->status = status; + ), + + TP_printk("node from=%d to=%d status=%d flags=%s refs=%d", + page_to_nid(__entry->from), page_to_nid(__entry->to), + __entry->status, + show_page_flags(__entry->from->flags & ((1UL << NR_PAGEFLAGS) - 1)), + page_ref_count(__entry->from)) +); #endif /* _TRACE_MIGRATE_H */ /* This part must be outside protection */ diff --git a/mm/migrate.c b/mm/migrate.c index 83fad87361bf..d97433da12c0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -997,6 +997,7 @@ static int move_to_new_page(struct page *newpage, struct page *page, page->mapping = NULL; } out: + trace_mm_migrate_move_page(page, newpage, rc); return rc; }