From patchwork Fri Feb 15 22:03:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 10815933 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A2D113B5 for ; Fri, 15 Feb 2019 22:03:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A2242FF77 for ; Fri, 15 Feb 2019 22:03:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2DD043041C; Fri, 15 Feb 2019 22:03:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0335A2FF77 for ; Fri, 15 Feb 2019 22:03:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 849E68E0003; Fri, 15 Feb 2019 17:03:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7F9828E0001; Fri, 15 Feb 2019 17:03:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 672488E0006; Fri, 15 Feb 2019 17:03:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 1CB4B8E0003 for ; Fri, 15 Feb 2019 17:03:51 -0500 (EST) Received: by mail-yw1-f72.google.com with SMTP id x64so6906223ywc.6 for ; Fri, 15 Feb 2019 14:03:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding:dkim-signature; bh=AK8H5tpJ34j47t7xdPUWcNrKavgktr5lUYER4OwE6o4=; b=T4R1e3luEeHPfNcf0KsRLAK/LPrdI8nVGZL72sV79d99T3W0N1ftDhO91PPXqz1cY0 h/gTL21Z+cmvlfNbzfoDOkp01w5lspBCiM33s8/7C1W5f3QsNI2RAC7lYPmpTgMSd6OB 1+uZxFqesN4PRIL787czcMq916wXfYUnsfHcmVStOPtP4xUeFonK7nBfoKoxWAnYlbXN oI+8LmJCMR/4iT6IAuSDXVAUSIBSXKExEtRjCoDi+J5w5LipbSjWUAUqfUCG9JqrfzUi e44DHcehpzIZ36fVi4YaLA2V6oa6yPEcq7wxk0/iVP9lMHqNziresD6UcITGrgZ64s9i 2B1g== X-Gm-Message-State: AHQUAuYDnNpjfNRZPT8ZmKuV6lPc5mXYNP68bwthWPAexbzfstuPNCgj sd7EoO4ONJQtY0Tek8cAaELU5bTKdyUQiYP6jC3/K5SQGZr+Qhr6QY7eZ6xAysSEkFZcsitWehE u/IgsfRMV2tDrvKETf9HJVdJe2smytH8Nrt6H8F91iBvKHCls3MJeZvIRlUZf3XWrtQ== X-Received: by 2002:a81:e0d:: with SMTP id 13mr9862544ywo.61.1550268230803; Fri, 15 Feb 2019 14:03:50 -0800 (PST) X-Google-Smtp-Source: AHgI3IZB6kd9HlNDbKx0V74xfCA9MSogTWvWZJME10yRjIv2bN2oZSYnGRDv0iOKqDiiKqCc/2z/ X-Received: by 2002:a81:e0d:: with SMTP id 13mr9862458ywo.61.1550268229652; Fri, 15 Feb 2019 14:03:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550268229; cv=none; d=google.com; s=arc-20160816; b=o9xzDf4N0+iULgZLb7faUO5YE/2ZKl6xHqLp0Kru8Q/LvxE6FpNPZRYSGo9insCIsC FJOdCP0RGurTsHyU+SJbliYK62tjCWzl2HrLgnJQA3hFkT/sA7qtVO56uOzQOsZ3brju bAKeaOkYG3oR2WNgKffx5Zqjs7cF63iO3ACR5kKLANjXdDRzLgfhn+pBZ3LNtdAmSye6 eBxdwXGk8K+Tk3RAlAf6UPCM9T6c2pm6cbdXLdK34w49Kf8v006I/p58TjgvlMyPeLbH BFp78u9efLlFyadGj3klML8wAFyOmD9qAbdWggeOEYSldE5+WAZOKhkleTWdJRz2rHJX tpmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=dkim-signature:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from; bh=AK8H5tpJ34j47t7xdPUWcNrKavgktr5lUYER4OwE6o4=; b=PrjsdfzQhiB7dZslpgE7amVSN5qrMGAWUuDAMBAOKUWgqweJtMnV+Dmw6bxP867q9C SA+PrWtfSPbwP3tbcd9IsgnkzPOLcbMM5p1jAXJOgZRlUQSThyxCwHpVVODb6AmvqSax /snreYcSuoEPpIDLRtbH/DDe/7g631++CZIhGwo7OJXYl3xuMoPjamkY5RIfkd6O3fxA oiGO5/ppS+lbw1YYrSrNcr00i2YgytyEq6vArBof9+YeVpYlBXuLd9VrKADoyJzRDJKv xprGRydpovJJulD8aAZaqNG6MXk4K8+T2GuiRV+lWCxEP6EcWmR1mRs1iBYt9VWhT6QO FuZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=llx8nrNY; spf=pass (google.com: domain of ziy@nvidia.com designates 216.228.121.65 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: from hqemgate16.nvidia.com (hqemgate16.nvidia.com. [216.228.121.65]) by mx.google.com with ESMTPS id b5si3852075ybo.313.2019.02.15.14.03.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Feb 2019 14:03:49 -0800 (PST) Received-SPF: pass (google.com: domain of ziy@nvidia.com designates 216.228.121.65 as permitted sender) client-ip=216.228.121.65; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=llx8nrNY; spf=pass (google.com: domain of ziy@nvidia.com designates 216.228.121.65 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Fri, 15 Feb 2019 14:03:51 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Fri, 15 Feb 2019 14:03:48 -0800 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Fri, 15 Feb 2019 14:03:48 -0800 Received: from nvrsysarch5.nvidia.com (172.20.13.39) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Fri, 15 Feb 2019 22:03:48 +0000 From: Zi Yan To: , CC: Dave Hansen , Michal Hocko , "Kirill A . Shutemov" , Andrew Morton , Vlastimil Babka , Mel Gorman , John Hubbard , Mark Hairgrove , Nitin Gupta , David Nellans , Zi Yan Subject: [RFC PATCH 02/31] mm: migrate: Add THP exchange support. Date: Fri, 15 Feb 2019 14:03:05 -0800 Message-ID: <20190215220334.29298-3-ziy@nvidia.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190215220334.29298-1-ziy@nvidia.com> References: <20190215220334.29298-1-ziy@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL103.nvidia.com (172.20.187.11) To HQMAIL101.nvidia.com (172.20.187.10) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1550268231; bh=AK8H5tpJ34j47t7xdPUWcNrKavgktr5lUYER4OwE6o4=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-Originating-IP: X-ClientProxiedBy:Content-Transfer-Encoding:Content-Type; b=llx8nrNYNyWAE9nMxnvaGVyiU6nNWXM9c63gW31afUQku2afncl5z2iFqFdrFKAdW MBkHBNmhaxoQw2LCootR2cHJ0kqgsQO0OjfABw7A9iuSucSndYl0ZcSfKg56AVzJSv NKy87FTDJ2owxBMBT14yW9N0n3n2ZYfK+1oBB5Huf9R6HCm4ey7MxK5XKvIiE0BQzh gJW0b/b+Jx2Xoen9iWY0845hNMm84tuHAgqHha9mh4qdQO3lawBmjTTo30uAygaKu8 iMAtqmtpUcv784RnR4Nk5i/nC0PqIMWgPxucrbFc4rd7/KTQk3FlOPvCopGP1yFkwX VnNhO1hCqWkTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add support for exchange between two THPs. Signed-off-by: Zi Yan --- mm/exchange.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/mm/exchange.c b/mm/exchange.c index a607348cc6f4..8cf286fc0f10 100644 --- a/mm/exchange.c +++ b/mm/exchange.c @@ -48,7 +48,8 @@ struct page_flags { unsigned int page_swapcache:1; unsigned int page_writeback:1; unsigned int page_private:1; - unsigned int __pad:3; + unsigned int page_doublemap:1; + unsigned int __pad:2; }; @@ -125,20 +126,23 @@ static void exchange_huge_page(struct page *dst, struct page *src) static void exchange_page_flags(struct page *to_page, struct page *from_page) { int from_cpupid, to_cpupid; - struct page_flags from_page_flags, to_page_flags; + struct page_flags from_page_flags = {0}, to_page_flags = {0}; struct mem_cgroup *to_memcg = page_memcg(to_page), *from_memcg = page_memcg(from_page); from_cpupid = page_cpupid_xchg_last(from_page, -1); - from_page_flags.page_error = TestClearPageError(from_page); + from_page_flags.page_error = PageError(from_page); + if (from_page_flags.page_error) + ClearPageError(from_page); from_page_flags.page_referenced = TestClearPageReferenced(from_page); from_page_flags.page_uptodate = PageUptodate(from_page); ClearPageUptodate(from_page); from_page_flags.page_active = TestClearPageActive(from_page); from_page_flags.page_unevictable = TestClearPageUnevictable(from_page); from_page_flags.page_checked = PageChecked(from_page); - ClearPageChecked(from_page); + if (from_page_flags.page_checked) + ClearPageChecked(from_page); from_page_flags.page_mappedtodisk = PageMappedToDisk(from_page); ClearPageMappedToDisk(from_page); from_page_flags.page_dirty = PageDirty(from_page); @@ -148,18 +152,22 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) clear_page_idle(from_page); from_page_flags.page_swapcache = PageSwapCache(from_page); from_page_flags.page_writeback = test_clear_page_writeback(from_page); + from_page_flags.page_doublemap = PageDoubleMap(from_page); to_cpupid = page_cpupid_xchg_last(to_page, -1); - to_page_flags.page_error = TestClearPageError(to_page); + to_page_flags.page_error = PageError(to_page); + if (to_page_flags.page_error) + ClearPageError(to_page); to_page_flags.page_referenced = TestClearPageReferenced(to_page); to_page_flags.page_uptodate = PageUptodate(to_page); ClearPageUptodate(to_page); to_page_flags.page_active = TestClearPageActive(to_page); to_page_flags.page_unevictable = TestClearPageUnevictable(to_page); to_page_flags.page_checked = PageChecked(to_page); - ClearPageChecked(to_page); + if (to_page_flags.page_checked) + ClearPageChecked(to_page); to_page_flags.page_mappedtodisk = PageMappedToDisk(to_page); ClearPageMappedToDisk(to_page); to_page_flags.page_dirty = PageDirty(to_page); @@ -169,6 +177,7 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) clear_page_idle(to_page); to_page_flags.page_swapcache = PageSwapCache(to_page); to_page_flags.page_writeback = test_clear_page_writeback(to_page); + to_page_flags.page_doublemap = PageDoubleMap(to_page); /* set to_page */ if (from_page_flags.page_error) @@ -195,6 +204,8 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) set_page_young(to_page); if (from_page_flags.page_is_idle) set_page_idle(to_page); + if (from_page_flags.page_doublemap) + SetPageDoubleMap(to_page); /* set from_page */ if (to_page_flags.page_error) @@ -221,6 +232,8 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) set_page_young(from_page); if (to_page_flags.page_is_idle) set_page_idle(from_page); + if (to_page_flags.page_doublemap) + SetPageDoubleMap(from_page); /* * Copy NUMA information to the new page, to prevent over-eager @@ -280,6 +293,7 @@ static int exchange_page_move_mapping(struct address_space *to_mapping, VM_BUG_ON_PAGE(to_mapping != page_mapping(to_page), to_page); VM_BUG_ON_PAGE(from_mapping != page_mapping(from_page), from_page); + VM_BUG_ON(PageCompound(from_page) != PageCompound(to_page)); if (!to_mapping) { /* Anonymous page without mapping */ @@ -600,7 +614,6 @@ static int unmap_and_exchange(struct page *from_page, to_mapping = to_page->mapping; from_index = from_page->index; to_index = to_page->index; - /* * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU @@ -691,6 +704,23 @@ static int unmap_and_exchange(struct page *from_page, return rc; } +static bool can_be_exchanged(struct page *from, struct page *to) +{ + if (PageCompound(from) != PageCompound(to)) + return false; + + if (PageHuge(from) != PageHuge(to)) + return false; + + if (PageHuge(from) || PageHuge(to)) + return false; + + if (compound_order(from) != compound_order(to)) + return false; + + return true; +} + /* * Exchange pages in the exchange_list * @@ -751,9 +781,8 @@ static int exchange_pages(struct list_head *exchange_list, continue; } - /* TODO: compound page not supported */ /* to_page can be file-backed page */ - if (PageCompound(from_page) || + if (!can_be_exchanged(from_page, to_page) || page_mapping(from_page) ) { ++failed;