From patchwork Wed May 9 08:38:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10388679 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E7A8C60353 for ; Wed, 9 May 2018 08:40:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8E6828E29 for ; Wed, 9 May 2018 08:40:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CD8FE28E2E; Wed, 9 May 2018 08:40:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0450128E1D for ; Wed, 9 May 2018 08:40:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F08716B04A9; Wed, 9 May 2018 04:39:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EB7E46B04AB; Wed, 9 May 2018 04:39:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCECC6B04AC; Wed, 9 May 2018 04:39:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id 99AD06B04A9 for ; Wed, 9 May 2018 04:39:57 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id x32-v6so3380833pld.16 for ; Wed, 09 May 2018 01:39:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9aqczudJ+2cLQCxSE5EeCNRKliDESjLojHscbMSbSvM=; b=G11qP5KHgVVpZJPmJUe50xJ1m2JcTHtQGjWopVmtgzpQqFJs/ixgfXuot+gv+e9S8F n29F3tvuLOTdGCt1R0apU4NjcPkYoijtte8Fw6lqqWWecahh1v6qnbypDsZfQdJFvX79 F9fFjE4fvG3UHUbnOfOPHZnG0ZvsQ6euC6SMb402Hj6NH92BuoZIq44jCPam/Y9P8EeX M41etVNJNzydr+Hjxfn7f1noNSn2EgX/Xcy9G1LxQEn56dSHhCC0Hg2Llf0snh9FzGO9 JVN7JxZWIgLdLTFjOx+0T8e3/mWIwO5Bovt8bC6TmfkCNQLzIfWphrwWM4Rh8tIkUuKW XJ7g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Gm-Message-State: ALQs6tBeFZO0GaETR+B4MFfrZxToVTkjhbdgav2lU1JsyDF04DCYy6G9 iCyZX2lZjo75z+4CHyJb998SihVLj0twxAVgEJNjgHNsCO0Ye4M/qNau6df6Ydm2pKzNrCqSSMi HBt2m3rkuWnew9L8f50W2/ZAg7fsdEwksMds5jnjQRlCLsWaW2zYplYK+FajTUKNSzQ== X-Received: by 10.98.196.19 with SMTP id y19mr43223416pff.97.1525855197279; Wed, 09 May 2018 01:39:57 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpAbuGNNr9JQT+vBgj2j6S3Kk4Y0phcn/P0l4JWOvEpimIZxrwB0CfuVPJHLH8PP3146UfW X-Received: by 10.98.196.19 with SMTP id y19mr43223376pff.97.1525855196147; Wed, 09 May 2018 01:39:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525855196; cv=none; d=google.com; s=arc-20160816; b=pWtJHQd1J6FB3qesX/N33Q06Ycx5vbL0GfEuGgOQGYhlMX3/WQJLDPsNE/ZQ+TQFgl pq+SFsERWKqs/rkN8ZND8BpEtNQXUpIlsZrkSAj97zL0rxWhc9CojSwDForjP6ipuKYS OBbwPsKKhbT8CU8OSSuGQpRRmFiWgQgqLFL/IieME8EP50o5KayFo0TQ4NBeL8oAWrA7 LXPuasM80xH3oiNQAi/Bcm4Ri3Pe0FR/6GNMOjSqbHSq/aQgdpknekrkCbjOqkup6s/U zwdnwdmWzY2CHSe/sRY9ml+USavt1Gor/25V6TNYt6+LUU1Y15QCmQ5obqN1pDjLh/Mq jBYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=9aqczudJ+2cLQCxSE5EeCNRKliDESjLojHscbMSbSvM=; b=yYF3U87Ahk5pFD0VlMCnS5Yym0N1Uwpq0le3BwWFtsCvCp5rFQwp8sFjAAfOp9t/lr rWUElNt4djFSoziBYQTMYR5lE/Vic6FDVQx0iNCqytqWTSWO9qzYVCtY38RMfIjSkKmd WYPd/Bnca4h1tzeDl9/iwTvFGq758IK3JMNUxOMWegKE42rVDAoFAWOM2ECj+2WoR8dE ZNR/p8EZh2baTaVcSbFMiiBPA0/hXCiTsO784F5Pq9oFuEK6Vty3yu7Ok8TqgC2rKq9I brVVCKkfA3m9kU19Xm/8h4lYWtGckjsvr9Y5NDucg1PBMY0b4Iv751AsjT8iu/cFhFzv z7vg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id z9-v6si21682595plk.94.2018.05.09.01.39.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 May 2018 01:39:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 May 2018 01:39:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,381,1520924400"; d="scan'208";a="52769717" Received: from yhuang-gentoo.sh.intel.com ([10.239.193.148]) by fmsmga004.fm.intel.com with ESMTP; 09 May 2018 01:39:50 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan Subject: [PATCH -mm -V2 15/21] mm, THP, swap: Support to copy PMD swap mapping when fork() Date: Wed, 9 May 2018 16:38:40 +0800 Message-Id: <20180509083846.14823-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180509083846.14823-1-ying.huang@intel.com> References: <20180509083846.14823-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c45df1c86817..6f3129c068bc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -944,6 +944,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -951,26 +952,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (thp_swap_supported() && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, true); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The swap cluster has been split, split the + * pmd map now + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);