From patchwork Tue Nov 20 08:54:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10690011 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 31ED713AD for ; Tue, 20 Nov 2018 08:55:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21BA12991D for ; Tue, 20 Nov 2018 08:55:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1619D29A1B; Tue, 20 Nov 2018 08:55:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F0472991D for ; Tue, 20 Nov 2018 08:55:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F02C6B1F49; Tue, 20 Nov 2018 03:55:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7031F6B1F4A; Tue, 20 Nov 2018 03:55:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B4E96B1F4B; Tue, 20 Nov 2018 03:55:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 089EB6B1F49 for ; Tue, 20 Nov 2018 03:55:41 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id 34-v6so944356plf.6 for ; Tue, 20 Nov 2018 00:55:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=xPFUqy0lRtJKEg6SL8GuqeAMvpHV9AOMSMwx2iQeFjM=; b=PO6rYkdQKjkjLTtEt6iP7Pzl7P/IDc6euxSAeUf8Eh8lEBxN4v+kuTcpwCJPypFXiV FXhU8ivLYUFj/nU0FLlpngIFOAzSViEU7fywu/ue6eMc+7zkpfw+zWFt5UeAGuJ1bnjj fHYYaCtrvyQ0fzK9hGeS7ebBSPkhLTeyk/Hu0vAJiHwngZ2BuTYxKNgQSLHdp/B0Ue70 rYa+4Wckr43/mk1d0mjhc+PhJnNiEy0mBIOKduU5YbciA+yBVZHkPe0YHbcXFGqCI4Ns EYi5W5VrWKbXH69W9V46TDDc7kdIoH4QfunuhOdR5DGjNy1eQtVNpFIoJQ996Fxqo4F8 efVg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWZbegxLQ4wP8rcr8HlD00c18lw9Lrd8r8pqDb2bKpfdJ/u4NRTA 1dTJ+1ElrR+3377TGJ5y8iyecFFMD7CV0FPvEftBOeDiB0mfRh90N+rCfuf5ZzQl3jeb4lKk/X/ P1dvsEEgSSSgFBZyFNUy7HMvj9/LRdYvrFUO3Sw5fl1OblWxMxIY4MOCHHPFFaCXcKA== X-Received: by 2002:a63:170c:: with SMTP id x12mr1096248pgl.364.1542704140692; Tue, 20 Nov 2018 00:55:40 -0800 (PST) X-Google-Smtp-Source: AFSGD/WnKLDCpfGT1aRDUVKupXroY0C6K7zpVUrE26b+lZKV3Y68xUvi6mv2KQewTnt/U9ctRLfJ X-Received: by 2002:a63:170c:: with SMTP id x12mr1096209pgl.364.1542704139483; Tue, 20 Nov 2018 00:55:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542704139; cv=none; d=google.com; s=arc-20160816; b=zlh7ZPzQ1b4icY0Pip6Yb0UHKChf39VQnjrYhft2MIbqCO8jRCVsmtq4qpvltjIw10 bkc5fl5l1HUKYfE446gTJO/n2ggMAeLBZ6DcP4TCfMBoA3t8x/0uoizIGXrpE8XBwWf/ KF/UzD6HgHhJxhqVfoXoZPN25Oh8uljHVksOUy5TMIukHDObDMA4/FgKxFhp0wuwEg2O IAh+dxIpF9DJrREMG7AIfWp3jp3VZOJY5VkNmmOJlJ6rhituhoESYI4Ie1ZOYHjuLT/Z 4ttylupKTov13zwAqDu4Yh2syJIxCJU5D6I+eowZmS2IkwyU4+YqMdLWDAry97rIWA29 b3Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=xPFUqy0lRtJKEg6SL8GuqeAMvpHV9AOMSMwx2iQeFjM=; b=ctFj7j74pTJJRyInU8eCUXZkivzY3UVtKL008Pcv4Vk2JHsyDZasW/g3uQLBwqa41M 2iYTVyb4jhkZFV1yBb1OG2v+I638bgNCoPK3F21FVO+kVTndRRHykGj/ROuPn3g5xtYO 5Sof0uxL6pQbaXkk/1dnXYXEf3MhrZZy19ccj8Y5vd5SavK8WpK+eoj/20D5CQ2h6o++ y2fFRhuGoS9qDgaot60UVur2C7q1/wF2Q4ZCJXc2Bn2MW45KRtKqmwPRysHxykCnOBVY f5jsTZzp/ZuiH1kar4IctPv7R075wI3DpgZ07HbC4gwFVVulUUAyl81YWlfKgWQcb/V9 cExA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id b15si24149550plm.431.2018.11.20.00.55.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Nov 2018 00:55:39 -0800 (PST) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Nov 2018 00:55:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,256,1539673200"; d="scan'208";a="106105888" Received: from yhuang-mobile.sh.intel.com ([10.239.197.245]) by fmsmga002.fm.intel.com with ESMTP; 20 Nov 2018 00:55:35 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -V7 RESEND 15/21] swap: Support to copy PMD swap mapping when fork() Date: Tue, 20 Nov 2018 16:54:43 +0800 Message-Id: <20181120085449.5542-16-ying.huang@intel.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20181120085449.5542-1-ying.huang@intel.com> References: <20181120085449.5542-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3aade329fe8b..2a49b2068902 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -985,6 +985,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -992,26 +993,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (IS_ENABLED(CONFIG_THP_SWAP) && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, HPAGE_PMD_NR); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The huge swap cluster has been split, split + * the PMD swap mapping and fallback to PTE + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);