From patchwork Tue Sep 25 07:13:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10613527 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 23066161F for ; Tue, 25 Sep 2018 07:14:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1214E29B12 for ; Tue, 25 Sep 2018 07:14:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0614E29B17; Tue, 25 Sep 2018 07:14:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68D0829B12 for ; Tue, 25 Sep 2018 07:14:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3BEB8E0078; Tue, 25 Sep 2018 03:14:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9C46F8E0072; Tue, 25 Sep 2018 03:14:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 844C38E0078; Tue, 25 Sep 2018 03:14:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 3B5C38E0072 for ; Tue, 25 Sep 2018 03:14:19 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id 132-v6so9233439pga.18 for ; Tue, 25 Sep 2018 00:14:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=rji6ltTsTgqZxrSEhTGzjmxih1WUcmAf4gDg6DXH4cs=; b=RDQsj48D9Paw1nTqpQQYMfOOg4JMeitNruQY/gqujv2whQFYk8aVw5tNZRA+0kIOQK BV/JPCwOFrZerBSffbmvw2/0eqbweHdztqPYaE/DS753Y2IoonD1tU6oVFtoOHjV9zkS tlQ9K8iHaWw43VzFcVfxhBFQfgwyf3iEq7lHbxBiALYIZYgu7js3WpKhnTMP9n6udjv6 64Ti996XqpU0g6M5v7cpEgTaOOaSGtNBpwsvaodCC7/NvMvFdehYkD67cWUBCI9HaEUW PXU/8NAl5Zwvq8/Q6KZM7eEsSoG/6OlruPVOwhAcnGSanRUk6yKyTD+K90vO5I3Ll+8Z 863Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoig/+x2UKvqC4JxnCOtzYdqUiuOvTHqFbaYTkxuHtNFcTFf+R/r 1ne1X/pFhQ63t6BKHeeu/dLThlZMQ2VIys6++NcOctVBGyvJWuXKyjoyQRu2Nyrs2j4kazqsvVm bFnlkY6UeKaY36CbFZ0qwLz4W5KpMKk4caC95VZohRvZ35mdGh+iqF+e0wsEsUmMzDQ== X-Received: by 2002:a17:902:8681:: with SMTP id g1-v6mr2154062plo.302.1537859658933; Tue, 25 Sep 2018 00:14:18 -0700 (PDT) X-Google-Smtp-Source: ACcGV62AXd6VP050q3VVIO+v7T1qyqUCAnFKYcbtBlZpx12eiUADhqUfMyIs6fIysy9vKtkyUi+N X-Received: by 2002:a17:902:8681:: with SMTP id g1-v6mr2154009plo.302.1537859658156; Tue, 25 Sep 2018 00:14:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537859658; cv=none; d=google.com; s=arc-20160816; b=HQW8TgWJQJ/TsXDuWY8Jsv7ZmTlvG0y2tvAF9tdudwoGIEAN31c0IoSPjyR3PF/Xjc X0oitISyRJ8vFPxX/edsnAaOTMKB2U6/ESgZD8asOI1gbqNQKanLamgdXRV5eU4IjXRx 5lv5K/fUX8fmpliTihOIrbAAE5HiY4eIs1bDySPLl1sfOsCWQZfMX8teGWCWmf2yXAHs dKOKjI06kSyTm/SptuAMsgCT2OOotqxjL4h6uQGg2timHxN65dRMr7xr5kzFwYlvYmMT ns6lmY8ImHXBDqNhVQh8zS25OV02zw4cGz6Dfp4Za9erPv1gHbvl6hAaROu0Woxpi3Y3 Mmsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=rji6ltTsTgqZxrSEhTGzjmxih1WUcmAf4gDg6DXH4cs=; b=0/oHDN2wYNE4PkDGTK84wg8uRj8+1pIZy8PVwnqluAVHe03ZJ6AFM5Km+DEHsU1gdn /UaqgCtbeamyIH2iyJVP1CxICph0HoCDzSEAqoJ2NaSnKv+Q39uBSsnPP2L4Jf9VI64x 4YoyfhqMil4M1igr04FIlihhH6jB/nLdxbcSFfkHcNzf/P5ghsZ8Lhx3UwWWLoJlswnt NobAFYCYU29da5NCHRrdAViNE1byf7nk6KWaBwTlFUz4fYSQdS2qUINdrY34/8kpk8gH UjPI8TPqoahi3LM6pSQh9OpSKJwZ/9rQwNxxIO8IMzrZJdN7UHq+Skn5Fqe6zvw7O1eN ZC6w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga14.intel.com (mga14.intel.com. [192.55.52.115]) by mx.google.com with ESMTPS id a1-v6si1378544pgw.9.2018.09.25.00.14.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 00:14:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) client-ip=192.55.52.115; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Sep 2018 00:14:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,301,1534834800"; d="scan'208";a="89093652" Received: from yhuang-mobile.sh.intel.com ([10.239.198.87]) by fmsmga002.fm.intel.com with ESMTP; 25 Sep 2018 00:14:14 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -V5 RESEND 15/21] swap: Support to copy PMD swap mapping when fork() Date: Tue, 25 Sep 2018 15:13:42 +0800 Message-Id: <20180925071348.31458-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180925071348.31458-1-ying.huang@intel.com> References: <20180925071348.31458-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 35c7243720bc..c569e5e8ee17 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -941,6 +941,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -948,26 +949,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (IS_ENABLED(CONFIG_THP_SWAP) && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, HPAGE_PMD_NR); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The huge swap cluster has been split, split + * the PMD swap mapping and fallback to PTE + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);