From patchwork Wed Oct 10 07:19:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10634123 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8253B14DB for ; Wed, 10 Oct 2018 07:28:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DE4A294F6 for ; Wed, 10 Oct 2018 07:28:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 62070296AF; Wed, 10 Oct 2018 07:28:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAE06294F6 for ; Wed, 10 Oct 2018 07:28:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36C0B6B027F; Wed, 10 Oct 2018 03:27:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1DDC96B0276; Wed, 10 Oct 2018 03:27:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED8A76B027D; Wed, 10 Oct 2018 03:27:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 7551A6B0276 for ; Wed, 10 Oct 2018 03:27:43 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id a64-v6so3893655pfg.16 for ; Wed, 10 Oct 2018 00:27:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=eRs1mVrGtRkrqceo3WVBrbltoam21f/sbNF58/KaFNU=; b=RlqcVMwI0nDD9aVNHi1oF8EaZhl8Sod1ls01ncZPAk8T38qCL0w/sRgMn4yH9VqKes SlCCVDv2och4eO+ZoC+3dmTr5PH6KehzPnqAOaaCy6VqLsBv+2SJQoHU30GO2jJenb2F 9RGeIpGO64CanFtphgerSvvOmXKSTIGq9YQmKimk/IVymyk+J8xD8DjdXA2QkFIvog0K +GE0BKIYvtBzx1z1utiwY8w+V/c9BFazTNv82gjUhfcaz3+8KmhwEzQZawU0iDJAhVdo MikTlWJngwzzv0MDMv07X8pAnCX2qhXScDw9GEZKwVR+Efoxg/NhwMOMh3mtHRu4j3CR bUVw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogrVFIJZx4it/m5VHqBBijI0fCVQDsDzdICHea2PoQ+GspvEK6k towpXgDb0qCH+bstT44+JI7cd2cOMGSMpMJp7ykn9QGs0ttlwBBlSxjvMAf/TrZGHQhZAeTLRbY BMxr1Vc+GtL0wo3KZ3guvNEPZgl0p6ta2KhapNMb+/mLdQ1Rnoo3WVjs1dvUrMjVzyw== X-Received: by 2002:a17:902:744c:: with SMTP id e12-v6mr32143104plt.186.1539156463138; Wed, 10 Oct 2018 00:27:43 -0700 (PDT) X-Google-Smtp-Source: ACcGV60x3PBuDi0n0wB+bi++9q0MmypsiTz+xdbEHemZDfml9Q8o4o2THGy7EPosUGfCDhiMht32 X-Received: by 2002:a17:902:744c:: with SMTP id e12-v6mr32143055plt.186.1539156462475; Wed, 10 Oct 2018 00:27:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539156462; cv=none; d=google.com; s=arc-20160816; b=ykDZsY949wRvEdddHD3oKpZV975nUmpWp8bSTbOluh31jsKYu6byb+Sewk9CjJ18JK Mpii2UVO8VlIvV60UKXuWRGVrOZSfmTVub4jTQbfbue0Qys3bEWZZ77kaVjzv789SACb mpYti5RGoaOI9przP4s3I0wl/WsbUdk4APtG+bwuwQzyIzQpUNjMtMaEuuBkYvuxBNkk qWBQMLzt5EmN3vVSW4Bt8/NE1d05VEQhuJ0eml3bu08+4aWumkfJIxQ3AlWYuN71RrlM U0tNCt3q7y/ARNda4tqQ4QwtnSAJw8988LNVKn+/AQtFMU8YOdTUDLcgeoZcWbj/wYeU P/iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=eRs1mVrGtRkrqceo3WVBrbltoam21f/sbNF58/KaFNU=; b=LJcHB9vYiCvkyEVSVA9fVdN2IlEzMLl032F7heynHwryt8f1ic1JSEKdp3EeNWL7J5 RNzlqB+9Tkhbch309Ai0uLALzwNEm2z7j7ftspGqvp3vQT8yPJOjsYcCklokc/55S4L+ dRKh03gcVaD5U+UOfMP3C0a5RFRxPn6Qg9XC/p0X7NzAreGqDU7uG04W17Xlp3VinHSB 7eSxx8obx/NnQC0k8gAqC+jseqC2nFwCqib0Cu6yu1LA7e47cEMlcgBNSwZy5Z8TaT7T UjTkYtkUawRkPp9jikAljKvPKj2o58lhv1aSNaeAzFQaAzhWZm3ObZGUuZp8N+pZpyei y8JQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id n84-v6si21364295pfg.127.2018.10.10.00.27.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Oct 2018 00:27:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Oct 2018 00:27:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,363,1534834800"; d="scan'208";a="93870234" Received: from yhuang-mobile.sh.intel.com ([10.239.198.87]) by fmsmga002.fm.intel.com with ESMTP; 10 Oct 2018 00:19:36 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -V6 15/21] swap: Support to copy PMD swap mapping when fork() Date: Wed, 10 Oct 2018 15:19:18 +0800 Message-Id: <20181010071924.18767-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181010071924.18767-1-ying.huang@intel.com> References: <20181010071924.18767-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ebd043528309..74c8621619cb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -987,6 +987,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -994,26 +995,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (IS_ENABLED(CONFIG_THP_SWAP) && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, HPAGE_PMD_NR); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The huge swap cluster has been split, split + * the PMD swap mapping and fallback to PTE + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);