From patchwork Sat Jul 31 06:39:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12412325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AA28C4338F for ; Sat, 31 Jul 2021 06:46:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D492460EC0 for ; Sat, 31 Jul 2021 06:46:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D492460EC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 62C976B0033; Sat, 31 Jul 2021 02:46:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DCA06B0036; Sat, 31 Jul 2021 02:46:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CAF18D0001; Sat, 31 Jul 2021 02:46:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0235.hostedemail.com [216.40.44.235]) by kanga.kvack.org (Postfix) with ESMTP id 31F736B0033 for ; Sat, 31 Jul 2021 02:46:42 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D481D181AEF1E for ; Sat, 31 Jul 2021 06:46:41 +0000 (UTC) X-FDA: 78421949802.08.C21C8B9 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf30.hostedemail.com (Postfix) with ESMTP id 7B78DE00BDF3 for ; Sat, 31 Jul 2021 06:46:41 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id q63-20020a17090a17c5b02901774f4b30ebso8938181pja.1 for ; Fri, 30 Jul 2021 23:46:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=xqUeIqZ8Laqm1hjc5SKPZeq+KLOGVogqPJNCfLRzT78=; b=ClY3nsfbH/n05/YnjoMbG/qYn5BNCyQMmrF74ixgKDvRAnPSYsJTjDxlmP44IoQsSw hfw8gCbNkYYxdASa+fCR3Y253bAcQsoiBr94+xLo1HneAmgr9KQYZbFgoZkwcmq4Nhql Nyhs+nRUE8W+hQMrTF2gi1xv/A4B4N6YHf1Flsb9MoYHEHcQQZsKMPeiTddthEBrfVcC KQ/D0Xo4vgYODAYM8Pj5xVLyR5A9QwboLvK/lFmE7sD7OkEezJDupwYpSsGqPKYoPuIr 1t7iE3Kh4a10k135s+tEsV6sqXG3JDHtZhI3DgTnpTs9yQ9NaxM7seLs12CKudUD/c4E D6PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=xqUeIqZ8Laqm1hjc5SKPZeq+KLOGVogqPJNCfLRzT78=; b=pPl4LXFMAbXk1DprpWa3cooJxpt/mDapoeQ99AHYTOahOt9xAJgyem4+LwiYJmXFT4 zJlMB9dbk6mE0YyINNaVHul0kXLkHpjJlN51p0X5UzFMDKQQ3yaXP1+rmt6i+pn994fQ /8HFrDsBBAV4XJfGXTHf6TYSZPFuIc58xs4LdI76QjUndTsTJLH4rWCSwNlL7+cowbjJ En89IX8GOjRmfdd59UhSOetFVTuJ73CZM9t9VqZgxc0H5xdWmkKJj/2vUCRVArBZE20m 1w9ZeAxixsQylxr/BOsI0Zderg6icv3j3WC+BL1Fa5yA+oNg2xJqrWHQoRp89hvUvn/e e/hw== X-Gm-Message-State: AOAM531Post4KGZTAtdET2Hr+n5Z/vASHFZFESolzZw2fDQxKjGxKCoR Sa4gznSXXPcAHFKRz9IwDbKqrlHaySg3MRU50g9Si2DMPj+VtQOLXIBmT8ugeIi+xoe7KsEBo1/ 6opQwW99cgrCpsk5zCgTTkI5CTR/ogmL/qDWqGkoWFE6EmVtL0gZcSmnI X-Google-Smtp-Source: ABdhPJzL5m4UjbecB/fBMg9vAeXnswEYCl2l2UqB/0Gqow4tu/XQMVb9YRr7DA/K6lY49DfANaZtcMqOlF8= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:2b92:c131:b08a:84c7]) (user=yuzhao job=sendgmr) by 2002:a0c:be8e:: with SMTP id n14mr6679960qvi.16.1627713581663; Fri, 30 Jul 2021 23:39:41 -0700 (PDT) Date: Sat, 31 Jul 2021 00:39:35 -0600 Message-Id: <20210731063938.1391602-1-yuzhao@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 0/3] mm: optimize thp for reclaim and migration From: Yu Zhao To: linux-mm@kvack.org Cc: Andrew Morton , Hugh Dickins , "Kirill A . Shutemov" , Matthew Wilcox , Vlastimil Babka , Yang Shi , Zi Yan , linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7B78DE00BDF3 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=ClY3nsfb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3LfAEYQYKCDAkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3LfAEYQYKCDAkglTMaSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--yuzhao.bounces.google.com X-Stat-Signature: 3nscw5m5nhrsihbc51jitmh6cupjw76z X-HE-Tag: 1627714001-219181 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Systems using /sys/kernel/mm/transparent_hugepage/enabled=always can experience memory pressure due to internal fragmentation from a large number of thp. Userspace can leave many subpages untouched but reclaim can't identify them based on the dirty bit (and drop them like what it does to clean pages). However, it's still possible to know whether a subpage is equivalent to being clean by checking its content. When splitting a thp for reclaim or migration, we can drop subpages that contain only zeros and therefore avoid writing them back or duplicating them. benchmark ========= The best case scenario is only the first byte of head is non zero. The worse case scenario is only the last byte of every subpage is non zero. zram 10GB ~~~~~~~~~ best case 7.170s 3.559s -50.36% worst case 8.216s 8.94s +8.81% swap 10GB ~~~~~~~~~ best case 70.466s 3.544s -94.97% worst case 67.014s 65.521s -2.22% (noise) (CONFIG_THP_SWAP=n) zram (before) ============= best case ~~~~~~~~~ time 7.170s 21.78% clear_page_erms 16.47% zram_bvec_rw 2.90% clear_huge_page 2.74% _raw_spin_lock 2.65% page_vma_mapped_walk 2.41% flush_tlb_func 2.21% shrink_page_list 1.55% try_to_unmap_one 1.29% _raw_spin_lock_irqsave 1.25% page_counter_cancel 1.21% __mod_node_page_state 1.17% __mod_lruvec_page_state 1.10% add_to_swap_cache 1.08% xas_create 1.07% xas_store worst case ~~~~~~~~~~ time 8.216s 17.27% clear_page_erms 13.25% lzo1x_1_do_compress 4.35% zram_bvec_rw 3.41% memset_erms 3.23% _raw_spin_lock 2.88% page_vma_mapped_walk 2.75% clear_huge_page 2.28% flush_tlb_func 1.92% shrink_page_list 1.39% try_to_unmap_one 1.21% page_counter_cancel 1.20% zs_free 1.14% _raw_spin_lock_irqsave 1.13% xas_create 1.02% __mod_lruvec_page_state zram (after) ============ best case ~~~~~~~~~ time 3.559s 44.55% clear_page_erms 27.74% memchr_inv 6.43% clear_huge_page 2.71% split_huge_page_to_list 1.79% page_vma_mapped_walk 1.31% __split_huge_pmd 1.21% remove_migration_pte 1.12% __free_one_page worst case ~~~~~~~~~~ time 8.94s 16.08% clear_page_erms 11.81% memchr_inv 9.62% lzo1x_1_do_compress 3.51% memset_erms 3.17% zram_bvec_rw 2.76% _raw_spin_lock 2.70% clear_huge_page 2.34% flush_tlb_func 2.25% page_vma_mapped_walk 1.56% shrink_page_list 1.33% try_to_unmap_one 1.07% _raw_spin_lock_irqsave 1.04% xas_create 1.01% zs_free disk (before) ============ best case ~~~~~~~~~ time 70.466s 23.38% clear_page_erms 3.91% clear_huge_page 3.32% _raw_spin_lock 2.95% page_vma_mapped_walk 2.43% shrink_page_list 2.25% flush_tlb_func 1.84% try_to_unmap_one 1.53% _raw_spin_lock_irqsave 1.45% __mod_memcg_lruvec_state 1.37% page_counter_cancel 1.17% _raw_spin_lock_irq 1.16% xas_create 1.12% _find_next_bit 1.11% xas_store 1.11% add_to_swap_cache 1.10% __mod_node_page_state 1.04% __mod_lruvec_page_state 1.04% __free_one_page worst case ~~~~~~~~~~ time 67.014s 25.54% clear_page_erms 4.36% clear_huge_page 3.51% _raw_spin_lock 2.97% page_vma_mapped_walk 2.85% flush_tlb_func 1.84% try_to_unmap_one 1.52% shrink_page_list 1.47% __mod_memcg_lruvec_state 1.42% page_counter_cancel 1.38% _raw_spin_lock_irq 1.28% __mod_lruvec_page_state 1.15% _raw_spin_lock_irqsave 1.13% xas_load 1.10% add_to_swap_cache 1.05% __mod_node_page_state disk (after) ============ best case ~~~~~~~~~ time 3.544s 42.58% clear_page_erms 27.44% memchr_inv 6.15% clear_huge_page 2.60% split_huge_page_to_list 1.74% page_vma_mapped_walk 1.26% __free_one_page 1.11% remove_migration_pte 1.10% __split_huge_pmd 1.08% __list_del_entry_valid worst case ~~~~~~~~~~ time 65.521s 21.83% clear_page_erms 14.39% memchr_inv 3.66% clear_huge_page 3.17% _raw_spin_lock 2.38% page_vma_mapped_walk 2.32% flush_tlb_func 1.59% try_to_unmap_one 1.39% shrink_page_list 1.26% page_counter_cancel 1.16% __mod_memcg_lruvec_state 1.11% add_to_swap_cache 1.05% _raw_spin_lock_irq test.c ====== #include #include #include #include #include #define handle_error(msg) \ do { perror(msg); exit(EXIT_FAILURE); } while (0) int main(int argc, char **argv) { char *addr; size_t size; int offset, num_dirty; int ret; if (argc != 4) { printf("Usage: ./a.out SIZE OFFSET NUM_DIRTY\n" "SIZE: Size of mmap region in MB\n" "OFFSET: Offset of (first) dirty byte in each 4KB page, 0 ~ 4095\n" "NUM_DIRTY: Number of dirty 4KB pages in each 2MB region, 0 ~ 512\n"); return -1; } size = (size_t)(((size_t)atoi(argv[1])) << 20); offset = atoi(argv[2]); num_dirty = atoi(argv[3]); addr = (char *)mmap(NULL, size , PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) handle_error("mmap"); ret = madvise(addr, size, MADV_HUGEPAGE); if (ret == -1) handle_error("mmap hugepage"); for (size_t i = 0; i < size; i += (1 << 21)) for (size_t j = 0; j < (num_dirty << 12); j += (1 << 12)) memset(addr + i + j + offset, 0xff, 1); ret = madvise(addr, size, MADV_PAGEOUT); if (ret == -1) handle_error("mmap pageout"); ret = munmap(addr, size); if (ret == -1) handle_error("munmap"); return 0; } Yu Zhao (3): mm: don't take lru lock when splitting isolated thp mm: free zapped tail pages when splitting isolated thp mm: don't remap clean subpages when splitting isolated thp include/linux/rmap.h | 2 +- include/linux/vm_event_item.h | 2 ++ mm/huge_memory.c | 63 ++++++++++++++++++++++++--------- mm/migrate.c | 65 ++++++++++++++++++++++++++++++----- mm/vmstat.c | 2 ++ 5 files changed, 107 insertions(+), 27 deletions(-)