From patchwork Tue Sep 11 20:58:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10596313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78AB4109C for ; Tue, 11 Sep 2018 20:58:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6950229DC1 for ; Tue, 11 Sep 2018 20:58:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5CD3729DD2; Tue, 11 Sep 2018 20:58:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6741829DC1 for ; Tue, 11 Sep 2018 20:58:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C9218E0005; Tue, 11 Sep 2018 16:58:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 278C08E0001; Tue, 11 Sep 2018 16:58:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 140FB8E0005; Tue, 11 Sep 2018 16:58:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id C8B588E0001 for ; Tue, 11 Sep 2018 16:58:35 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id p22-v6so13349354pfj.7 for ; Tue, 11 Sep 2018 13:58:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id; bh=f5mqQ4rZYi8WQitaZpChCVqIUW2rxy8OG7kuKb3E544=; b=qdiEymBheDatqJ0UJc9OAo1xAFa9l3WJQ5bWl2QaEkBRKMY6tJv/cii50x+JdcGs8D uVfNfqtDhPbZVEB1QCqf9FSHmzCzwuzGRCBa3ncCsoPAQPPB/sVtFvkX5ad5kVTwrqpX TL/ryGoicDObvDnGNW2n1gPeUzbmD6/Lha3Za/z3XsWPNvUeM7pRNEQzd7/VXzBieaIE BhYi7Bs6icsSOqoYRHrMRXbtHiOKgohH107VicJEiDlKpvAmqLkw7Z6eVURbkMVUuSWt 7vp8NOfRaHXZ2vpC2YpcehkvnEe/Lg1LZpXcxUJ4AoSFz3DR9qwHBRtS0hVIYgJnkwof EyTQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.131 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APzg51B3iYcHdJJgLsI2+8tQRLF0MwWL+5fcuyMuEr6qF1GlEw/pPIFO RTHzYnIcfz7zvz8p6KKgr2NRIEZkchJcd19wejDmkyD8rFidU99meH5/rgd6NjrBTN/yThhU7It TNZebJLeVfXxV/59PJGGqsLAiuU6ZPmTN0JPFAVVIHd1wp5UdfWuW1EHhcB/9BDTgbg== X-Received: by 2002:a62:6547:: with SMTP id z68-v6mr31405926pfb.20.1536699515461; Tue, 11 Sep 2018 13:58:35 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbhVBL5FwD2ahl7uj0yegawGAwDyMjsoKUwIHmVKDMQLURLXQYFEcel+TwDvdDxlwK9NQmm X-Received: by 2002:a62:6547:: with SMTP id z68-v6mr31405863pfb.20.1536699514110; Tue, 11 Sep 2018 13:58:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536699514; cv=none; d=google.com; s=arc-20160816; b=Y2lH6MFUgKVDVxqCN0TkMSRzqqX//5gbm4dl9Fo2sFPqV72bwwgTs3zW1owUTZKoLs f3eJe7ESAaQYa3+3NMxoptJdPdRkapArzpNuHwNP+0nA/GqYr17QCmv5+Tnv54MY9yW/ 1q+4+KGRy0EqZvauk2wQMuYZjij+W5+QLu9wERUhiDkpUY6YTxh5fS3Hx2K+zGFdmq/D xIrrJCSt5fLpMkdYkBo6Nf5FrWcr3N3I31jVAkb+1uMGp1JoNEu7bN9fx8smOR4kBcet CpV0jFTpARd5bkKmWAdFw1I2uE2j1Aih71KV1SkROHlUpBe5XaN8ws+Bs9IgA1p/KoCU +YAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=f5mqQ4rZYi8WQitaZpChCVqIUW2rxy8OG7kuKb3E544=; b=ddTtcLPF/LuvJmsjUMROo1Sd0hLbFabtmmKyxiKztWNcsoPRilDuYAEoYw0s5d9dTg iKudTw/izXa8dlCO/YMJq4LPNXTry6o6m5opKPw5KjlJ0v+xrGiDUPcjZKhuZMhx5rfa 6uH79rYuqQgaaGhbELtZn1rk1miLxKCMsZvD0tgoLcjlUfer1NdQYt6GGgHOONY3jPsB cZou+cQN/St2pOnHgI6EJCrCULq76JSGGbmhcnRe3T17RJjIR2GOks5Mlq8ghIKxt6MT Y2EYgCIUYjRIdoKDFu+RHuR4oNPi3VZ6qmu/2JCuX2jhpuX3PEH/d2qUivHXn8TzgiUP DufQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.131 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com. [115.124.30.131]) by mx.google.com with ESMTPS id l30-v6si20836496plg.12.2018.09.11.13.58.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 13:58:34 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.131 as permitted sender) client-ip=115.124.30.131; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.131 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01422;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0T8Sg6Yy_1536699501; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0T8Sg6Yy_1536699501) by smtp.aliyun-inc.com(127.0.0.1); Wed, 12 Sep 2018 04:58:29 +0800 From: Yang Shi To: mhocko@kernel.org, willy@infradead.org, ldufour@linux.vnet.ibm.com, vbabka@suse.cz, akpm@linux-foundation.org, dave.hansen@intel.com, oleg@redhat.com, srikar@linux.vnet.ibm.com Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC v9 PATCH 0/4] mm: zap pages with read mmap_sem in munmap for large mapping Date: Wed, 12 Sep 2018 04:58:09 +0800 Message-Id: <1536699493-69195-1-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Background: Recently, when we ran some vm scalability tests on machines with large memory, we ran into a couple of mmap_sem scalability issues when unmapping large memory space, please refer to https://lkml.org/lkml/2017/12/14/733 and https://lkml.org/lkml/2018/2/20/576. History: Then akpm suggested to unmap large mapping section by section and drop mmap_sem at a time to mitigate it (see https://lkml.org/lkml/2018/3/6/784). V1 patch series was submitted to the mailing list per Andrew's suggestion (see https://lkml.org/lkml/2018/3/20/786). Then I received a lot great feedback and suggestions. Then this topic was discussed on LSFMM summit 2018. In the summit, Michal Hocko suggested (also in the v1 patches review) to try "two phases" approach. Zapping pages with read mmap_sem, then doing via cleanup with write mmap_sem (for discussion detail, see https://lwn.net/Articles/753269/) Approach: Zapping pages is the most time consuming part, according to the suggestion from Michal Hocko [1], zapping pages can be done with holding read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write mmap_sem to cleanup vmas. But, we can't call MADV_DONTNEED directly, since there are two major drawbacks: * The unexpected state from PF if it wins the race in the middle of munmap. It may return zero page, instead of the content or SIGSEGV. * Can't handle VM_LOCKED | VM_HUGETLB | VM_PFNMAP and uprobe mappings, which is a showstopper from akpm But, some part may need write mmap_sem, for example, vma splitting. So, the design is as follows: acquire write mmap_sem lookup vmas (find and split vmas) deal with special mappings detach vmas downgrade_write zap pages free page tables release mmap_sem The vm events with read mmap_sem may come in during page zapping, but since vmas have been detached before, they, i.e. page fault, gup, etc, will not be able to find valid vma, then just return SIGSEGV or -EFAULT as expected. If the vma has VM_HUGETLB | VM_PFNMAP, they are considered as special mappings. They will be handled by falling back to regular do_munmap() with exclusive mmap_sem held in this patch since they may update vm flags. But, with the "detach vmas first" approach, the vmas have been detached when vm flags are updated, so it sounds safe to update vm flags with read mmap_sem for this specific case. So, VM_HUGETLB and VM_PFNMAP will be handled by using the optimized path in the following separate patches for bisectable sake. Unmapping uprobe areas may need update mm flags (MMF_RECALC_UPROBES). However it is fine to have false-positive MMF_RECALC_UPROBES according to uprobes developer. So, uprobe unmap will not be handled by the regular path. With the "detach vmas first" approach we don't have to re-acquire mmap_sem again to clean up vmas to avoid race window which might get the address space changed since downgrade_write() doesn't release the lock to lead regression, which simply downgrades to read lock. And, since the lock acquire/release cost is managed to the minimum and almost as same as before, the optimization could be extended to any size of mapping without incurring significant penalty to small mappings. For the time being, just do this in munmap syscall path. Other vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain intact due to some implementation difficulties since they acquire write mmap_sem from very beginning and hold it until the end, do_munmap() might be called in the middle. But, the optimized do_munmap would like to be called without mmap_sem held so that we can do the optimization. So, if we want to do the similar optimization for mmap/mremap path, I'm afraid we would have to redesign them. mremap might be called on very large area depending on the usecases, the optimization to it will be considered in the future. Changelog v8 -> v9: * Uprobe developer (Oleg Nesterov and Srikar Dronamraju) helped to confirm it is fine to have a false-positive MMF_RECALC_UPROBES. So, unmapping uprobe areas doesn't have to be handled by regular path and we can drop the uprobe related patch. Thanks Oleg and Srikar. * Dave hansen helped to confirm mpx unmap has to be called under write mmap_sem, but it has not to be after unmap_region(). So move arch_unmap() before downgrade_write(). The other user of arch_unmap() is PowerPC, which just set mm->context.vdso_base, so it sounds fine for this change too. Thanks Dave. * The above two resolved the concerns from Vlastimil for v8. v7 -> v8: * Added Acked-by from Vlastimil for patch 1/5. Thanks. * Fixed the wrong "evolution" direction. Converted VM_HUGETLB and VM_PFNMAP mapping use the optimized path in separate patches respectively for safe and bisectable sake per Michal's suggestion. * Extracted has_uprobes() helper from uprobes_munmap() to check if mm or vmas have uprobes, which could save some cycles instead of calling vma_has_uprobes() directly for some cases. Per Vlastimil's suggestion. * Keep unmapping uprobes area using regular do_munmap() since it might update mm flags, that might be not safe with read mmap_sem even though vmas have been detached. * Fixed some comments from Willy. v6 -> v7: * Rename some helper functions per Michal and Vlastimil's comments. * Refactor munmap_lookup_vma() to return the pointer of start vma per Michal's suggestion. * Rephrase some commit log for patch 2/4 per Michal's comments. * Deal with special mappings (VM_HUGETLB | VM_PFNMAP | uprobes) with regular do_munmap() in a separate patch per Michal's suggestion. * Bring the patch which makes vma_has_uprobes() non-static back since it is needed to check if a vma has uprobes or not. v5 -> v6: * Fixed the comments from Kirill and Laurent * Added Laurent's reviewed-by to patch 1/2. Thanks. v4 -> v5: * Detach vmas before zapping pages so that we don't have to use VM_DEAD to mark a being unmapping vma since they have been detached from rbtree when zapping pages. Per Kirill * Eliminate VM_DEAD stuff * With this change we don't have to re-acquire write mmap_sem to do cleanup. So, we could eliminate a potential race window * Eliminate PUD_SIZE check, and extend this optimization to all size v3 -> v4: * Extend check_stable_address_space to check VM_DEAD as Michal suggested * Deal with vm_flags update of VM_LOCKED | VM_HUGETLB | VM_PFNMAP and uprobe mappings with exclusive lock held. The actual unmapping is still done with read mmap_sem to solve akpm's concern * Clean up vmas with calling do_munmap to prevent from race condition by not carrying vmas as Kirill suggested * Extracted more common code * Solved some code cleanup comments from akpm * Dropped uprobe and arch specific code, now all the changes are mm only * Still keep PUD_SIZE threshold, if everyone thinks it is better to extend to all sizes or smaller size, will remove it * Make this optimization 64 bit only explicitly per akpm's suggestion v2 -> v3: * Refactor do_munmap code to extract the common part per Peter's sugestion * Introduced VM_DEAD flag per Michal's suggestion. Just handled VM_DEAD in x86's page fault handler for the time being. Other architectures will be covered once the patch series is reviewed * Now lookup vma (find and split) and set VM_DEAD flag with write mmap_sem, then zap mapping with read mmap_sem, then clean up pgtables and vmas with write mmap_sem per Peter's suggestion v1 -> v2: * Re-implemented the code per the discussion on LSFMM summit Regression and performance data: Did the below regression test with setting thresh to 4K manually in the code: * Full LTP * Trinity (munmap/all vm syscalls) * Stress-ng: mmap/mmapfork/mmapfixed/mmapaddr/mmapmany/vm * mm-tests: kernbench, phpbench, sysbench-mariadb, will-it-scale * vm-scalability With the patches, exclusive mmap_sem hold time when munmap a 80GB address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to us level from second. munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock() { munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us | unmap_region(); munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | } Here the excution time of unmap_region() is used to evaluate the time of holding read mmap_sem, then the remaining time is used with holding exclusive lock. Yang Shi (4): mm: refactor do_munmap() to extract the common part mm: mmap: zap pages with read mmap_sem in munmap mm: unmap VM_HUGETLB mappings with optimized path mm: unmap VM_PFNMAP mappings with optimized path mm/mmap.c | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 156 insertions(+), 34 deletions(-)