From patchwork Thu Jul 26 18:10:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10546289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4DB61112B for ; Thu, 26 Jul 2018 18:10:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3ED902BBD1 for ; Thu, 26 Jul 2018 18:10:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 331FC2BBC5; Thu, 26 Jul 2018 18:10:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07CF72BBC1 for ; Thu, 26 Jul 2018 18:10:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 002B36B0008; Thu, 26 Jul 2018 14:10:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ECA796B000A; Thu, 26 Jul 2018 14:10:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6C756B000C; Thu, 26 Jul 2018 14:10:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id 91F8F6B0008 for ; Thu, 26 Jul 2018 14:10:41 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id b5-v6so1737629ple.20 for ; Thu, 26 Jul 2018 11:10:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=g7wmYscaUA/dnL4E2U5tg5ixFBD3yg13Z8EZcyDBZ9U=; b=Eg7mJCkDNihCuuubaFIQrvisMGwh7e/jKYwraOxq3580P7DZePwOEdwsXsR6QMVxAG x0GGg50wpw6whC298vBvlLK23qwEfVBLMacEcSCSS/V+jZBh3qAyIgZiXy/UgZeJ/6UP /mj9mEpiEj8P/C7lPH+Tf3rBhcli/I21ALT3xSkx8SC++RHw5Z57bkD3EslxTY2qqYiq Rxjt100oiEh/b/YLqR0rmJTbTTvnANvVhOTLR0PUxj1ELCNCWSro1DiBmTzfhs2jTKnz Wz84yl64hiklJWQ2Yzoyw0CCC78/jF3BZ4srS/kMf6raP7JHzpjRdbCXouVaaKFWRuWl mQVw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: AOUpUlHjW9YB68+WoWbwNcKiEPJtnL657E+TdqlM6UlUUP9UITsNKPL5 K1LI3Ei82mSThgYN1lKYJe2+CMxdh0gtCJG2GKE0rpTbXBDX7d+KDDa36yeAjjmtoNKwma7VyaD HA/X67yWXxw0YtTBL3NhvO0yYQaYmlbl8I2oDbwQGxT+SRBgM9BbZZ7qKqtwLvztGZQ== X-Received: by 2002:a63:c20:: with SMTP id b32-v6mr2853458pgl.400.1532628641224; Thu, 26 Jul 2018 11:10:41 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfr3rATbpu2wS+QRQoW7vo1e04GxPH2LGVscDn/3d2vlX1xzPtSxGkTGVQgbeoZXhX2OH5C X-Received: by 2002:a63:c20:: with SMTP id b32-v6mr2853407pgl.400.1532628640081; Thu, 26 Jul 2018 11:10:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532628640; cv=none; d=google.com; s=arc-20160816; b=VoyctOTfcD2KVaxmN3aCXMuiCGHnw2rK2jhYhVRe5dSPoDjy+8OjgyoMH4shLOe5Rg lnizITTd4JrVkS+dWD9wmgiKk+RHwCK0hWgeP/ocpLd8S6htpQNQL5SEBixoyMCk3HWd OHRxcNIaMErsS+dlFcZg+s1jR13zDFwjh5xtnk8O6EyR8cz2Y8zjasqIDctVXxfIbx5O s6R89EkqYqfa9O26qxdJTZ8XK3qp5GhgCN6ZFx1prFQsLu7NxLxaBa1xuED20reniXOX c7YVrPVqSQalB5NDg38MLzzEeaye46McVETmS4LPVObJ+dpZADyrQJSNQ3AACSxoXXHY /s4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=g7wmYscaUA/dnL4E2U5tg5ixFBD3yg13Z8EZcyDBZ9U=; b=x9LQrRoRuUNmup9i9b9Fu7TOhsRfjJdKSNpz6DMbfOomkV/ry0lsGcmr0WhqQmDh6P qGOyd/+rhSXmEszMujieuOqAHPnMzePJwXZ0sjGR4qbdUh/TlSy5ec/CiAHiw/bmx1Is CUlzUxhkAhY106MGLmVpKtJpQRoqX5J/Gf3AVpbNEcl4S5o9bZhQAxIc8LEK5ZEELFFw FVAYbTmtLbr6I+AUrNf92PIlAzCu4g5C8D2whV38xhBFOPnD4zfyodhAZGLoWK9u94/R 912msix7MjOXpHTRy6//Frq/rGtHhBPPXqKRItgK+WFZC1s8DibVg4JPpnA4dNW8aSXn zuaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com. [115.124.30.130]) by mx.google.com with ESMTPS id g18-v6si1604782plo.341.2018.07.26.11.10.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jul 2018 11:10:40 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) client-ip=115.124.30.130; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07487;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0T5ObfW8_1532628628; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0T5ObfW8_1532628628) by smtp.aliyun-inc.com(127.0.0.1); Fri, 27 Jul 2018 02:10:35 +0800 From: Yang Shi To: mhocko@kernel.org, willy@infradead.org, ldufour@linux.vnet.ibm.com, kirill@shutemov.name, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC v6 PATCH 1/2] mm: refactor do_munmap() to extract the common part Date: Fri, 27 Jul 2018 02:10:13 +0800 Message-Id: <1532628614-111702-2-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1532628614-111702-1-git-send-email-yang.shi@linux.alibaba.com> References: <1532628614-111702-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduces three new helper functions: * munmap_addr_sanity() * munmap_lookup_vma() * munmap_mlock_vma() They will be used by do_munmap() and the new do_munmap with zapping large mapping early in the later patch. There is no functional change, just code refactor. Reviewed-by: Laurent Dufour Signed-off-by: Yang Shi --- mm/mmap.c | 120 ++++++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 82 insertions(+), 38 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index d1eb87e..2504094 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2686,34 +2686,44 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma, return __split_vma(mm, vma, addr, new_below); } -/* Munmap is split into 2 main parts -- this part which finds - * what needs doing, and the areas themselves, which do the - * work. This now handles partial unmappings. - * Jeremy Fitzhardinge - */ -int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, - struct list_head *uf) +static inline bool munmap_addr_sanity(unsigned long start, size_t len) { - unsigned long end; - struct vm_area_struct *vma, *prev, *last; - if ((offset_in_page(start)) || start > TASK_SIZE || len > TASK_SIZE-start) - return -EINVAL; + return false; - len = PAGE_ALIGN(len); - if (len == 0) - return -EINVAL; + if (PAGE_ALIGN(len) == 0) + return false; + + return true; +} + +/* + * munmap_lookup_vma: find the first overlap vma and split overlap vmas. + * @mm: mm_struct + * @vma: the first overlapping vma + * @prev: vma's prev + * @start: start address + * @end: end address + * + * returns 1 if successful, 0 or errno otherwise + */ +static int munmap_lookup_vma(struct mm_struct *mm, struct vm_area_struct **vma, + struct vm_area_struct **prev, unsigned long start, + unsigned long end) +{ + struct vm_area_struct *tmp, *last; /* Find the first overlapping VMA */ - vma = find_vma(mm, start); - if (!vma) + tmp = find_vma(mm, start); + if (!tmp) return 0; - prev = vma->vm_prev; - /* we have start < vma->vm_end */ + + *prev = tmp->vm_prev; + + /* we have start < vma->vm_end */ /* if it doesn't overlap, we have nothing.. */ - end = start + len; - if (vma->vm_start >= end) + if (tmp->vm_start >= end) return 0; /* @@ -2723,7 +2733,7 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, * unmapped vm_area_struct will remain in use: so lower split_vma * places tmp vma above, and higher split_vma places tmp vma below. */ - if (start > vma->vm_start) { + if (start > tmp->vm_start) { int error; /* @@ -2731,13 +2741,14 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, * not exceed its limit; but let map_count go just above * its limit temporarily, to help free resources as expected. */ - if (end < vma->vm_end && mm->map_count >= sysctl_max_map_count) + if (end < tmp->vm_end && + mm->map_count > sysctl_max_map_count) return -ENOMEM; - error = __split_vma(mm, vma, start, 0); + error = __split_vma(mm, tmp, start, 0); if (error) return error; - prev = vma; + *prev = tmp; } /* Does it split the last one? */ @@ -2747,7 +2758,48 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, if (error) return error; } - vma = prev ? prev->vm_next : mm->mmap; + + *vma = *prev ? (*prev)->vm_next : mm->mmap; + + return 1; +} + +static inline void munmap_mlock_vma(struct vm_area_struct *vma, + unsigned long end) +{ + struct vm_area_struct *tmp = vma; + + while (tmp && tmp->vm_start < end) { + if (tmp->vm_flags & VM_LOCKED) { + vma->vm_mm->locked_vm -= vma_pages(tmp); + munlock_vma_pages_all(tmp); + } + tmp = tmp->vm_next; + } +} + +/* Munmap is split into 2 main parts -- this part which finds + * what needs doing, and the areas themselves, which do the + * work. This now handles partial unmappings. + * Jeremy Fitzhardinge + */ +int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, + struct list_head *uf) +{ + unsigned long end; + struct vm_area_struct *vma = NULL, *prev; + int ret = 0; + + if (!munmap_addr_sanity(start, len)) + return -EINVAL; + + len = PAGE_ALIGN(len); + + end = start + len; + + ret = munmap_lookup_vma(mm, &vma, &prev, start, end); + if (ret != 1) + return ret; if (unlikely(uf)) { /* @@ -2759,24 +2811,16 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, * split, despite we could. This is unlikely enough * failure that it's not worth optimizing it for. */ - int error = userfaultfd_unmap_prep(vma, start, end, uf); - if (error) - return error; + ret = userfaultfd_unmap_prep(vma, start, end, uf); + if (ret) + return ret; } /* * unlock any mlock()ed ranges before detaching vmas */ - if (mm->locked_vm) { - struct vm_area_struct *tmp = vma; - while (tmp && tmp->vm_start < end) { - if (tmp->vm_flags & VM_LOCKED) { - mm->locked_vm -= vma_pages(tmp); - munlock_vma_pages_all(tmp); - } - tmp = tmp->vm_next; - } - } + if (mm->locked_vm) + munmap_mlock_vma(vma, end); /* * Remove the vma's, and unmap the actual pages From patchwork Thu Jul 26 18:10:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10546291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0E37180E for ; Thu, 26 Jul 2018 18:10:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B43512BB4F for ; Thu, 26 Jul 2018 18:10:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B221D2BBF0; Thu, 26 Jul 2018 18:10:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A3FB2BB4F for ; Thu, 26 Jul 2018 18:10:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77B806B000C; Thu, 26 Jul 2018 14:10:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 702156B000D; Thu, 26 Jul 2018 14:10:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A4816B000E; Thu, 26 Jul 2018 14:10:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 041896B000C for ; Thu, 26 Jul 2018 14:10:54 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id n19-v6so1425670pgv.14 for ; Thu, 26 Jul 2018 11:10:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=LJIpTS7AipAVAK+VnSEOKQOMNRzQyPX3KP84kQq/oBM=; b=lIZ/l5Q12eK0RUL8ZbsLHogyjIPJ0rIgNM/EiNpW0PPaDECtQjEyxLMYYHufrDd725 Qd3JHE9EK/r6Fa/TGAqqoXMCxvGGy5dAMBiEvGsjGf2ZI23i1U9BiEh/1OaY4RXjnUXs 9BSq+/eUcHOev56HrbbcRZqEUemrDSA/WSOdQhEuf4KUoFCxy/OQnRuhEDccBMvrfZQI ydHe7XyzRzQGslVJaaCOn+qVBVM3pz5OJUpKuGmEusLPbp9OzQZXxWZgqi0wLAHQYPJO ViDvtnn5lHGSfmCrlkSgSYGypdhSg6lwaedAk0gNnxqgvCmFX/IRO1jq7Oig2gz1dYpe LwsA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: AOUpUlEN5fAMlZmcVotHli07mTpQ8IZvUYYKmZGfVGIhZeId95zMw0sa ifi0Cak0XdNWKWv77flUVV5bDjSNHfIxaEQvBfateTxJu3dtSpZAU2a52ZVpj2FJZHkvOwCMH2v E7/TqYID+bw7pNyyQa+1k0aS6tB4STw5rBO7R00bFph9WbyjlIgrP8GbIguuTutPm1w== X-Received: by 2002:a63:d401:: with SMTP id a1-v6mr2895478pgh.414.1532628653616; Thu, 26 Jul 2018 11:10:53 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcdOZWo18V0mxGll+2dP56Qjl+pdA3y3ihdWUfGCxsLJFmu1dg5jn7a832/P/wUCMw05U8X X-Received: by 2002:a63:d401:: with SMTP id a1-v6mr2895418pgh.414.1532628652366; Thu, 26 Jul 2018 11:10:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532628652; cv=none; d=google.com; s=arc-20160816; b=AeS94uUrA+35MtjiTQ1m2s7ujQZzS6M1kSmtcxeJd+HozxAT2lyZAvTiISGPjdiK8y lNEBRqRlWl0ophLX0Yk5w6UWyzS4tAtTfhw2LIZvFGY+7cPfbzydPJQBK+x4KEmrsxcD jdL4Mskb3jJASghI3z2Hpr0ToRHo9Rwkft5i73z6sKZLQ0nNnnEe1Qctb3n1Aogr8jOw S/uwUq6WRANyYLKV9JoWelOawA48jDz+QezzKSJc0CHi8Qqt1y2Q8/3tXOUaF4MSB1/s 9QDsfF4kQL4utwCjoo16ztmJLeo6VfrHKzH55Es6dBESg0msTd1qQTC5wOwASKGnNt48 JDcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=LJIpTS7AipAVAK+VnSEOKQOMNRzQyPX3KP84kQq/oBM=; b=g9bKAtINz/JwY13Q53LKGjQtHCnOAiHeSK9cKNeMziCp+HKlhWvEAmrw8MG5f1oZLz TAXj1k+FVFkMpVRJSuOvOf4f2gPJV9gQts0El3KeOjfcm2GuGWIKXQY5hD3affawXuaO XOhnZTN2GyFn7FINLUfEqZ5oMSjBtCOvYWZRiX2LpMwsWpYLeRodiPPOOJU+9PRIxEb/ QH9Y4ZRq8Y2y1+jwL53ttpe4Z+UEae3qZsDpv3/EYnXJLP6q2K12t+7emI2u21BCzgOg 6HsyRFTQcQf8BxFDECh6tL5wsspOsNRGHiA/SnzQmk7q80x/DPGCE996Phxc1Tmdz0fZ V1kw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4437.biz.mail.alibaba.com (out4437.biz.mail.alibaba.com. [47.88.44.37]) by mx.google.com with ESMTPS id s16-v6si1860571pgg.538.2018.07.26.11.10.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jul 2018 11:10:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) client-ip=47.88.44.37; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07402;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0T5ObfW8_1532628628; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0T5ObfW8_1532628628) by smtp.aliyun-inc.com(127.0.0.1); Fri, 27 Jul 2018 02:10:35 +0800 From: Yang Shi To: mhocko@kernel.org, willy@infradead.org, ldufour@linux.vnet.ibm.com, kirill@shutemov.name, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC v6 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap Date: Fri, 27 Jul 2018 02:10:14 +0800 Message-Id: <1532628614-111702-3-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1532628614-111702-1-git-send-email-yang.shi@linux.alibaba.com> References: <1532628614-111702-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When running some mmap/munmap scalability tests with large memory (i.e. > 300GB), the below hung task issue may happen occasionally. INFO: task ps:14018 blocked for more than 120 seconds. Tainted: G E 4.9.79-009.ali3000.alios7.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ps D 0 14018 1 0x00000004 ffff885582f84000 ffff885e8682f000 ffff880972943000 ffff885ebf499bc0 ffff8828ee120000 ffffc900349bfca8 ffffffff817154d0 0000000000000040 00ffffff812f872a ffff885ebf499bc0 024000d000948300 ffff880972943000 Call Trace: [] ? __schedule+0x250/0x730 [] schedule+0x36/0x80 [] rwsem_down_read_failed+0xf0/0x150 [] call_rwsem_down_read_failed+0x18/0x30 [] down_read+0x20/0x40 [] proc_pid_cmdline_read+0xd9/0x4e0 [] ? do_filp_open+0xa5/0x100 [] __vfs_read+0x37/0x150 [] ? security_file_permission+0x9b/0xc0 [] vfs_read+0x96/0x130 [] SyS_read+0x55/0xc0 [] entry_SYSCALL_64_fastpath+0x1a/0xc5 It is because munmap holds mmap_sem exclusively from very beginning to all the way down to the end, and doesn't release it in the middle. When unmapping large mapping, it may take long time (take ~18 seconds to unmap 320GB mapping with every single page mapped on an idle machine). Zapping pages is the most time consuming part, according to the suggestion from Michal Hocko [1], zapping pages can be done with holding read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write mmap_sem to cleanup vmas. But, some part may need write mmap_sem, for example, vma splitting. So, the design is as follows: acquire write mmap_sem lookup vmas (find and split vmas) detach vmas deal with special mappings downgrade_write zap pages free page tables release mmap_sem The vm events with read mmap_sem may come in during page zapping, but since vmas have been detached before, they, i.e. page fault, gup, etc, will not be able to find valid vma, then just return SIGSEGV or -EFAULT as expected. If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are considered as special mappings. They will be dealt with before zapping pages with write mmap_sem held. Basically, just update vm_flags. And, since they are also manipulated by unmap_single_vma() which is called by unmap_vma() with read mmap_sem held in this case, to prevent from updating vm_flags in read critical section, a new parameter, called "skip_flags" is added to unmap_region(), unmap_vmas() and unmap_single_vma(). If it is true, then just skip unmap those special mappings. Currently, the only place which pass true to this parameter is us. With this approach we don't have to re-acquire mmap_sem again to clean up vmas to avoid race window which might get the address space changed. And, since the lock acquire/release cost is managed to the minimum and almost as same as before, the optimization could be extended to any size of mapping without incurring significant penalty to small mappings. For the time being, just do this in munmap syscall path. Other vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain intact for stability reason. With the patches, exclusive mmap_sem hold time when munmap a 80GB address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to us level from second. munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock() { munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us | unmap_region(); munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | } Here the excution time of unmap_region() is used to evaluate the time of holding read mmap_sem, then the remaining time is used with holding exclusive lock. [1] https://lwn.net/Articles/753269/ Suggested-by: Michal Hocko Suggested-by: Kirill A. Shutemov Cc: Matthew Wilcox Cc: Laurent Dufour Cc: Andrew Morton Signed-off-by: Yang Shi --- include/linux/mm.h | 2 +- mm/memory.c | 41 ++++++++++++++++------ mm/mmap.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 123 insertions(+), 19 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a0fbb9f..e4480d8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool skip_vm_flags); /** * mm_walk - callbacks for walk_page_range diff --git a/mm/memory.c b/mm/memory.c index 7206a63..6a772bd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1514,7 +1514,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool skip_vm_flags) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1525,11 +1525,19 @@ static void unmap_single_vma(struct mmu_gather *tlb, if (end <= vma->vm_start) return; - if (vma->vm_file) - uprobe_munmap(vma, start, end); + /* + * Since unmap_single_vma might be called with read mmap_sem held + * in munmap optimization, so vm_flags can't be updated in this case. + * They have been updated before this call with write mmap_sem held. + * Here if skip_vm_flags is true, just skip the update. + */ + if (!skip_vm_flags) { + if (vma->vm_file) + uprobe_munmap(vma, start, end); - if (unlikely(vma->vm_flags & VM_PFNMAP)) - untrack_pfn(vma, 0, 0); + if (unlikely(vma->vm_flags & VM_PFNMAP)) + untrack_pfn(vma, 0, 0); + } if (start != end) { if (unlikely(is_vm_hugetlb_page(vma))) { @@ -1546,7 +1554,19 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ if (vma->vm_file) { i_mmap_lock_write(vma->vm_file->f_mapping); - __unmap_hugepage_range_final(tlb, vma, start, end, NULL); + if (!skip_vm_flags) { + /* + * The vma is being unmapped with read + * mmap_sem. + * Can't update vm_flags here, it has + * been updated before this call with + * write mmap_sem held. + */ + __unmap_hugepage_range(tlb, vma, start, + end, NULL); + } else + __unmap_hugepage_range_final(tlb, vma, + start, end, NULL); i_mmap_unlock_write(vma->vm_file->f_mapping); } } else @@ -1574,13 +1594,14 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool skip_vm_flags) { struct mm_struct *mm = vma->vm_mm; mmu_notifier_invalidate_range_start(mm, start_addr, end_addr); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); + unmap_single_vma(tlb, vma, start_addr, end_addr, NULL, + skip_vm_flags); mmu_notifier_invalidate_range_end(mm, start_addr, end_addr); } @@ -1604,7 +1625,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(mm); mmu_notifier_invalidate_range_start(mm, start, end); for ( ; vma && vma->vm_start < end; vma = vma->vm_next) { - unmap_single_vma(&tlb, vma, start, end, NULL); + unmap_single_vma(&tlb, vma, start, end, NULL, false); /* * zap_page_range does not specify whether mmap_sem should be @@ -1641,7 +1662,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addr tlb_gather_mmu(&tlb, mm, address, end); update_hiwater_rss(mm); mmu_notifier_invalidate_range_start(mm, address, end); - unmap_single_vma(&tlb, vma, address, end, details); + unmap_single_vma(&tlb, vma, address, end, details, false); mmu_notifier_invalidate_range_end(mm, address, end); tlb_finish_mmu(&tlb, address, end); } diff --git a/mm/mmap.c b/mm/mmap.c index 2504094..663a0c5 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -73,7 +73,7 @@ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool skip_flags); /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected @@ -1824,7 +1824,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, fput(file); /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, false); charged = 0; if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); @@ -2559,7 +2559,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool skip_flags) { struct vm_area_struct *next = prev ? prev->vm_next : mm->mmap; struct mmu_gather tlb; @@ -2567,7 +2567,7 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm, start, end); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, skip_flags); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, next ? next->vm_start : USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb, start, end); @@ -2778,6 +2778,79 @@ static inline void munmap_mlock_vma(struct vm_area_struct *vma, } } +/* + * Zap pages with read mmap_sem held + * + * uf is the list for userfaultfd + */ +static int do_munmap_zap_rlock(struct mm_struct *mm, unsigned long start, + size_t len, struct list_head *uf) +{ + unsigned long end = 0; + struct vm_area_struct *start_vma = NULL, *prev, *vma; + int ret = 0; + + if (!munmap_addr_sanity(start, len)) + return -EINVAL; + + len = PAGE_ALIGN(len); + + end = start + len; + + /* + * Need write mmap_sem to split vmas and detach vmas + * splitting vma up-front to save PITA to clean if it is failed + */ + if (down_write_killable(&mm->mmap_sem)) + return -EINTR; + + ret = munmap_lookup_vma(mm, &start_vma, &prev, start, end); + if (ret != 1) + goto out; + + if (unlikely(uf)) { + ret = userfaultfd_unmap_prep(start_vma, start, end, uf); + if (ret) + goto out; + } + + /* Handle mlocked vmas */ + if (mm->locked_vm) + munmap_mlock_vma(start_vma, end); + + /* Detach vmas from rbtree */ + detach_vmas_to_be_unmapped(mm, start_vma, prev, end); + + /* + * Clear uprobe, VM_PFNMAP and hugetlb mapping in advance since they + * need update vm_flags with write mmap_sem + */ + vma = start_vma; + for ( ; vma && vma->vm_start < end; vma = vma->vm_next) { + if (vma->vm_file) + uprobe_munmap(vma, vma->vm_start, vma->vm_end); + if (unlikely(vma->vm_flags & VM_PFNMAP)) + untrack_pfn(vma, 0, 0); + if (is_vm_hugetlb_page(vma)) + vma->vm_flags &= ~VM_MAYSHARE; + } + + downgrade_write(&mm->mmap_sem); + + /* Zap mappings with read mmap_sem */ + unmap_region(mm, start_vma, prev, start, end, true); + + arch_unmap(mm, start_vma, start, end); + remove_vma_list(mm, start_vma); + up_read(&mm->mmap_sem); + + return 0; + +out: + up_write(&mm->mmap_sem); + return ret; +} + /* Munmap is split into 2 main parts -- this part which finds * what needs doing, and the areas themselves, which do the * work. This now handles partial unmappings. @@ -2826,7 +2899,7 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, * Remove the vma's, and unmap the actual pages */ detach_vmas_to_be_unmapped(mm, vma, prev, end); - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, false); arch_unmap(mm, vma, start, end); @@ -2836,6 +2909,17 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, return 0; } +static int vm_munmap_zap_rlock(unsigned long start, size_t len) +{ + int ret; + struct mm_struct *mm = current->mm; + LIST_HEAD(uf); + + ret = do_munmap_zap_rlock(mm, start, len, &uf); + userfaultfd_unmap_complete(mm, &uf); + return ret; +} + int vm_munmap(unsigned long start, size_t len) { int ret; @@ -2855,10 +2939,9 @@ int vm_munmap(unsigned long start, size_t len) SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len) { profile_munmap(addr); - return vm_munmap(addr, len); + return vm_munmap_zap_rlock(addr, len); } - /* * Emulation of deprecated remap_file_pages() syscall. */ @@ -3146,7 +3229,7 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu(&tlb, mm, 0, -1); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); + unmap_vmas(&tlb, vma, 0, -1, false); free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb, 0, -1);