From patchwork Thu Jun 2 14:56:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jakub_Mat=C4=9Bna?= X-Patchwork-Id: 12867950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CA5CC43334 for ; Thu, 2 Jun 2022 14:56:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B01066B0071; Thu, 2 Jun 2022 10:56:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AAF576B0073; Thu, 2 Jun 2022 10:56:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 972076B0074; Thu, 2 Jun 2022 10:56:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8AFA86B0071 for ; Thu, 2 Jun 2022 10:56:28 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5083C60EEE for ; Thu, 2 Jun 2022 14:56:28 +0000 (UTC) X-FDA: 79533596856.26.92FAE47 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf15.hostedemail.com (Postfix) with ESMTP id 050B0A0064 for ; Thu, 2 Jun 2022 14:56:01 +0000 (UTC) Received: by mail-ej1-f41.google.com with SMTP id fu3so8998399ejc.7 for ; Thu, 02 Jun 2022 07:56:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xHh+H7BQlsfCtAThje/xIUlqgtGS6GOCshB5FBGaRKo=; b=Xh4ANWAuyg1cGJ+Dc3w4iprlH+ngwcPVcnneCJIhSUiAvDORma8aDPn+jO+RerLhrD WQOw4rwXU3QNw1K8AKEt4d+PZA7qERjd99V7OhPKdQsNjMdr6fyz4mz88nOKJ4nMWW5r PfVMkS7Q2ZIiWJ1TcouljeNAz58TyHZZbg2zarFPk44XO9QCZWGU7wLrn7NhnSgjXDEm Ve4QHydnVC83yUg2pkJc6wmb+VdAkwcoyB9dzUGjFHICuB/OwFiFrb3dLd32vT97O6A0 HTrTO22GsYmirQRf1EPplon/oZqfyKAVGMB3h4DzpjCe8jkTt8L/4w19GbLaM8/ndhPq a0UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xHh+H7BQlsfCtAThje/xIUlqgtGS6GOCshB5FBGaRKo=; b=5l+HZgs04X7ShWLMZbbGJZAWiTfWRa7R92e9UG8joy3fAjhonlk2GUlpoapzI1n694 7tsVXTDKh7GQ/Zy10w1rvBLMqGC9nfXE1rLqQ4K6dYVX+/4oTC9prcitYb2lQiMYgM5p +lO1LPyj9mrX5UGIuuFYsiHDou2B4eZBUqcOZRvzY6ME6HfqOXtajbf7+LFl9h078+hP eKN5Dvg5e9apkInusq5ZxOzI/pL3Sen+uXq4UtMFDOiSyTNZA3dlI+yLRlBR6NwpjXIy +etuWfpahV3vqhOvFijGEcA/mbx2ngE9li6gxC0/Ch6gOSajDukSEG0vnSax/Kc+AGHO jXuQ== X-Gm-Message-State: AOAM533cTnT54H4rFBow/0wOKADdKcsxiCUrbKLzv13SFsbbsFerZCmC 2UwE8C2J9vRULcIRg/BP12s= X-Google-Smtp-Source: ABdhPJzFpDSOoca0Grhs4EW9ifXaEdKhAsPuLn92ZsZoqiBqc/ljbC4VpVXsTCJR4BkYygkcveUQcg== X-Received: by 2002:a17:906:7254:b0:6fe:5637:cbe6 with SMTP id n20-20020a170906725400b006fe5637cbe6mr4645754ejk.612.1654181784247; Thu, 02 Jun 2022 07:56:24 -0700 (PDT) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id x9-20020a1709064a8900b006f3ef214e2csm1873452eju.146.2022.06.02.07.56.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jun 2022 07:56:23 -0700 (PDT) Received: by orion.localdomain (Postfix, from userid 1003) id 2A742A0886; Thu, 2 Jun 2022 16:56:57 +0200 (CEST) From: =?utf-8?q?Jakub_Mat=C4=9Bna?= To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?utf-8?q?Jakub_Mat=C4=9Bna?= Subject: [PATCH 1/2] [PATCH 1/2] mm: refactor of vma_merge() Date: Thu, 2 Jun 2022 16:56:41 +0200 Message-Id: <20220602145642.16948-2-matenajakub@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220602145642.16948-1-matenajakub@gmail.com> References: <20220602145642.16948-1-matenajakub@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 664f8cukw6kgzguwppictg7o8ng1q7q4 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Xh4ANWAu; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of matenajakub@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=matenajakub@gmail.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 050B0A0064 X-HE-Tag: 1654181761-905568 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refactor vma_merge() to make it shorter and more understandable. Main change is the elimination of code duplicity in the case of merge next check. This is done by first doing checks and caching the results before executing the merge itself. The variable 'area' is divided into 'mid' and 'res' as previously it was used for two purposes, as the middle VMA between prev and next and also as the result of the merge itself. Exit paths are also unified. Signed-off-by: Jakub Matěna --- mm/mmap.c | 87 +++++++++++++++++++++++-------------------------------- 1 file changed, 37 insertions(+), 50 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 313b57d55a63..91100fdc400a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1170,8 +1170,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, struct anon_vma_name *anon_name) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT; - struct vm_area_struct *area, *next; - int err; + struct vm_area_struct *mid, *next, *res; + int err = -1; + bool merge_prev = false; + bool merge_next = false; /* * We later require that vma->vm_flags == vm_flags, @@ -1181,75 +1183,60 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, return NULL; next = vma_next(mm, prev); - area = next; - if (area && area->vm_end == end) /* cases 6, 7, 8 */ + mid = next; + if (next && next->vm_end == end) /* cases 6, 7, 8 */ next = next->vm_next; /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); - VM_WARN_ON(area && end > area->vm_end); + VM_WARN_ON(mid && end > mid->vm_end); VM_WARN_ON(addr >= end); - /* - * Can it merge with the predecessor? - */ + /* Can we merge the predecessor? */ if (prev && prev->vm_end == addr && mpol_equal(vma_policy(prev), policy) && can_vma_merge_after(prev, vm_flags, anon_vma, file, pgoff, vm_userfaultfd_ctx, anon_name)) { - /* - * OK, it can. Can we now merge in the successor as well? - */ - if (next && end == next->vm_start && - mpol_equal(policy, vma_policy(next)) && - can_vma_merge_before(next, vm_flags, - anon_vma, file, - pgoff+pglen, - vm_userfaultfd_ctx, anon_name) && - is_mergeable_anon_vma(prev->anon_vma, - next->anon_vma, NULL)) { - /* cases 1, 6 */ - err = __vma_adjust(prev, prev->vm_start, - next->vm_end, prev->vm_pgoff, NULL, - prev); - } else /* cases 2, 5, 7 */ - err = __vma_adjust(prev, prev->vm_start, - end, prev->vm_pgoff, NULL, prev); - if (err) - return NULL; - khugepaged_enter_vma_merge(prev, vm_flags); - return prev; + merge_prev = true; } - - /* - * Can this new request be merged in front of next? - */ + /* Can we merge the successor? */ if (next && end == next->vm_start && mpol_equal(policy, vma_policy(next)) && can_vma_merge_before(next, vm_flags, anon_vma, file, pgoff+pglen, vm_userfaultfd_ctx, anon_name)) { + merge_next = true; + } + /* Can we merge both the predecessor and the successor? */ + if (merge_prev && merge_next && + is_mergeable_anon_vma(prev->anon_vma, + next->anon_vma, NULL)) { /* cases 1, 6 */ + err = __vma_adjust(prev, prev->vm_start, + next->vm_end, prev->vm_pgoff, NULL, + prev); + res = prev; + } else if (merge_prev) { /* cases 2, 5, 7 */ + err = __vma_adjust(prev, prev->vm_start, + end, prev->vm_pgoff, NULL, prev); + res = prev; + } else if (merge_next) { if (prev && addr < prev->vm_end) /* case 4 */ err = __vma_adjust(prev, prev->vm_start, - addr, prev->vm_pgoff, NULL, next); - else { /* cases 3, 8 */ - err = __vma_adjust(area, addr, next->vm_end, - next->vm_pgoff - pglen, NULL, next); - /* - * In case 3 area is already equal to next and - * this is a noop, but in case 8 "area" has - * been removed and next was expanded over it. - */ - area = next; - } - if (err) - return NULL; - khugepaged_enter_vma_merge(area, vm_flags); - return area; + addr, prev->vm_pgoff, NULL, next); + else /* cases 3, 8 */ + err = __vma_adjust(mid, addr, next->vm_end, + next->vm_pgoff - pglen, NULL, next); + res = next; } - return NULL; + /* + * Cannot merge with predecessor or successor or error in __vma_adjust? + */ + if (err) + return NULL; + khugepaged_enter_vma_merge(res, vm_flags); + return res; } /* From patchwork Thu Jun 2 14:56:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jakub_Mat=C4=9Bna?= X-Patchwork-Id: 12867952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A036ACCA478 for ; Thu, 2 Jun 2022 14:56:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C06246B0074; Thu, 2 Jun 2022 10:56:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB54A6B0075; Thu, 2 Jun 2022 10:56:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990966B007B; Thu, 2 Jun 2022 10:56:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 65F826B0075 for ; Thu, 2 Jun 2022 10:56:29 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 34DE535800 for ; Thu, 2 Jun 2022 14:56:29 +0000 (UTC) X-FDA: 79533596898.16.1142F27 Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by imf08.hostedemail.com (Postfix) with ESMTP id 93BAB160066 for ; Thu, 2 Jun 2022 14:56:02 +0000 (UTC) Received: by mail-ej1-f53.google.com with SMTP id rq11so10502928ejc.4 for ; Thu, 02 Jun 2022 07:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=plkRHxZRk9CMIn7YOznv+vuUGRy66YRIHBjRF967Ovg=; b=HftDeftCoaRwOnKYKg0+jnbxtHNvBbjODuhXYFO2QbjQW5f2p1vZ7QR7lSPhi1Tpo6 zQIf0SsRnaoMYlNDJFjxwnMh0QxywoReN/NUiqPmEw9khk+s1id55epnrkyctNSlBng6 yZgze0/sBJmFciSLBnov0zru92Wm/r0oJ/bJQETbLT9tapPwyTQklrP8gqMqqvP6OWGS AUb1idz8HGGjiiKLtrWKIUKRczniTp5YLo1dlGC7Zlj0qRCzdWwvRhOIqYYXPuIxSM13 SoaHQWmF9A4AIz75alU5K+N87Ip4goG9RRLb7djoRNAixjUaLSzb+gafziTdUn3BhBUu J7ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=plkRHxZRk9CMIn7YOznv+vuUGRy66YRIHBjRF967Ovg=; b=SnR5ywAPiBV9HOh5C3e4WzfHPk5mKcpM1yVz0fVvMAdfhP6PDQ8+eZZUzJCePUyeV3 NV1NrnF31RKM55fKmn2j8W7nNC3MWv9NlemisV5UljOjIhpNVGJgb2Gh+qRYUFRWD5Ah 4RaxI3/iiYK6bdObb/tMdnohN6iq9KlWXhLP1vsgKOIAxjsW7/fpaMqWa8GqaMLr4ugK TXHjHpeIfuhqLFFmYPBs0bLrwXwEgTJ8k5SaZJvXSOZC9WrKaVOdJaj4Fcgv03aE++6V 2xI3yMPD7PiJkwaEOFJu3SBa8YFQ6+2h29RL+zl5iGD3gaxcgOtStp8SZRIRpBvVLnTi Utqg== X-Gm-Message-State: AOAM5336KjEJlZfTRXOnZ83vW3xxLf3HcK3cWkwwnwvPyCzCmSrhzhda 8t8SIAy9TEVcGsrN7Ylz9rk= X-Google-Smtp-Source: ABdhPJydHQyQWjXz9p+kXVNMSBYSkBhpBXvIgTo6+N7WpZ4t+zvxXoay5Q2cKcCAKpn4Duv4qsQt0A== X-Received: by 2002:a17:907:728a:b0:6fe:f10b:80cc with SMTP id dt10-20020a170907728a00b006fef10b80ccmr4452079ejc.22.1654181784494; Thu, 02 Jun 2022 07:56:24 -0700 (PDT) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id h15-20020a170906590f00b007081c5ce04dsm1589173ejq.58.2022.06.02.07.56.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jun 2022 07:56:23 -0700 (PDT) Received: by orion.localdomain (Postfix, from userid 1003) id 2E073A0887; Thu, 2 Jun 2022 16:56:57 +0200 (CEST) From: =?utf-8?q?Jakub_Mat=C4=9Bna?= To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?utf-8?q?Jakub_Mat=C4=9Bna?= Subject: [PATCH 2/2] [PATCH 2/2] mm: add merging after mremap resize Date: Thu, 2 Jun 2022 16:56:42 +0200 Message-Id: <20220602145642.16948-3-matenajakub@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220602145642.16948-1-matenajakub@gmail.com> References: <20220602145642.16948-1-matenajakub@gmail.com> MIME-Version: 1.0 X-Stat-Signature: af7ehc6s5tyqq6dkw87px6wc1e4q8yw6 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HftDeftC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of matenajakub@gmail.com designates 209.85.218.53 as permitted sender) smtp.mailfrom=matenajakub@gmail.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 93BAB160066 X-HE-Tag: 1654181762-302667 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When mremap call results in expansion, it might be possible to merge the VMA with the next VMA which might become adjacent. This patch adds vma_merge call after the expansion is done to try and merge. Signed-off-by: Jakub Matěna --- mm/mremap.c | 19 +++++++++- tools/testing/selftests/vm/mremap_test.c | 47 +++++++++++++++++++++++- 2 files changed, 63 insertions(+), 3 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 0b93fac76851..66970dcd636a 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -23,6 +24,7 @@ #include #include #include +#include #include #include @@ -1014,6 +1016,9 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, /* can we just expand the current mapping? */ if (vma_expandable(vma, new_len - old_len)) { long pages = (new_len - old_len) >> PAGE_SHIFT; + unsigned long extension_start = addr + old_len; + unsigned long extension_end = addr + new_len; + pgoff_t extension_pgoff = vma->vm_pgoff + (old_len >> PAGE_SHIFT); if (vma->vm_flags & VM_ACCOUNT) { if (security_vm_enough_memory_mm(mm, pages)) { @@ -1022,8 +1027,18 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, } } - if (vma_adjust(vma, vma->vm_start, addr + new_len, - vma->vm_pgoff, NULL)) { + /* + * Function vma_merge() is called on the extension we are adding to + * the already existing vma, vma_merge() will merge this extension with + * the already existing vma (expand operation itself) and possibly also + * with the next vma if it becomes adjacent to the expanded vma and + * otherwise compatible. + */ + vma = vma_merge(mm, vma, extension_start, extension_end, + vma->vm_flags, vma->anon_vma, vma->vm_file, + extension_pgoff, vma_policy(vma), + vma->vm_userfaultfd_ctx, anon_vma_name(vma)); + if (!vma) { vm_unacct_memory(pages); ret = -ENOMEM; goto out; diff --git a/tools/testing/selftests/vm/mremap_test.c b/tools/testing/selftests/vm/mremap_test.c index db0270127aeb..0865a6cb5bdb 100644 --- a/tools/testing/selftests/vm/mremap_test.c +++ b/tools/testing/selftests/vm/mremap_test.c @@ -118,6 +118,48 @@ static unsigned long long get_mmap_min_addr(void) return addr; } +/* + * This test validates that merge is called when expanding a mapping. + * Mapping containing three pages is created, middle page is unmapped + * and then the mapping containing the first page is expanded so that + * it fills the created hole. The two parts should merge creating + * single mapping with three pages. + */ +static void mremap_expand_merge(unsigned long page_size) +{ + char *test_name = "mremap expand merge"; + FILE *fp; + char *line = NULL; + size_t len = 0; + bool success = false; + + char *start = mmap(NULL, 3 * page_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + munmap(start + page_size, page_size); + mremap(start, page_size, 2 * page_size, 0); + + fp = fopen("/proc/self/maps", "r"); + if (fp == NULL) { + ksft_test_result_fail("%s\n", test_name); + return; + } + + while(getline(&line, &len, fp) != -1) { + char *first = strtok(line,"- "); + void *first_val = (void *) strtol(first, NULL, 16); + char *second = strtok(NULL,"- "); + void *second_val = (void *) strtol(second, NULL, 16); + if (first_val == start && second_val == start + 3 * page_size) { + success = true; + break; + } + } + if (success) + ksft_test_result_pass("%s\n", test_name); + else + ksft_test_result_fail("%s\n", test_name); + fclose(fp); +} + /* * Returns the start address of the mapping on success, else returns * NULL on failure. @@ -336,6 +378,7 @@ int main(int argc, char **argv) int i, run_perf_tests; unsigned int threshold_mb = VALIDATION_DEFAULT_THRESHOLD; unsigned int pattern_seed; + int num_expand_tests = 1; struct test test_cases[MAX_TEST]; struct test perf_test_cases[MAX_PERF_TEST]; int page_size; @@ -407,12 +450,14 @@ int main(int argc, char **argv) (threshold_mb * _1MB >= _1GB); ksft_set_plan(ARRAY_SIZE(test_cases) + (run_perf_tests ? - ARRAY_SIZE(perf_test_cases) : 0)); + ARRAY_SIZE(perf_test_cases) : 0) + num_expand_tests); for (i = 0; i < ARRAY_SIZE(test_cases); i++) run_mremap_test_case(test_cases[i], &failures, threshold_mb, pattern_seed); + mremap_expand_merge(page_size); + if (run_perf_tests) { ksft_print_msg("\n%s\n", "mremap HAVE_MOVE_PMD/PUD optimization time comparison for 1GB region:");