From patchwork Mon May 16 12:53:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jakub_Mat=C4=9Bna?= X-Patchwork-Id: 12850765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF8DCC433EF for ; Mon, 16 May 2022 12:53:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E409C6B007D; Mon, 16 May 2022 08:53:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DF4FB6B0075; Mon, 16 May 2022 08:53:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4D116B007E; Mon, 16 May 2022 08:53:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AC5D36B0078 for ; Mon, 16 May 2022 08:53:48 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 888DA32C97 for ; Mon, 16 May 2022 12:53:48 +0000 (UTC) X-FDA: 79471598136.24.F4E2F3C Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf01.hostedemail.com (Postfix) with ESMTP id 80663400C7 for ; Mon, 16 May 2022 12:53:33 +0000 (UTC) Received: by mail-wm1-f45.google.com with SMTP id i20-20020a05600c355400b0039456976dcaso7495916wmq.1 for ; Mon, 16 May 2022 05:53:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=KVPhUnjtby93H1A1uu/hc4Ko2DVpGt4uhH1faHyityo=; b=XllTgNYK50kbDMyE4dyj5x0qu8CtLWg5XpRR9eNfxroTcObnNNxtEtVnFy+w2hyVNz LWHe4GKmSaj+24mz/CvwS9Z6/tvBfWPCh0u4pR8T8/Hp2Ux4ztCAlNXCRgSF2R3eMiNt yuktsXLDqqjhKMCIM0g2AM7wguoyBjHqlrRPuJBmbjDuNS2/3C3oXCbgqmmXChRldavi dU4VOzbV4yJMLPuf0V6g2sYUj4XDWZGd3OR6r8hPiPQ5Xghsf3P+ZjKCL5Mzx0hnmJmB mSYx3cfFHk/x6b0Tnud7aLMe0DE8V1SyJTZfO4obXZNNpb7LMplZgKQc/kJjxOhV1lPz ofJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=KVPhUnjtby93H1A1uu/hc4Ko2DVpGt4uhH1faHyityo=; b=ZwQ0os1hVlbGWv9KRDqV5/nPcl1sV8d2Tj0keITO/k3Z+6BidI2lrT/RwEbE4bLv6j hHaDn1JXHPNkL9eV/AxENnEuUFR0cmYDWHyrZQWtD7KBGk65WMarGG/zE+Skgrl1J3YR v/HjRKfquq5s/WgKf2Xrdfx0d6/cfDZLTJTYQNwF02qbxGig63ih7LWFrWEFq56TySFc 7M4K78PlTqottRYhfZAN+5p5UUu+OHdeo9Ypkx35BPAZ1F5Bs+WbM4gUJ417NvIHOIXS 21Zx+B4RrzqEpET7rutcfeyfxEupMmElh3fTSbjXFdVhXsyIpIY/074965vrf2mIeCRL QVZA== X-Gm-Message-State: AOAM5311jeWc3+580uE48zNWfFPNB7gggZFaj2qiYH8+OEYxm+3FRyEG t6BkK6af1OXpIo52+Y590tA= X-Google-Smtp-Source: ABdhPJz836i6dyGykr8mqqydrdqtrl2t/GqFcsjXQE6SUcFQQ98MmBNXIlmgtRUQ79168UBbsAaoQg== X-Received: by 2002:a1c:29c3:0:b0:350:9797:b38f with SMTP id p186-20020a1c29c3000000b003509797b38fmr27109456wmp.22.1652705626773; Mon, 16 May 2022 05:53:46 -0700 (PDT) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id d6-20020adfc086000000b0020c5253d8fcsm11550041wrf.72.2022.05.16.05.53.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 May 2022 05:53:45 -0700 (PDT) Received: by orion.localdomain (Postfix, from userid 1003) id 89B18A0E75; Mon, 16 May 2022 14:54:07 +0200 (CEST) From: =?utf-8?q?Jakub_Mat=C4=9Bna?= To: linux-mm@kvack.org Cc: patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, david@redhat.com, =?utf-8?q?Jakub_Mat=C4=9Bna?= Subject: [RFC PATCH v3 0/6] Removing limitations of merging anonymous VMAs Date: Mon, 16 May 2022 14:53:59 +0200 Message-Id: <20220516125405.1675-1-matenajakub@gmail.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 X-Stat-Signature: wknrdrupxmhpf9hwhrucjt56gxazsgcr X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 80663400C7 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=XllTgNYK; spf=pass (imf01.hostedemail.com: domain of matenajakub@gmail.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=matenajakub@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-HE-Tag: 1652705613-317789 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a series of patches that try to improve merge success rate when VMAs are being moved, resized or otherwise modified. Motivation In the current kernel it is impossible to merge two anonymous VMAs if one of them was moved. That is because VMA's page offset is set according to the virtual address where it was created and in order to merge two VMAs page offsets need to follow up. Another problem when merging two faulted VMA's is their anon_vma. In current kernel these anon_vmas have to be the one and the same. Otherwise merge is again not allowed. There are several places from which vma_merge() is called and therefore several use cases that might profit from this upgrade. These include mmap (that fills a hole between two VMAs), mremap (that moves VMA next to another one or again perfectly fills a hole), mprotect (that modifies protection and allows merging with a neighbor) and brk (that expands VMA so that it is adjacent to a neighbor). Missed merge opportunities increase the number of VMAs of a process and in some cases can cause problems when a max count is reached. Solution The series solves the first problem with page offsets by updating them when the VMA is moved to a different virtual address (patch 4). As for the second problem, merging of VMAs with different anon_vma is allowed under some conditions (patch 5). Another missed opportunity in the current kernel is when mremap enlarges an already existing VMA and it is possible to merge with following VMA (patch 2). Patch 1 refactors function vma_merge and makes it easier to understand and also allows relatively seamless tracing of successful merges introduced by the patch 6. Patch 3 introduces migration waiting and rmap locking into the pagewalk mechnism, which is necessary for patche 4 and 5. Limitations For both problems solution works only for VMAs that do not share physical pages with other processes (usually child or parent processes). This is checked by looking at anon_vma of the respective VMA and also by looking at mapcount of individual pages. The reason why it is not possible or at least not easy to accomplish is that each physical page has a pointer to anon_vma and page offset. And when this physical page is shared we cannot simply change these parameters without affecting all of the VMAs mapping this physical page. Good thing is that this case amounts only for about 1-3% of all merges (measured on jemalloc (0%), redis (2.7%) and kcbench (1.2%) tests) that fail to merge in the current kernel. Measuring also shows slight increase in running time, jemalloc (0.3%), redis (1%), kcbench (1%). More extensive data can be viewed at https://home.alabanda.cz/share/results.png Changelog Pagewalk - previously page struct has been accessed using follow_page() which goes through the whole pagewalk for each call. This version uses walk_page_vma() which goes through all the necessary pages at the pte level (vm_normal_page() is used there). Pgoff update was previously performed at the beginning of copy_vma() for all the pages (page->index specifically) and also for the pgoff variable used to construct the VMA copy. Now the update of individual pages is done later in move_page_tables(). This makes more sense because move_page_tables() moves all the pages to the new VMA anyway and this again spares some otherwise duplicate page walking. Anon_vma update for mprotect cases is done in __vma_adjust(). For mremap cases the update is done in move_page_tables() together with the page offset update. Previously the anon_vma update was always handled in __vma_adjust() but it was not done in all necessary cases. More details are mentioned in the concerned patches. Questions Is it necessary to check mapcount of individual pages of the VMA to determine if they are shared with other processes? Is it even possible when VMA or respectivelly its anon_vma is not shared? So far as my knowledge of kernel goes, it seems that checking individual pages is not necessary and check on the level of anon_vma is suficient. KSM would theoretically interfere with page mapcount, but it is temporarily disabled before move_vma() in mremap syscall happens. Does anyone know about something else that can change mapcount without anon_vma knowing? This series of patches and documentation of the related code will be part of my master's thesis. This patch series is based on tag v5.18-rc2. This is a third version. Jakub Matěna (6): mm: refactor of vma_merge() mm: add merging after mremap resize mm: add migration waiting and rmap locking to pagewalk mm: adjust page offset in mremap mm: enable merging of VMAs with different anon_vmas mm: add tracing for VMA merges fs/exec.c | 2 +- fs/proc/task_mmu.c | 4 +- include/linux/mm.h | 4 +- include/linux/pagewalk.h | 15 +- include/linux/rmap.h | 19 ++- include/trace/events/mmap.h | 83 ++++++++++ mm/internal.h | 12 ++ mm/mmap.c | 291 ++++++++++++++++++++++++++---------- mm/mremap.c | 153 ++++++++++++++----- mm/pagewalk.c | 75 +++++++++- mm/rmap.c | 144 ++++++++++++++++++ 11 files changed, 670 insertions(+), 132 deletions(-)