From patchwork Sat Jan 11 00:46:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13935660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D9AE77188 for ; Sat, 11 Jan 2025 00:46:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4AD36B00AB; Fri, 10 Jan 2025 19:46:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFB0D6B00AC; Fri, 10 Jan 2025 19:46:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC31D6B00AE; Fri, 10 Jan 2025 19:46:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 963A66B00AB for ; Fri, 10 Jan 2025 19:46:25 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 46066AFA0E for ; Sat, 11 Jan 2025 00:46:25 +0000 (UTC) X-FDA: 82993329930.14.ABB806D Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf20.hostedemail.com (Postfix) with ESMTP id BB3411C0015 for ; Sat, 11 Jan 2025 00:46:23 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=tYIIPFSQ; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of sj@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736556383; a=rsa-sha256; cv=none; b=cqTYuRSyP86gBrBx1AtGdsCqacgMlb8IuHSWzjuxcNSlRIEp3ShLlKtEOJOYIcO2bwTkRz VQNR6JrVsbVh5YwBn90u/zIrlOxmY/TepcYlbyTWXEHGnoJX1ppT6qrMOprsVf4xwNytqB NL32bEq1ozionv3FytKl10SqsjXsQ/4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=tYIIPFSQ; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of sj@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736556383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=J63PkMl+j2hfQ/awInNp2AdFScgVH7Fz9tlt6ZrfB48=; b=rbboO51s/oj3Oi4gOUNi26w42aDLp+d3eguLYFhcnfnDiSBeMH4Rub/OpIa29cnvSGOqRN BRohmE1qVF3Wbbr8VVF8H5Hg/dLYv5W0z8TzZPiLAld7T8S4woDt/ZnOLl5umTWG67sEjg HFLg//+RH+p5JLbvMiKWAJtkYSqZi7U= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 86556A42773; Sat, 11 Jan 2025 00:44:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 68FB9C4CED6; Sat, 11 Jan 2025 00:46:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1736556382; bh=foRgl5imLj1rA3LH5J0pIYz2bryPDPVy04AvXejUU24=; h=From:To:Cc:Subject:Date:From; b=tYIIPFSQ81oLpVqvO80tl0IGWoNcSpjWJ1GC+7gG2FcPcNuTUHhFVFwm/m2l9GLua 1PqkNwtXZRSH0F3mI6hZkC2vBaAaXo39am3xMa08bfDOQNIPYOkyuRnHhw2waacTwv 6E+SmRpRBTFIsYN7epk3QlagY34xEeLhzFq9P7F2Q7mlFVKkB811BTp4srxrDnWHFG 2SFQOctqnUyyk007eyIG296aFVV/YQytcRxwFhRN3wuGMYC7sb4wrl/sHcP6lBT8ad +Sy6fbytnHEt+H6FZWHpGTBjvZcKi3UO9P2+HiU/JJTapAWHk4H9GTxiB9eSRetUin J4HeR8PZxuO7g== From: SeongJae Park To: Cc: SeongJae Park , Andrew Morton , Jens Axboe , Pavel Begunkov , damon@lists.linux.dev, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH] mm/madvise: remove redundant mmap_lock operations from process_madvise() Date: Fri, 10 Jan 2025 16:46:18 -0800 Message-Id: <20250111004618.1566-1-sj@kernel.org> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BB3411C0015 X-Stat-Signature: 4fuj3whkxb64zo54irb9qoocor154dpd X-Rspam-User: X-HE-Tag: 1736556383-390342 X-HE-Meta: U2FsdGVkX19iDwCOi17ix2ozqJVBQuj+7utAosauXdHbIc5G++rAyJ1LzfKZ0DNvs771Gm2r/ZcPVWofzTzbCB2FXiypBf64mJ7zBGtYG7hPZrAg62qH9i8vfyHkWlujl9GSjHw3MgWe+nuCGoBgtLy91W8SJPAWEqf3X2VIjIigQ+7K3Qly/7ync09AihBLL2jnOvBA2f96mLVNf7torqSv7+75i/jEds/L8uRj9rGvsv8Mvm9XQYFOo7gJu+qk6jjTWAma9SSP6Bae5FBQxdYH0EIdfH3m+24We8fSUDOLIO5Rkw/npV4MD8jMLKAHXA/Rdv/mZ7K/H1p3Jy02WMCWrYxYeHx6Fyx8snS1G75GgTkllQO0nVOlNWoyabK11OlcP6FoWcWXUk4llXA9bUpjlLW7zBKrsq5lqLeaqQTH1pM+w8BUVQ1fFz++jZfbbau+LBDKmtcLDNgOwMS5Df5AnfgOWkGt3RiARdxTD+RKteeiJsVz1eULsqtxQDoypb71SmhaLGnIVDCYYr1bvFGGISnEnN7IUFBFjW57jaliVMhzTtBv3/ZCULouV07QrWK84YferZbuX+sf4nhPyRimt2Z74hT5k29AoCPWm0MYlqATQ6thc21ZKCpTQg22HUI9o9FgOM/Yet9osK9vjeef6Xnlox1S+DKoiF9hLedkgG/ZMrTcHwiYKyhQBDnnYlxkCx2MojZRAMMdgvXev/sFzyRK8ReHGdwjm7BsvjKodm6bzwRYRfG2QnXv1BukrAwvzyw0KNLE4qKuqkPb2CoWY5Xrggm/zFW6iR/4PYyk+M6L3G7wXieQM6Y5yzQxVa62GybJ3ABNVBw/qj7m5OxSeqNnz9MmUmF850YfNZr2KVEKi6EUWcYadHYdaihEYSbSswYrzArBT2jYdfMBudG/DXchGOo2Qc1v4zoHoEB81d3l0V3BAM++u4aWl7hZwXjRNzYtPeR6EXO0oAB npGlKiKC D3QsCiiMwd+rJR6ibXRTtlwqPmKpz67HTzrGHhFrj9bE1Ul3i9BVazBAV1oOG9EVdmVN9zP7xLlhyxDjvNZybSht6I7ZpyY74LCxgr+uzrR1AWxUk0agrfYDsi3YLGz99Zha56K5gWLEsK/bfCI08N9pLg6h0++2BbTgkO3H/heyGyGG7iBaHEnVVRzjBb3alzyLu7Bn071drFJ1704zCjot7SebWRwbwebAr/iJkHXoMdIKBLMpCBGwjt7bQH+AJ6mtcI+H4IUgl4rc3i6bHo14796mydKWa87QTxhD6kWWZsq/hB4RGtouY6QSW8TX0qT3RrHaVk6QGOYvT3oBcgaDidKX3J5cG8KRJquVlt3SIFb/KZeTiS+92ig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: process_madvise() calls do_madvise() for each address range. Then, each do_madvise() invocation holds and releases same mmap_lock. Optimize the redundant lock operations by doing the locking in process_madvise(), and inform do_madvise() that the lock is already held and therefore can be skipped. Evaluation ========== I measured the time to apply MADV_DONTNEED advice to 256 MiB memory using multiple madvise() calls, 4 KiB per each call. I also do the same with process_madvise(), but with varying iovec size from 1 to 1024. The source code for the measurement is available at GitHub[1]. The measurement results are as below. 'sz_batches' column shows the iovec size of process_madvise() calls. '0' is for madvise() calls case. 'before' and 'after' columns are the measured time to apply MADV_DONTNEED to the 256 MiB memory buffer in nanoseconds, on kernels that built without and with this patch, respectively. So lower value means better efficiency. 'after/before' column is the ratio of 'after' to 'before'. sz_batches before after after/before 0 124062365 96670188 0.779206393494111 1 136341258 113915688 0.835518827323714 2 105314942 78898211 0.749164453796119 4 82012858 59778998 0.728897875989153 8 82562651 51003069 0.617749895167489 16 71474930 47575960 0.665631431888076 32 71391211 42902076 0.600943385033768 64 68225932 41337835 0.605896230190011 128 71053578 42467240 0.597679120395598 256 85094126 41630463 0.489228398679364 512 68531628 44049763 0.6427654542221 1024 79338892 43370866 0.546653285755491 The measurement shows this patch reduces the process_madvise() latency, proportional to the batching size, from about 25% with the batch size 2 to about 55% with the batch size 1,024. The trend is somewhat we can expect. Interestingly, this patch has also optimize madvise() and single batch size process_madvise(), though. I ran this test multiple times, but the results are consistent. I'm still investigating if there are something I'm missing. But I believe the investigation may not necessarily be a blocker of this RFC, so just posting this. I will add updates of the madvise() and single batch size process_madvise() investigation later. [1] https://github.com/sjp38/eval_proc_madvise Signed-off-by: SeongJae Park --- include/linux/mm.h | 3 ++- io_uring/advise.c | 2 +- mm/damon/vaddr.c | 2 +- mm/madvise.c | 54 +++++++++++++++++++++++++++++++++++----------- 4 files changed, 45 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 612b513ebfbd..e3ca5967ebd4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3459,7 +3459,8 @@ int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, unsigned long end, struct list_head *uf, bool unlock); extern int do_munmap(struct mm_struct *, unsigned long, size_t, struct list_head *uf); -extern int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior); +extern int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, + int behavior, bool lock_held); #ifdef CONFIG_MMU extern int __mm_populate(unsigned long addr, unsigned long len, diff --git a/io_uring/advise.c b/io_uring/advise.c index cb7b881665e5..010b55d5a26e 100644 --- a/io_uring/advise.c +++ b/io_uring/advise.c @@ -56,7 +56,7 @@ int io_madvise(struct io_kiocb *req, unsigned int issue_flags) WARN_ON_ONCE(issue_flags & IO_URING_F_NONBLOCK); - ret = do_madvise(current->mm, ma->addr, ma->len, ma->advice); + ret = do_madvise(current->mm, ma->addr, ma->len, ma->advice, false); io_req_set_res(req, ret, 0); return IOU_OK; #else diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index a6174f725bd7..30b5a251d73e 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -646,7 +646,7 @@ static unsigned long damos_madvise(struct damon_target *target, if (!mm) return 0; - applied = do_madvise(mm, start, len, behavior) ? 0 : len; + applied = do_madvise(mm, start, len, behavior, false) ? 0 : len; mmput(mm); return applied; diff --git a/mm/madvise.c b/mm/madvise.c index 49f3a75046f6..c107376db9d5 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1637,7 +1637,8 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, * -EAGAIN - a kernel resource was temporarily unavailable. * -EPERM - memory is sealed. */ -int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior) +int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, + int behavior, bool lock_held) { unsigned long end; int error; @@ -1668,12 +1669,14 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh return madvise_inject_error(behavior, start, start + len_in); #endif - write = madvise_need_mmap_write(behavior); - if (write) { - if (mmap_write_lock_killable(mm)) - return -EINTR; - } else { - mmap_read_lock(mm); + if (!lock_held) { + write = madvise_need_mmap_write(behavior); + if (write) { + if (mmap_write_lock_killable(mm)) + return -EINTR; + } else { + mmap_read_lock(mm); + } } start = untagged_addr_remote(mm, start); @@ -1692,17 +1695,19 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh } blk_finish_plug(&plug); - if (write) - mmap_write_unlock(mm); - else - mmap_read_unlock(mm); + if (!lock_held) { + if (write) + mmap_write_unlock(mm); + else + mmap_read_unlock(mm); + } return error; } SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior) { - return do_madvise(current->mm, start, len_in, behavior); + return do_madvise(current->mm, start, len_in, behavior, false); } /* Perform an madvise operation over a vector of addresses and lengths. */ @@ -1711,12 +1716,28 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, { ssize_t ret = 0; size_t total_len; + bool hold_lock = true; + int write; total_len = iov_iter_count(iter); +#ifdef CONFIG_MEMORY_FAILURE + if (behavior == MADV_HWPOISON || behavior == MADV_SOFT_OFFLINE) + hold_lock = false; +#endif + if (hold_lock) { + write = madvise_need_mmap_write(behavior); + if (write) { + if (mmap_write_lock_killable(mm)) + return -EINTR; + } else { + mmap_read_lock(mm); + } + } + while (iov_iter_count(iter)) { ret = do_madvise(mm, (unsigned long)iter_iov_addr(iter), - iter_iov_len(iter), behavior); + iter_iov_len(iter), behavior, hold_lock); /* * An madvise operation is attempting to restart the syscall, * but we cannot proceed as it would not be correct to repeat @@ -1739,6 +1760,13 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, iov_iter_advance(iter, iter_iov_len(iter)); } + if (hold_lock) { + if (write) + mmap_write_unlock(mm); + else + mmap_read_unlock(mm); + } + ret = (total_len - iov_iter_count(iter)) ? : ret; return ret;