From patchwork Tue Mar 19 09:27:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13596500 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A4DCC54E5D for ; Tue, 19 Mar 2024 09:41:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F4F66B0088; Tue, 19 Mar 2024 05:41:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47EC16B0089; Tue, 19 Mar 2024 05:41:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31F046B008C; Tue, 19 Mar 2024 05:41:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1A8DD6B0089 for ; Tue, 19 Mar 2024 05:41:46 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D19C6121057 for ; Tue, 19 Mar 2024 09:41:44 +0000 (UTC) X-FDA: 81913296528.20.FC4C460 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf22.hostedemail.com (Postfix) with ESMTP id 0583FC0022 for ; Tue, 19 Mar 2024 09:41:41 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gV38qlRP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710841302; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=puEj9jlVApVqcnYFYETJHMNHyp/J8GRj2biAnc/m+zM=; b=vANXmlji1l6+3DCVUg4Q5AvdMVjsKfEDmqNiQrICS1jmpOJgFlqLDSsmICQWNeRBvSEbNH UwMg7vrKibcVJUGRPVAE/AIhLttl07CAjGBJ9sWOK1kxR8TUwLk0WmHNNSkIyUKpE2Zl/d 9GO2s6S5/NaYPPz+jHJO3OpcJlJF3pc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gV38qlRP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710841302; a=rsa-sha256; cv=none; b=hKsp9dKyIJM4v2BlT13/jbUGsLbNR6zHjHiLN5hYk8bXwvq8yVRQk+aPIeEgGYz3rLPLzi AYHEWOKbeeqVzhzGntV0RXmfdg22Uknac/3CRvN6CI6bSP78hN2z0oc3jKM80bfzQ4Em5l UFXV2qiSrRic8H/URQIUGmsrjf0DsdA= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6e6afb754fcso5052113b3a.3 for ; Tue, 19 Mar 2024 02:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841300; x=1711446100; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=puEj9jlVApVqcnYFYETJHMNHyp/J8GRj2biAnc/m+zM=; b=gV38qlRPk7+4OzkCjOdkSOIiGk5vkyZ2bA9L1UqtHvpAji/ZercrjmPFZ86FFFYaav DHj/RVPMaeVRUw4M+R+dm6pD2iaEtkhV1nP1keAfp2X/2b1e0ejlnJmLSZK4yadx6gas sZ0fDY0FmylxmZ1IaaRsrzb9KyzyJ2zkiBRVbip6469YBK1gH4K9KYzkJVLagXCT/jO/ +Bx2ahYlOgEgMA19ULXcRSsgHgZGR29s9Qub+/thZXZzFQm0iIYJZ0C4XYMqI2CtHNLk 6ibdHKLtjILiG9PAIP1llirQMhOsx8UabRNBfD+8TTRSbLyW2T3HFaYNJ31TrhliuKqI zIjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841300; x=1711446100; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=puEj9jlVApVqcnYFYETJHMNHyp/J8GRj2biAnc/m+zM=; b=JUS/3d2pyALX9DIXqFJgs04gRIydINyA7f3y5LhNW344H59H0R6YalK/ZvZEWClEjY q++iGnUdapEXzCIySlGcqFOjZeqW4XiWTMlsm4vEceg17ACqanvR64xjB8YVpAKbhwAR YpP/EroFsfLUsu216SicHEfBwFlCTHfQn8WmsOjOLHExuekmDWK4yzng8cBg5iMKzwUK 8W9zoe5UJm0RAhYyxDCpUFjJKYtU+xNWQC16OpVECQfAcTuvQHLXdgSiB19ugHwDgAiD klbhaQJwDWnQSjFMudqHe8dCLEq/cswZJ+f7JOo7KMkw3xMae+HQ9tXPPXpe+4t7eU25 AvNw== X-Gm-Message-State: AOJu0YylcL264YY/k6qVNZXJfLCNmDRgcjuvsRkzEyJYBeL/FfGd52Bo Rnf7KCyckSC/P46gXSCl9gWAB6C0oEsA/2VQdF4JJBM2tM9eFZSFS2JyH2RAPV+YWRH2 X-Google-Smtp-Source: AGHT+IG4o+wBrs96/pLzi9W1461HZ72r8FyQRkQy+dgKMIyHVuf32UmjCcid72BjHEib7UXR5q9H/Q== X-Received: by 2002:a05:6a00:2301:b0:6e5:64cd:8f52 with SMTP id h1-20020a056a00230100b006e564cd8f52mr18473179pfh.0.1710841299968; Tue, 19 Mar 2024 02:41:39 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:39 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 0/4] mm/filemap: optimize folio adding and splitting Date: Tue, 19 Mar 2024 17:27:29 +0800 Message-ID: <20240319092733.4501-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0583FC0022 X-Stat-Signature: 81irm8upq7tunn7nfyyp3hkri4ahagsn X-Rspam-User: X-HE-Tag: 1710841301-167217 X-HE-Meta: U2FsdGVkX198tXug13fRXL7u2KJO1lqaC+AH5Age3eeHNGsz/vArLWAhiXnr/RIExjZXFh0UXRzFQ4E2sH8crCppXBeH0gFktC06Pzr3MsJrh9aGdn7kF+rD6v6ffpisBcD8K8r3eLxraC/XYNiD2vXf5dfXDyzy3qq0a1GwNtwgrCvrT1NAyWG/CeYfenUINCo7d4QGf7X/QUvDUc/1i/Rn5AQ5F0NYYKCdSvE2FkhMA+XSNw0dIxUJ9/Gx1Pu0Bf7rq4i0Aww5w+cBKlnsPce6DrCxyXsp2Kd9WMto2XQDOximqWo6r69Iin5YnpM5563CMoeVrF9ThL5u4/hBwobS+93LVBaFTDaSKtqlZCeSKA4Xh9/p0w3yI8acFJ6YcvtGMQUGTYu6F2KLgnc6iolQrs3msBxoTibpGmZ8lNzABKtl18+WmjyboLvRJJH0Zl/qe0PAjFD8QD22GGQBj7dpT0tWIytM0uZkb+rZwDbWVIvKMZWsViKM+in87986Dj1ijRlFfJM+Nal3VZHlJHoD9bDya2xebAd/5Nz4CMdUymqUstci2nx/S+i2JR9eyldkzLG4it38xu77sEYxsZpkjdJqZV8IYqlXcG9Vagu1EF/z+01p2FcSZr17Yc0tgcXstDAr3ziysbxA9bhigE7z8ZFDe63UkC6mL9vkFyxTrmrgjrcUCLuAZO1rQp4NBJUn7QTXS2igc2qTcp1jUiUnb0g8VbQpn8+wpRIZ1Lss2pwF5EfhAbFXum3FfZPoNsECZpgQLIvZC7ES2MZOilIVC3IlWbLpj0c8WoSHD5sqq+FWWldScfOfJOdggY0ALJovWV7Q9SIyXBR2rBWqeVvHsRTTmgfH2ETRg1ob7M/gdMlO6kew2KKKwRvAXflNb98NISbxiLf7KqR/Ky1dV8aS8l5ZaFDSPbnQvaW1mu6yI4liY7bAPp7feeGL9s8ACeXc5irhzV3k858zxEe yYqABmWL IOb4b+G2gbLocE/ln52ak+GOjC56sC3V6gzoy4UBXFm5NMgp/C7+UeIDXtVByqM5/848Cab2IZ4Doryth9P5vZirGZLe85oBGMfLGMRYBSG/fMnFltbht5FmjC3bnkFcce0bZW0WjGQ2MrM0whhKZwjRpMTgLsUO42//tu+xfFmLvcZejVTLK02e+w7wQaKCR7tBlLu6JZaJjJCsKWUUYdKrTfBjFW6//1djdIxxik/d7Y2mxSB5D3Huq5NCrTMQHoZMncJiffAK3KUJo5iRerrIiQLagGfBSrbcheJfYYIqz2Y5ZSrebM5sCvrEN/mtTONx1lABS+S02l7BhI94hNWgPWSBsTnLyVWUEm7j325olsNQ1Agl1F0G5XJU7439slZhOkmFvMiccKZCg/naUME2plXrfc3IkAFSVKoAnJywV2PY54Zcm/oCxNWYZqCSdOUdlBvad0i5J3jKN0sKVAFboL/an57AgIkwVizfXDw95DGZWBgJxFLKusyQ89meoKXTkXS9r/P9JZpJBWqvXUG/UJ8t8WvTZ8Zcys8jJJfDmNFO7xXmAeNR07NhXRh2VMdPR3A0yeyMsTFoRjIgSgKhYe5Q7JMpuRCm9BFo9RuLulzVav97+9tP84w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, at least 3 tree walks are needed for filemap folio adding a previously evicted folio. One for getting the order, one for ranged conflict check, and one for another order retrieving. If a split is needed, more walks are needed. This series is trying to merge these walks, and speed up filemap_add_folio. Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranced conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4k pages, in an 8G cgroup, with 20G brd as block device: fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 790, max= 3665, per=100.00%, avg=2499.17, stdev=20.64, samples=8698 iops : min=202295, max=938417, avg=639785.81, stdev=5284.08, samples=8698 After (+4%): bw ( MiB/s): min= 451, max= 3868, per=100.00%, avg=2599.83, stdev=23.39, samples=8653 iops : min=115596, max=990364, avg=665556.34, stdev=5988.20, samples=8653 Test result with THP (do a THP test then switch to 4K page in hope it issues a lot of splitting): fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine mmap -thp=1 --readonly \ --rw=randread --random_distribution=random \ --time_based --runtime=5m --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine mmap --readonly \ --rw=randread --random_distribution=random \ --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min=28071, max=62359, per=100.00%, avg=53542.44, stdev=179.77, samples=9520 iops : min= 7012, max=15586, avg=13379.39, stdev=44.94, samples=9520 bw ( MiB/s): min= 2457, max= 6193, per=100.00%, avg=3923.21, stdev=82.48, samples=144 iops : min=629220, max=1585642, avg=1004340.78, stdev=21116.07, samples=144 After (+-0.0%): bw ( KiB/s): min=30561, max=63064, per=100.00%, avg=53635.82, stdev=177.21, samples=9520 iops : min= 7636, max=15762, avg=13402.82, stdev=44.29, samples=9520 bw ( MiB/s): min= 2449, max= 6145, per=100.00%, avg=3914.68, stdev=81.15, samples=144 iops : min=627106, max=1573156, avg=1002158.11, stdev=20774.77, samples=144 The performance is better (+4%) for 4K cached read and unchanged for THP. Kairui Song (4): mm/filemap: return early if failed to allocate memory for split mm/filemap: clean up hugetlb exclusion code lib/xarray: introduce a new helper xas_get_order mm/filemap: optimize filemap folio adding include/linux/xarray.h | 6 ++ lib/xarray.c | 49 +++++++++----- mm/filemap.c | 145 ++++++++++++++++++++++++----------------- 3 files changed, 121 insertions(+), 79 deletions(-)