From patchwork Tue Mar 19 09:27:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13596499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABD4DC54E60 for ; Tue, 19 Mar 2024 09:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0891A6B0087; Tue, 19 Mar 2024 05:41:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 011C56B0088; Tue, 19 Mar 2024 05:41:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCD626B0089; Tue, 19 Mar 2024 05:41:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C7EF16B0087 for ; Tue, 19 Mar 2024 05:41:45 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8EE8D1C11A8 for ; Tue, 19 Mar 2024 09:41:45 +0000 (UTC) X-FDA: 81913296570.16.9717EB9 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf27.hostedemail.com (Postfix) with ESMTP id C208E40009 for ; Tue, 19 Mar 2024 09:41:43 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ABo4DgVm; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710841303; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZwvTELM6ByipvSR/wfDTLW1X7/KlLM1aOXjOWDL7DtM=; b=y8d10pFZIt5bZk2vbruc7g/JO/3qK9HyLAmZgM/G/gLYAuSlQJuqrdE+j6EK81azcB92uU 3KkYDfytJhBIrf/Gi6CBKSqeK8BsuhkjlK7tex882AX1XSjcg2smM3ItAHcspmHNv1yFxO Do9F9QdIJhYV3wzVcIIV7AzVHmaZzGQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710841303; a=rsa-sha256; cv=none; b=KgOsaZLJHUvK3lIJ2jHPJHecNYR84YYkb+EUq9GVdNVh/e4DmhIz5qUkaUJtQzBYBKeV6n lPRHLjdAOd4GKhJ4rpWf+iFbtX9TvpfNQBjj+jJGbvgh/KlvlL+VqXqlla38h2bTVvRlYS LSpaLzoM160q0s9gkNmTI0s2faqqQXA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ABo4DgVm; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3c39bc142edso85988b6e.0 for ; Tue, 19 Mar 2024 02:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841302; x=1711446102; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZwvTELM6ByipvSR/wfDTLW1X7/KlLM1aOXjOWDL7DtM=; b=ABo4DgVmZkl0TlYTbYvkCf/F3kApWxplqoYQl07MhSOl1iIHEqgPLo0RJVUAHTarvu RN0UhYsM2TTf2lL0cMBDSKLN+9UnIqWT8/xk2SLCYZBO+9rLPlCV5ouiGNGwtT2HvlKg sFixxgeRiSso8WkJNbY1Z2yhrwxJF5maWc4lihLhJgGiYatIlc5BDDI6cIEw7/vpsIaX uzSQLEIqmrwD5yM/kiVWCyMSxFtSFkf/aNJji4mI6ODsRI5zIAAa01hpHnJia6oDZPHm A0OUEfNv4bzCAsaZY5ag2QIaurzaFBl+qof4+Pt+7eAg5Q62R0UZNj5/vBOPc9v1TF1z MWnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841302; x=1711446102; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZwvTELM6ByipvSR/wfDTLW1X7/KlLM1aOXjOWDL7DtM=; b=Enii0lLdZBdPYqJUOm1Pp1JK6+GyeW7i4v2EMFJ15qag3G64c4Wd0UzcSt1c7eWlhH QZl0koibcuUJhx90MqH67wwVBzauek5jcDfVTOqQPj4km0I95TqhKvNgRarxMuuo7VKf quxN7IhHWd+/8a5qFOU6xxMSXCltQ4F2aMuUQmdQ/XGFnfS1YmSEVkVSYCou8u0dFMFU 7nO62KTZshV8Vg9RGLt7HVDznsnIbhP/o1X6xQ3XAyvV64HbY3Fhd4XZkiQQ9tc27IlJ uyabLds7gQxknLzE7eFhKQNf77uYnWuui7bmPF5Tk0NU7w8eWVr4GTXyMCSPnRuCVxF0 bcZg== X-Gm-Message-State: AOJu0YxYDTUzUQ9FwLx0mTLxl6aGOyMRpm9tklHo34rfIXp8qPUtfhvL m6Q/cJJ7r0aal52PSjRwnBZtv0BT+xmC0S+3dNxsxy5Ju0IuftcjvGryafVa32osvLyu X-Google-Smtp-Source: AGHT+IGwOyvn+IjcJV9WoC0uqc9TVdMrZljGfip5NbPFBLB5b5fIZHNI4sP+MYpz3oKDJCbVHW9TSQ== X-Received: by 2002:a05:6808:1524:b0:3c3:88f3:96f2 with SMTP id u36-20020a056808152400b003c388f396f2mr6552697oiw.0.1710841302069; Tue, 19 Mar 2024 02:41:42 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:41 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 19 Mar 2024 17:27:30 +0800 Message-ID: <20240319092733.4501-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: 1aaay8oqmjyptqzc6t6susem1g5wfc14 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C208E40009 X-Rspam-User: X-HE-Tag: 1710841303-718610 X-HE-Meta: U2FsdGVkX18au76dAG3Oy1/hDCLYKbRcdQQSC2BflvDtGgoTgnVRbidkBfbI4dtPCOjklbEciocXa1MJc8sLnlCY9VEDpRQRdTZGanO+xlUaraMuadKqzXdLkpYEv+BBfr9AdmNKQfT4Llqd/gwfOXLvjrFbbF5sKaF2hmGPP52ma5KqJYuhN7Ecn3EvKdvor/soLAT2T7EAbRFVHkHfvxKeGdfGwcsC+9pWoVZr9ehY7Rz3YT2kptHLQEhe5m4MvTcg4/W0qT8Int3cxSC9vBhH9ekt+j1PrJTcWtCu1QM7XLagfCNIyS3YkOxI0HAVrT6MlUHOkocHI9UFx/7CQbsGaxONaSZO5DUVRsm/OLj5YA4/QxprEFuslTvdMtHCShdnrQflIEAWdHvF1RXlQ03bAO4nH6KwVzvRDCkD1GKRmtGkobh0dk56d3Jxq2NoI8ZORImp9iaJUgCLinaN7507MeLjtUEQvmjwxOZimOZMfriTqDGjZiee8nOOAXJf35Mbl+ZoMLnhfx2pyqB2j0Mg6I1Nh08OXOzrXji5iYAZ/1ftkS7/9yB6nMiFe+zX6N0K42gS4MlSxNzr6hF5i8r3fxFIlrIqyXyZiPANsUZNBpugwCPPLSaCne6RT1rGQVy3kH+h+q39awjbXUDDWDsW1poDGtXnCDTdPJGpIqwqhGZF165nVmFqZJzSl8s1Rnr3tSZYUocM2obqccOOhiM9IlrmTcgGXdSvUvL2UVPqgAsaIBGI5g4O0L8bo4+CY04YA0AxGsip95QWS9biiWGvcInWVHhqCIfBBMfwojcQF0Vwr+ra8DT1J0hUj3TA1rhyBcnxFFVzi8LxqaHZG019Bi438VRz4k/44r/whIbrixN+ouyUMtXT9qPNNUTg5/Q29eBxIoe5D0EYXWQrRGs8uucdHJ077I59kO97V4j3SGS9xe7Gl+QA9Ns5Pll9m4ZPu0Zt4FVtEWMHlK2 YUp74CPv WTFS4gJXhltLwMSucHFtUaBuGx5WIggzhkCZY879+rSFTefvvaqPTNB5V19Nh9xrs0CSj2Zlv4QUnnPADqOziEwOdyVJiqtS/7CRsm9dRCj7JEzcTfI+EZWLp1Pu9iOi393gmtQJeinf52gUh/l95tdg3vxcEe0uC2JQKQWKPAhkrzZsdIIz27EdH2tSTyAxeI9CM5JcDp052jq+Q4BctPWM+9tCS2t9heQB6diwFzOyfkiruuKDyCUI8p6ifpd07glsNpOs67dGyoHjdzpT6r6iT2SFg7KrgJsjpE+j8OO0KX0VNf8Kt/vFmDhgmO9uWuzhAg1oCSaztl2fYRveHM+xJtqyXytw01i8hyglXWHmXOa7NgTOLzth7DLBRhcmcASY5G3wHT5nDEpblvXt/PLZHWLd4xr4a23rcP4Wv3Lx6Nte9GkmE2XBrEZvP3Q482M5oihNp2iN0X9oKIJzIHoOk7ZSyJRV5exS85+zeftHsb12X8Xg0WTPrZJmJwZbPEQ4h5djUDdeoLmkqaDZe6LEGuLrZDWAyQi2jLVN/zTvz01BUWhR3EV9y2CzvtoxSFpLVdfZamfPZ6vcF5sonfEYT6OZGG0xFMshMPcTha8iZHIR96m6AI/lBdw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.004087, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7437b2bd75c1..f07ea0b97698 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, unsigned int order = xa_get_order(xas.xa, xas.xa_index); void *entry, *old = NULL; - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; From patchwork Tue Mar 19 09:27:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13596501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F86C54E5D for ; Tue, 19 Mar 2024 09:41:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 718A36B008C; Tue, 19 Mar 2024 05:41:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67B516B0092; Tue, 19 Mar 2024 05:41:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 432956B0093; Tue, 19 Mar 2024 05:41:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1E5366B008C for ; Tue, 19 Mar 2024 05:41:48 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B87D31C0EEC for ; Tue, 19 Mar 2024 09:41:47 +0000 (UTC) X-FDA: 81913296654.01.49F26FC Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) by imf26.hostedemail.com (Postfix) with ESMTP id E596A140007 for ; Tue, 19 Mar 2024 09:41:45 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=D0lwQSmS; spf=pass (imf26.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710841305; a=rsa-sha256; cv=none; b=4dgaHaGjQa8tcL4LJhVEKDiGvI2b0+hUP7SKylf8awn5epZcEmjZsbJye7+qK4rT7cZgIT W9ImFLRVTxJh4MHDkQ8J1o83rLQ83RVG47QSBJWoJu5Si2nN0lmpMQYsmIt51u5E/Cdpgh pOTtClLXtpTfpbFXASkEdGmLbPAWJt8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=D0lwQSmS; spf=pass (imf26.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710841305; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J34Qf2qKoII5Mgpe9QZtGvKa+BRn934GAZXO0p+CcPY=; b=7QDzKhgpmv2I1z6gxwYPyKJLXUjoDzJqenu2tWEFdrz7IygPUzYF5kLggj0DnkQ1WZTnvx 1hBxgS1Wq0P172E6K5e8gGml2xpIJ1/qbi0z2SwjrNrhlrOFymhxWj+KyU/vip8PfnzYQw qTNu7S7QMSrIUOr7xQnzDrMP0fHpfLM= Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3c387e02f23so976445b6e.3 for ; Tue, 19 Mar 2024 02:41:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841304; x=1711446104; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=J34Qf2qKoII5Mgpe9QZtGvKa+BRn934GAZXO0p+CcPY=; b=D0lwQSmSQZ1DFuVvJ044XFc0Mln3Kr++fd7dSB+hDlKxR1snGD1iH2PFcT8EYmwkxY Dv4pPlta2nmm8g3cFNHlgbP+CSb3/wip6FaoWT8lrzglW0jTV6o9FaP9G/AUR5LHgW0b NuKWl6vknNpBkZQvDf4ruXi6+M45gwi1RJdurp2y3hE9DJSDdY4PGx/fngDPyNPyqGkH EbYc5vxU2WTdEDUMNhS4f3ieG8iOuJdJ4mmg/EdkuXwNKFtH2KHGhs7BCvhChgfX9S12 nL92Wpq7lytEx5Wl8osyy1y9aGUTFIvBIQHxt7lDqLe8URCcIxnOyI8qm7g2UaI5Diz3 KATg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841304; x=1711446104; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=J34Qf2qKoII5Mgpe9QZtGvKa+BRn934GAZXO0p+CcPY=; b=QCFCS6SsFevZIN05pSmiDdJ4OsRvdualzQ5G3t9xTsUuqnlyg6HJwyo1ob7Z8sXMSb wYi5lgrQ28tzoQsE9FTcdP59O3XYRXs3nxIEHqIaQfdTbJBFjN5FQir98hL2WmZvembg SYd6Nh427gbC2j48FNmCM3PZHNaXeWfxjQ70iSbUWat0W9pgeriDSY7OJhvCDEsdc9Bf xZDY6U/dKkb0potcCG53x093dM/gCck2BO+STxhVSvW1x2o3S8Zn3uqnrUKnToA/UXQJ K0nKfnc4k7jM5ccxpGM14Yo8jDRBLNz/YHsf+gHvsBff8xXBsUJCRM38npN+7y4pOjoy Zl6A== X-Gm-Message-State: AOJu0Yw/j9tXIGMYS1qAITEE8nGBcxAmTJqt+eVtFXfrbFSlxwJ9WjGO BqM4MkMcQNF9xkBr39FAkMTdc6+3upwNf053wBI7jjZfkYHFmFmlWqVOCZpaiPTYUU6b X-Google-Smtp-Source: AGHT+IEJHFnNbXQdn3Vy4EOryeIqwhnOxZinPVrn7nSGg3zBBB2/OolNOe72ReevQN6mJSPYvWaGpA== X-Received: by 2002:a05:6808:81:b0:3c2:616a:22 with SMTP id s1-20020a056808008100b003c2616a0022mr12406922oic.55.1710841304361; Tue, 19 Mar 2024 02:41:44 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:43 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 19 Mar 2024 17:27:31 +0800 Message-ID: <20240319092733.4501-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E596A140007 X-Stat-Signature: 3z14z4po44hjyskkfet4hntoyf9wnxbn X-Rspam-User: X-HE-Tag: 1710841305-855167 X-HE-Meta: U2FsdGVkX187t53BhA0X9HjIjviaPXEgKyg1IWoPVAsIzniIl0ATuSYOR8drUqXa0YyYkLqmjKZHwEtiBEAKkSoyfdEDlHHASa3wHdDWdDrxjtVDGn9iJxHOCxTnUk426q+h5ToSTJHrytVVHRXIm3/5BT40mOcBysE2UgXKK4/z+63tWEcmYIKVtinxcjLNQxtcxDXXvqqLcwwi9kjW4NKwI3SRjRfWamTfFa9CtBO/3Cu02hEbcK3RtUXUiUAe4oglyK7RlNEauPiWMeDQi7nMHiiiSiMYWpPpDCWQd4AnIx+02rWb3Mg49BU0r8wGeUrSgDu/mYSpNFMCxSxh0eSPIlM3vmz3LffouzBE/9z8BYPzXERqFVNc7BnA2WoUIFPxc/s77IvLTa7+MR2iN9cpNwlVwcgjLmqgwUrKnT3rzVIzPU5hisHcoH2dUlMFRcby5zWU8WZgCyPS7X/qecKJqxEYLJ0m82pclcFfhgMOJbNo2I/+8sFxIogtnGPwjYdCQwfcW7Gwxy14aKrvJU3PaX16Xs08zxKjldPMOsWnDge/vVT+djA+3XnG4DwzRZIWFYWBYccYTYy4y92M4W/fYr5zbgA6OGC7Lr0eBLTN/+vMWLCck4G/mbb8IzScH6D36L+niLbUSrKzIvvi99Ax1S79TbJA2srFW7G2IiEMUqmFY4h3f8TXVOxNmzkIKCRLLoP5ihsqa+TUpNN/8Di4JaL0Vp8TFFaW5iAa81UkdSFYMt0kEIonnHi7QZHhSVfBDmv9tcdF+4N+4uCgyXgRv2lUF7JxUsu24XeQJxGKXiDT2+/Kt9CaJUnMHIQRwz50By4hCBVSSip/yKncdQmmXWP7qAM69uT33LAOu5nCXy0VhN+zEDZWw/vzlyzmlFrZOmLmh+iSeDSRsR6rCjDy5f1BMtwOtaBw3+qKSLuMWnfffFm5BxMD8Nq2NWzEeNMAgjcHFFy2Z6Y6hYD sTjQRP+S V3O15GogWVtR1f6I4xClbN7yHpABwVlwvsvvZRirbLL0WKPm+/9qA5/cYPQz4pVfilPgG5I7Gi/vvFpqsr037fYMQMmg//kFrU3RvXD011KjJc84zbIHNg1kC+pI6C2Qr2W2Vr4++kQ/MHuybVkpbF+DY8hyq+gbtXOYH13CmWG9RvN8WwxtAuErQ/17jA030vrHzlwIYAAV+TthrMSqq4eT1wVcJBvZMmgUd4eCH9QA1m1AuR0Y96C9kpiUqxiO/5WnnPdzQJuj1L2ibeUAHKEG7VLn6LiiZJHLWwa4F8o0TMdei0g1ZlmKuiWA+U0iAdNxahz28UWwrR4nYpG0R5zafXWHc8LI1zoPxI+5Omr8FDskZmT2UwVDFlWThrIrxI4sEMK7+eNCO9ckehDKSSog9ZzlYE/7HUE7BENadqCWxKhKuFav6cO1cB6SvR/chfEu7XdWp2jZxG625kIrkHAnPSdEdE9HNg4l7vMZcnLw363dP1fPU8XllotRBo/WCUPQ1xWFfTg2enAa97CgkTmOEf7cpYmJsu0yNodXSAtPwviy9z36py3QO9pDQbHFDzlJepbtts0EYmSPCv2CPnxaCUPB63qf9nYf7Dqcg1fYaF+uWkz/yfup+vA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f07ea0b97698..6bbec8783793 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge = folio_test_hugetlb(folio); - bool charged = false; - long nr = 1; + long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); - if (!huge) { - int error = mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged = true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr = folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping = NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, void *shadow = NULL; int ret; + ret = mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret = __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like From patchwork Tue Mar 19 09:27:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13596502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C296C54E60 for ; Tue, 19 Mar 2024 09:41:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A92B6B0092; Tue, 19 Mar 2024 05:41:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 439316B0093; Tue, 19 Mar 2024 05:41:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E7C96B0095; Tue, 19 Mar 2024 05:41:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 02BA46B0092 for ; Tue, 19 Mar 2024 05:41:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B26AD140918 for ; Tue, 19 Mar 2024 09:41:49 +0000 (UTC) X-FDA: 81913296738.19.4C329E7 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) by imf28.hostedemail.com (Postfix) with ESMTP id 18BD9C0015 for ; Tue, 19 Mar 2024 09:41:47 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iw+8pbIq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710841308; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XqzN+wWZGV0/ntTi34b/frY6opMs0aB0oKSPnzSGTOk=; b=NcPxT8uNIOM+A0emtX+NqtX5JsYoIMNK2GC9x8wRrLOApwGlRk1jKuBLWmrXGQDsaCGZKr dQ2e1Q3S1ycjI9ub79BPeH0uW6rWwVt253y3FrZ1moUyzUXNzcez6nzHJ8e5ZlP54DLkPT VBYYzN1o4kcrrWuMBQnfzyyN9S6IcLs= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iw+8pbIq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710841308; a=rsa-sha256; cv=none; b=rtQ0Uj/jvLmRtpbwxDKcD4EoRaY9qf4DFqkn7M0RF2iAxFDT21ISjRLBs/iftZZ+4/Tqeq 8Xo1EbWyQWeHCROo/iJZ8I6l8Rmk7sIRzhCdLCmRnSSnuL/UjXzAdHwxOSuKlozspOVWTu XAdC4mFkTncGPkc7qKn6NnLDLDOnQXw= Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3bd72353d9fso3169092b6e.3 for ; Tue, 19 Mar 2024 02:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841306; x=1711446106; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=XqzN+wWZGV0/ntTi34b/frY6opMs0aB0oKSPnzSGTOk=; b=iw+8pbIqNbWs84QeiUoujuro6YgAmFNtjNvc0OMH0lQm1g0LJXaPEtt9a1Bn//J7yJ oZ2tMZHmxCKgALwS6EWBI/fIqrmunNJc2jkTflV7bOWytCrvcvS3Wu2aVCDIt7R5tfUj DzLFiSPfl1rAhZWnV18/FEVF/GBoUfgjKQXBRxz6ydLfgviANnLWpveNQn+/hnj7vwy4 I1DeFZlGFoC4dGL4L11PpDba5rFQ8SFN3/+jXZp4fHorhZ5z4lyq/QxtlGr2FQxc6Dyk 7GhLPocuLyFqcPlATodxhsNSKc3V83nQ3VfqfDQsTx5Yv+KoSQzj2+qWn95iVLqVJBfC axqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841306; x=1711446106; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XqzN+wWZGV0/ntTi34b/frY6opMs0aB0oKSPnzSGTOk=; b=K99/FCt/OEs3qTrOIlF45gvfJcUZ2Ia8Fqza4hPPhxssQW/He744wYSsnGVI7n3oLF SRaYMaYiOfPhojxn/ytBkX/VbfuitqzEzToNW16FabIn6KqiNO7+i1lUmXPBC6GNP3nn Rvsd79ZJ4uuekL739Buv1Dber2NMh92ijucJmLLpiPKjaInWM6VhySRnNTktKnxybfD1 WXQbZH+rnmDCWBoJsI/gn3tevWjbJYodmU79m78UqSPzPsknlWsgI3CxXXzzoqbTdenP g063wzKnGSLLTwcOwPrXr8FwyNoX+1G0ltchb6pPSVl+pLUtg+UQidxC+z6VwCyj2xs3 wMtw== X-Gm-Message-State: AOJu0Yyyqv8fj06Wy2eHap8RA0N0DBjFlsUEmFjlwUeksBonx1fJuPw1 XzRywXvSJj2P312TM+pynsfGphOwaudJMCtjtGVFUMPBBmJZrqeBXMGsOIqyVDipWjub X-Google-Smtp-Source: AGHT+IFpWzLUcgwPhIA7GrafY8so+K48aVsYG/cukRTuKwm2t4oZkaJoNlfObveyt98ye1pVmlIUsQ== X-Received: by 2002:a05:6808:180b:b0:3c3:82c3:ddcc with SMTP id bh11-20020a056808180b00b003c382c3ddccmr2678100oib.31.1710841306463; Tue, 19 Mar 2024 02:41:46 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:46 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 19 Mar 2024 17:27:32 +0800 Message-ID: <20240319092733.4501-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 3oan8zx9wcop1zhypnpbfss3ysoncwuc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 18BD9C0015 X-HE-Tag: 1710841307-733918 X-HE-Meta: U2FsdGVkX1/66N9wDHlyUbJTS0E7ifbZbGdLUoBAMm1cdhlWiTCDSeIdY2Xg7Lb2JUEE+7EYJqCeVVCd2aMsD9zJNW2VImUWMMW81dDfOe5hRnOkhvrY25jfYZIZWs3jFQb0gus1q2gMq+fhfV+ZIpEXFtrTTenkkjKbqMcRPyJRO4/imDbcLfO1/HB62M7BGr6amxwAP93cXhlPsY9mcVywe5kgjlrBvoKKGfUPjeAsFtkE4zXnhiBlxKxfTmt0uoToPO+qfdJjGTNPIZkfEfaYEJymF4TsSZ9m2UKhwEYxTzo9H8krSE/dexLsOyHOFUE3UkTX6P3YXsU3srFGTgYg0NyygMBlJ8a73xdo9QlSYfovIRnmfwHE0X/BNgUJ3HFC4oXCtrOvlYFmRt4iXzIh4FHgGW9pAx4X0wS5KYgfEUREjlYpKWLChi3JDopGZeuK7twZgZJxV9ITDQhNV1iNDwuaeuDqOhLQgXyP43OuNrhOq5YB5bxvA1KX2VHGqdtIagftGgoWBHkqGDUMQ7qz6bR7446el3hoO5zzXg9cZN6pyuXMUq43WUvLAhWDtPPVGfB+YT2PbCLE4VAWJ9+QS2rkZiQGHBbGjSEzries+S4XLNOdYspCW/z7uXoUUkswn4g8s5+5AzB4b/F3kXKLJiA3kwPXoGpRlPa3BRvGKo1Edls4qs1zYo2IJ03PJbXr85sieANknfDMHz9MZsKMBZqr+qVjd7BEraYSCL9MVoiErVlWbV0iTMlbnVZ2QNyp8rS2NuJ+f3EfK+7GpUcICj9MsS8HnhyLuONPbcMy/Xd1UomjIsUs7Tb2wzBeykJXlIHgfehotQ0/ehG+O7uyin1rJngdUFG3WFC2ufXzaahLCNSXoSjTY9g5lip/juth/whiFuD1T1IAzMsSPV8gZ07JaAUuCp6gqHc4VGXWLO2Av7o1oj7aPPry4x0cA1BQE1pNbvsj7Vnt42H 1y2B7CMK btyYZ7nU2cDDTqS0ILjUhWR1e6y749oJR8kuDnldILBGegpI1LhaUiNJfY7lpmXo3q9o4K0xAsWu4LMGoEuz6GhnlyRkIhoemWqK+s9T5uNx5Hnv27fAy/7IBpjMef6A1cIBKV5xHWn/+WYZOuWfiH86uD8ZCOuTc8MrVJeKxjqmWCeJ6DtioT22qGyG/l5HmKzHGlCmG2SqiKfqVhu2lwIxbumQDqAsnczucyryOyovCBsCrhUqcp5TKiipNNGRjDsYi5q1Atp5rLdD25EaSPK1hwoyx2HwKfqYb+yb0Gcd1I2rwAF8rWEn/hQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- include/linux/xarray.h | 6 ++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 2 files changed, 37 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, gfp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, unsigned long index) return 0; } +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..88fbea481e1b 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned long first, EXPORT_SYMBOL(xa_store_range); /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order = 0; - rcu_read_lock(); - entry = xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; for (;;) { - unsigned int slot = xas.xa_offset + (1 << order); + unsigned int slot = xas->xa_offset + (1 << order); if (slot >= XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } - order += xas.xa_node->shift; -unlock: + order += xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order = 0; + void *entry; + + rcu_read_lock(); + entry = xas_load(&xas); + if (entry) + order = xas_get_order(&xas); rcu_read_unlock(); return order; From patchwork Tue Mar 19 09:27:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13596503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFD54C54E5D for ; Tue, 19 Mar 2024 09:41:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F3916B0095; Tue, 19 Mar 2024 05:41:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 655F26B0096; Tue, 19 Mar 2024 05:41:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4584E6B0098; Tue, 19 Mar 2024 05:41:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2D4CD6B0095 for ; Tue, 19 Mar 2024 05:41:52 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D55721A10CB for ; Tue, 19 Mar 2024 09:41:51 +0000 (UTC) X-FDA: 81913296822.05.BB2BFC0 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf06.hostedemail.com (Postfix) with ESMTP id 15532180008 for ; Tue, 19 Mar 2024 09:41:49 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ThcG16pQ; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710841310; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+IPrpGzoHpDGCt5ZLHs8DWvQZcRMV24wjwQMoVjv/9U=; b=TJJaYyOf0pKjrEOzPWSTf2bpkwBbz4jxdTyhMFDAS0RkWYlQRvSiPfTkUDQyn4fPA5rBU0 xiVIcYWrCtsWONsVTOWweXSIkvPAXoo01XhjsJp6kls2s6Z61g87yCWpTWvYdL9WeSZzNS 7e9yTd+GgIO4A+fqm+a8aE/pXGLEQHU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710841310; a=rsa-sha256; cv=none; b=Wf3z73BiPjF89L2H3luwdYNhZ3Pb213BkpxCgbM5UhLLDeBdRlau1WJlJfZhysRK5v92qL q9qwInvy3eunuvJJbjE60IljxRaZpm5mswWXX293xcp/dKApNEz2kZ694b4xllmL5zuyFb /kaSodMEgZjldK9gyfvD8LjH23sjPog= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ThcG16pQ; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6e6bee809b8so4922179b3a.1 for ; Tue, 19 Mar 2024 02:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841308; x=1711446108; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+IPrpGzoHpDGCt5ZLHs8DWvQZcRMV24wjwQMoVjv/9U=; b=ThcG16pQPY1PSjxNLjMpPCyUnp17bee8cDOqp1YT2vEy4f8BRiC4Gcf1XFcJ8ZDAVt g5vySVIXn51ZCaB82YWWzxVOVFWoyUlv3edxNLhB3kjzHLG53FyYnUeI7Uo5Q/1NrG7Y HUxsjgetBUKoY89a49t/apEOm0M3Rqka8ewU8lVZj215PQx/AxZS7sgmkAF/wugyPD6E BKCDqzIBzwJRz0vKSEUVggmwLx5MffPFx9LKmSmcV7byWz6mSBfTodnSons+vaZZ+LtR LSLKptdRH0V1vRwdGubcpr3Ak/crQRbblUl1if3xL4cs9f0CZrwIOGyh8t6ztOM20Oto 9aRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841308; x=1711446108; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+IPrpGzoHpDGCt5ZLHs8DWvQZcRMV24wjwQMoVjv/9U=; b=XEOyg1w6mdtxQvGltY7ya+p0A7Nk3UsTAOUSaev9CyILwfLesJXiw6pykYa7Vn1LF0 fUvq1KrcvNqbSIwv41xrC17jbI7jd55bQ3aJmoLY1gQQfKMERBdFRFaYiOjdqMBCFngH iKe0Drw3NMjrfzdppV2OHZr2yG4Ci+9c8F/+e9cg8ZV2th2yi3lW2Y7HwyfGH00rvYx1 JLgfPhZdWupyxxpNMeYrJbruYu+K2/kQTj62ySIOWiMNjha7dyBy253Ffd4/IgjAKjXR mozByiv0ahn+M4KoJVqW92Yu4qva8e2ThiBe/Yz5b2en8ho9CnOZAcED7++JVIluBL7k hJMw== X-Gm-Message-State: AOJu0Yyt/UNKzdIlOiav9QknHhUUfszMrT2pCp074n+EPCF4KEKmxCSI vAgq0P+ihjDb9XPoj4lfMpEFgWtLGTCbF9ERHWtFxvYd1AF9pxyRGnv3B7ff7Kkksaeg X-Google-Smtp-Source: AGHT+IH9clmpXo+eDYRjLfrDjT4vum1cmIxLp7ORYVYCH84rdbd4eGTduRgvmmpEsgUtIio/VDV/8g== X-Received: by 2002:a05:6a00:9389:b0:6e6:a1ff:3661 with SMTP id ka9-20020a056a00938900b006e6a1ff3661mr19465566pfb.31.1710841308559; Tue, 19 Mar 2024 02:41:48 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:48 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 19 Mar 2024 17:27:33 +0800 Message-ID: <20240319092733.4501-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 15532180008 X-Rspam-User: X-Stat-Signature: i48fys9sxz6eb9nxzq3zm9x76ni3o6tc X-Rspamd-Server: rspam03 X-HE-Tag: 1710841309-632968 X-HE-Meta: U2FsdGVkX18gCwBnWk4wUy5kE30oUoI/HfeSaJHzDGnsrMQtOyqfLv/LgWDZ34SjHFt9LhluYo04h+jXjvwaNaLN5Irk/LIfO1ZqVbPQrRZ9ec00ZHcHPcWnH/2hlBsl51U3DWrGAzyemlPwh+EfCAGERliD5sh6kHV0bY1oqrjsr/fVP/270ibH/hzYsCKDzzTujnoMFmflCZ2udnk3sDTVydCKWYdT9iknVW7wJow7YJ7v+0bfbrKAHfSp2XhZFrg/QysBIgOa92RLllpRoFEcDZAvLahZHQs+s2AD1jHCO6kOtvya6xTVm5L/CiiqAD2FAJkCB33WwVDjVQHU4+sK6LSPg2ICq/eKZAcxXJHfKdcFpntrYHIZIh33f+Rys/4tKIMsrfplULNEwsnR0CCo0NKAOwRt/hy3C+6DMhKkyvItJAKB+X1hIRlU0kJsb6zM8oDb654Qpr1PJLnKPp84f0FH6OCYKhVXn9kHz3kSrPVdtEntgwbZF+S+EjQi7TwjnGm7R11DGUMvOFe586dTyyskovr7FUNikbR/jnmRX291gHA+VquBNGZLZw9ZVAiS1lzNUaJ8lTX1tWcKv0lbqkMJWrIejiPJL9x6MejZXcHI14t0rdHPTsAgK8A1NlswcX5yUF2MKHoNe++kgisWlb+0jvZgI14We6s7uDE9lQ1semJyRLxwz6tCj5FQMx/Ybz0S6MK88QV8WYFZhc2kqqy0gSsLDyPILcT+NK+7EZLllauDVw74k77o6+4sSrZybpRx2DeZsE6AVGTc5RRvcjtpLcfvYflL//n2kcWveg3/6bKQ5VqgEm4mCnJgsKvw1qipvTdr7U1Klwdhq1vXcW+scGd2dIYs0qtICVmNkZaPdkZm5wvPi3zdu+dfVLZErPrQly0hNVN2Qy1q4XwCXLP0B8R9rSjE7uWEcuAFmwZxv3rR2sDbAVY05QhIh+OmA7s9AsWi+tGZScL Bk4GsHe4 D6fQidsnMzQErw3AWSAtn2xSVNNBXybuthZqV8vPmLGASXUlX1pShmZL1fIbV+1jWULeZlYtgtigsV9R/unQiKxzty8OQU0/yBOTdEG30E0HF4rHvSAJYI3r6ZhKlOaSM58VZQ+ZGf8/ALHyeypMN+VarSiZ5fVefFTTL1z6+iVlWckM2w1wcH4GMOBEFBj1ywyx+z2Kc+WZ0/uqF/OJpqQ+jJwKXVVG1VyqcaAP4RCBIXTddBHfjyKkyCfhgShjs4iOY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranced conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4k pages, in an 8G cgroup, with 20G brd as block device: fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 790, max= 3665, per=100.00%, avg=2499.17, stdev=20.64, samples=8698 iops : min=202295, max=938417, avg=639785.81, stdev=5284.08, samples=8698 After (+4%): bw ( MiB/s): min= 451, max= 3868, per=100.00%, avg=2599.83, stdev=23.39, samples=8653 iops : min=115596, max=990364, avg=665556.34, stdev=5988.20, samples=8653 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine mmap -thp=1 --readonly \ --rw=randread --random_distribution=random \ --time_based --runtime=5m --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine mmap --readonly \ --rw=randread --random_distribution=random \ --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min=28071, max=62359, per=100.00%, avg=53542.44, stdev=179.77, samples=9520 iops : min= 7012, max=15586, avg=13379.39, stdev=44.94, samples=9520 bw ( MiB/s): min= 2457, max= 6193, per=100.00%, avg=3923.21, stdev=82.48, samples=144 iops : min=629220, max=1585642, avg=1004340.78, stdev=21116.07, samples=144 After (+-0.0%): bw ( KiB/s): min=30561, max=63064, per=100.00%, avg=53635.82, stdev=177.21, samples=9520 iops : min= 7636, max=15762, avg=13402.82, stdev=44.29, samples=9520 bw ( MiB/s): min= 2449, max= 6145, per=100.00%, avg=3914.68, stdev=81.15, samples=144 iops : min=627106, max=1573156, avg=1002158.11, stdev=20774.77, samples=144 The performance is better (+4%) for 4K cached read and unchanged for THP. Signed-off-by: Kairui Song --- mm/filemap.c | 127 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 76 insertions(+), 51 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6bbec8783793..c1484bcdbddb 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -848,12 +848,77 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) } EXPORT_SYMBOL_GPL(replace_page_cache_folio); +static int __split_add_folio_locked(struct xa_state *xas, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp) +{ + void *entry, *shadow, *alloced_shadow = NULL; + int order, alloced_order = 0; + + gfp &= GFP_RECLAIM_MASK; + for (;;) { + shadow = NULL; + order = 0; + + xas_for_each_conflict(xas, entry) { + if (!xa_is_value(entry)) + return -EEXIST; + shadow = entry; + } + + if (shadow) { + if (shadow == xas_reload(xas)) { + order = xas_get_order(xas); + if (order && order > folio_order(folio)) { + /* entry may have been split before we acquired lock */ + if (shadow != alloced_shadow || order != alloced_order) + goto unlock; + xas_split(xas, shadow, order); + xas_reset(xas); + } + order = 0; + } + if (shadowp) + *shadowp = shadow; + } + + xas_store(xas, folio); + /* Success, return with mapping locked */ + if (!xas_error(xas)) + return 0; +unlock: + /* + * Unlock path, if errored, return unlocked. + * If allocation needed, alloc and retry. + */ + xas_unlock_irq(xas); + if (order) { + if (unlikely(alloced_order)) + xas_destroy(xas); + xas_split_alloc(xas, shadow, order, gfp); + if (!xas_error(xas)) { + alloced_shadow = shadow; + alloced_order = order; + } + goto next; + } + /* xas_nomem result checked by xas_error below */ + xas_nomem(xas, gfp); +next: + xas_lock_irq(xas); + if (xas_error(xas)) + return xas_error(xas); + + xas_reset(xas); + } +} + noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); bool huge = folio_test_hugetlb(folio); long nr; + int ret; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); @@ -863,70 +928,30 @@ noinline int __filemap_add_folio(struct address_space *mapping, xas_set_order(&xas, index, folio_order(folio)); nr = folio_nr_pages(folio); - gfp &= GFP_RECLAIM_MASK; folio_ref_add(folio, nr); folio->mapping = mapping; folio->index = xas.xa_index; - do { - unsigned int order = xa_get_order(xas.xa, xas.xa_index); - void *entry, *old = NULL; - - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } - xas_lock_irq(&xas); - xas_for_each_conflict(&xas, entry) { - old = entry; - if (!xa_is_value(entry)) { - xas_set_err(&xas, -EEXIST); - goto unlock; - } - } - - if (old) { - if (shadowp) - *shadowp = old; - /* entry may have been split before we acquired lock */ - order = xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { - /* How to handle large swap entries? */ - BUG_ON(shmem_mapping(mapping)); - xas_split(&xas, old, order); - xas_reset(&xas); - } - } - - xas_store(&xas, folio); - if (xas_error(&xas)) - goto unlock; - + xas_lock_irq(&xas); + ret = __split_add_folio_locked(&xas, folio, index, gfp, shadowp); + if (likely(!ret)) { mapping->nrpages += nr; - - /* hugetlb pages do not participate in page cache accounting */ if (!huge) { __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); if (folio_test_pmd_mappable(folio)) __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } -unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); - - if (xas_error(&xas)) - goto error; + trace_mm_filemap_add_to_page_cache(folio); + } else { + xas_unlock_irq(&xas); + folio->mapping = NULL; + /* Leave page->index set: truncation relies upon it */ + folio_put_refs(folio, nr); + } - trace_mm_filemap_add_to_page_cache(folio); - return 0; -error: - folio->mapping = NULL; - /* Leave page->index set: truncation relies upon it */ - folio_put_refs(folio, nr); - return xas_error(&xas); + return ret; } ALLOW_ERROR_INJECTION(__filemap_add_folio, ERRNO);