From patchwork Mon Mar 25 17:14:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13602532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D76A3CD11DD for ; Mon, 25 Mar 2024 17:14:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 725DF6B0087; Mon, 25 Mar 2024 13:14:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63ABB6B0093; Mon, 25 Mar 2024 13:14:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DB1D6B0095; Mon, 25 Mar 2024 13:14:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 40DC76B0087 for ; Mon, 25 Mar 2024 13:14:17 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0B85D80778 for ; Mon, 25 Mar 2024 17:14:17 +0000 (UTC) X-FDA: 81936209754.24.25B5179 Received: from mail-il1-f175.google.com (mail-il1-f175.google.com [209.85.166.175]) by imf14.hostedemail.com (Postfix) with ESMTP id 30E8F10001D for ; Mon, 25 Mar 2024 17:14:14 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Vyt71LZz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711386855; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rSbkdxxh6gTQxH+/m3hblAGdhy4V4YDv1roCho2fnaw=; b=GgiF0r8tNODsBmhOTEOO/S58LE0jtXaIiLuIxgBC3AmYpi1uQStMRscVeLoHGv0LdttydD eVxruFOlO2VFx63heqkHP+BKzQPszQo8E9ydAjeZDK7De9VpDlTQN3u1kmTUWGof3jiLGk WGjM/ew7Mxk7/1DM8UKza8xwh+It0BE= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Vyt71LZz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711386855; a=rsa-sha256; cv=none; b=2eXcz5G2LaTckFGes+YnefhouniLYNL2ohofzXZNW8hhSzRs+CMG0CrJGnjnBXyK6SFkiV 5r15TKtN8vBZoPTtIvRMrIkYNbU6xZiagEYgs1P9a8whuwnrqboppPr1HP8bUfE8XIINuR Jyo3BSIEycYYUcwlwWGgsuzm6u5XerY= Received: by mail-il1-f175.google.com with SMTP id e9e14a558f8ab-366ac8bdd2cso21512105ab.3 for ; Mon, 25 Mar 2024 10:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386853; x=1711991653; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=rSbkdxxh6gTQxH+/m3hblAGdhy4V4YDv1roCho2fnaw=; b=Vyt71LZz2CLVMaDWB+SX9PuVeRdFp3btY6mCwv7Apzvr8vJEZkKDSqqtL9ZjXmd9/b 0Qv+NhFh3pOMDY9NqhsgUqTmkyA3VElRxtZ6k8h50ej4HB0rBlk5Uu/Kl9Azo76H1S0N lUPDtE+eP2YuF/ECIQQKLImbFFaswJHZs207KFc724CHF+ABwTB7FJ4ivzgEnhNn0Bvt PzshCZqnoEUoNXMRWpxIwUhHVp1B1SpR4mEZWggOh7/m1CD8uGwOBcu8vTAg6hej9n3e X1m5ObBIwyedSnHRcc54a9aGyqDZAKfmgYZ0f4RjILG3AnUyKv4HD9GeW6s2FWnnU9fb vJFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386853; x=1711991653; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=rSbkdxxh6gTQxH+/m3hblAGdhy4V4YDv1roCho2fnaw=; b=K4We0SSZilLdYcEyvXAsNabEuRZY/XK2DoSzVD3YQLy9GrY6KDmWWnNVLTZcJz2CI9 VONYynLxjZHdK1W5RSmrQmSJnfKUqlDjWMWlXSf7dG+NCWgRxFNDlJMrcYQul9+I6VTI dXaIsEMWg+rX0JNzLRxK9NBFdUjB5X0keyfSi+Nm5QCMkXDYXwu5JeJtAEE70IRZ4yOu Aye19XHaI44dnKCmW+K5nc+mQXy1WQBGIW+g8eDV8bBkLAwENuuGVKwOU16m2TPTLPZg D/MCv2jLVidRIM3L+DiUP4nRylbjMUrkIoo8nQIuW60P4UH+JrNYkdBJk8D+DGhFaG8b bIpQ== X-Gm-Message-State: AOJu0YwN5kEGqU5FdqdHBuvupabr+bmIBA1Ze14kX6OBUemxm5zbU4Ej 5iOiXQGNvgdX774w6wGL6tDbCtpLUpmAztK4nF/J3fMXqBaI9mv0dqDiMTN50UpZcF5v X-Google-Smtp-Source: AGHT+IHSMojcSRK5YMTPHTkFJtuXwtvofR+6Do+4UqVrwzKlW3e5+hTuNcI2HZhXYkvzvs6sSeSj4Q== X-Received: by 2002:a92:c983:0:b0:368:9ba9:fcb5 with SMTP id y3-20020a92c983000000b003689ba9fcb5mr604940iln.4.1711386853393; Mon, 25 Mar 2024 10:14:13 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:12 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 26 Mar 2024 01:14:02 +0800 Message-ID: <20240325171405.99971-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: sykrhxmbcbscm4uyu3y9p8yda48f8t71 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 30E8F10001D X-HE-Tag: 1711386854-828976 X-HE-Meta: U2FsdGVkX19o+AnSyxkA5fVJKVEUNzfvMbdKXLAtuLpQ7vFNLKE2XiEO5uvqxvxauQL5h1Llx1mX/Zmj/WKhiOH09wBUWoSzRrcAJIlnKJ4MpVyV6kzXmGrUsNXjfcNiqt5hQLZbBj32NfIcPV58Kz6725L9aEEle7yy4ICXEkOhGEJCB8cFcZIb0+wvsdKyoRQ+cGyNtA0PO/EuQd20Winq7d7ILXyJCza49sNp6V0ee7PVahKwAuxcx+09RSHCJk3W5OmKqWTK4yy8if6XOSZcpgzB/CqmhLmmsRBeMOmPpSTUBreegmxYua3jkjH0vCQLkEBRzP+iVFI/e2n2+yjm/pSAErrbM9ibNBrLxuHeeG8UQoPj6y0lsJAp/0/JGGYrsDzeD9w15B2erUWHO10NhA2jR5yqXYSd612XdJ710XCNaiyAONchJsomR1VCbwDBzfVlz87q4VqW4NEjrTPDI3oeQe2m32UpuLDmur34/fmBuV+8yOe9eyrTs34FV98fxFXPmGnVnonAQwHopv/tKjPgWBLh5w2kmhjB0HTVhbJgqDCzPA/O8jGjef3O2d173cxLN+3BiGzvq+uVjCVFY0p88PIk42y4ZYc8z0DScTG6ZUWB/uECzMWw6fWfTc5HmhVH4xej71eRrEsXQxF9HQyvJzX6EzrxX0gJ8fH6HqnVuKzz6LMMNZEBBipM9DncZSOypyU5wCvd+kLRjzxDS9cmLlPFrckaHUAHw0HmXO5ICdyMHuUGNMG7Kzo9mAiYTYJX/dTeXSdbsMXavhd2ylYSHuuhZgrNO1hDio3vJcv2qpoTfMwEdPlbAsyEU/O3LO+iq698mE3+oPaQo801XULa2DqekSAKdPwEqVnDN1tcPR01UU3CHheJmL8h5yatgDac5DMXsv5O+KK0ySSPfGdspzhNU7DQpaBqTREx+q78CSBazXbV4EhfVscBF95UG1FZBUnEjlhsUGl xZUsPhIY mf9xtiL/+q12ldZMb+F4epskVL7t3btOownfvlIxKHRC3fU6jLAd0IVqUpQHee0pLXqQh4gC3aro0ZuDmYiZ+opJZkEVC/YULOImL0bMEtyPKOo/71SECRqZhyKLfHh7Y42ZgW2rfjPmcJPvF7ePOwpj79CF7kydMwsWrIGnkUowBPN/k3/t6lEBKc4QcXvWWX2jXQm5WK7e4L0ojLe313YHVXacvEqDtcpb5lJz1WLpQfhw9CKhXEzKmOzoNQYW45Ygdp6hrErB6bLKU0OrwvPwpiA0qze92bjnz8b4qZnh94h35PUwhLx8+mWKDB9PVB4Z7aVlM/czmsCFgIdYpUt8M6vMRA1Mesk0xumKLWOc5f0mnbWRt2JGTRrZwzXKeUJ7Wmd9UsgQxXSWVsz4WeLCiuCHr2wLeH79nOvhCTX/OQKM5y4yUcnqQq4Y+WWy50NX+w9fcSqwOAwHHOVDpWfpKjEFEpK0GoPZORpOZs5atC8iveZnyG1t2T6ZYBF3umNHKPBZTC0qJExV0tedogr3Tgh0eteriuap+Jfl8MYrCi9FH0K/2kYCmoMzlWLHjuMWO5kf+thu412XM3cLrha5JopZmXJu+h1VpAzdYCQIXo1nogulUiD8zR5i6EzFbZGfE9K+rE2GB0br1WBVhE7w9IdCjfWS67hWdUv1OA4wxcQa6rKVC4tlXDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7437b2bd75c1..f07ea0b97698 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, unsigned int order = xa_get_order(xas.xa, xas.xa_index); void *entry, *old = NULL; - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; From patchwork Mon Mar 25 17:14:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13602533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED27CC54E64 for ; Mon, 25 Mar 2024 17:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E9B36B0096; Mon, 25 Mar 2024 13:14:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 772716B0098; Mon, 25 Mar 2024 13:14:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C79F6B0099; Mon, 25 Mar 2024 13:14:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3FE916B0096 for ; Mon, 25 Mar 2024 13:14:19 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 107D51C0111 for ; Mon, 25 Mar 2024 17:14:19 +0000 (UTC) X-FDA: 81936209838.07.6E27B09 Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) by imf06.hostedemail.com (Postfix) with ESMTP id 39CBA18000F for ; Mon, 25 Mar 2024 17:14:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ftu7dtsJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711386857; a=rsa-sha256; cv=none; b=m8keoP1MKLmY9cCBfCJHCe/COVPDFqKnvj9Y12NF66DCqtXQew0MoZ8gvdxiTevZYFgmTv vDj+cy2iC0ZTOJXd7VzlBabN4b/o/WyjHjNLp1r1NUET7iMxdNOrIvS/j+h+0pnii6NF/u OVQCQ0rDfUy2cj9K3Aj5fO/TXj72Aaw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ftu7dtsJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711386857; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bN9a48YnOEXH7sPm0Ke4Oa5WKVmRd1RkDjnQJHDaF+o=; b=oiss4mr1qZoSjHoBn4vW8OGNQsIDf4AFEjQ44PpxcEtf7qePzBEUULD6SDpHaXn/nSLk7D 4ofHaNmwwc/iTh0NmX5k7Hxz4T8V68DZQTLXpFa6jkFtAIt/bmZ1x0QHMLKAv7SyMVvtdS Y8Wsz/kcEbzMYlUM3EMNMutCXYSYorA= Received: by mail-il1-f169.google.com with SMTP id e9e14a558f8ab-3684faf6286so19499945ab.1 for ; Mon, 25 Mar 2024 10:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386855; x=1711991655; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=bN9a48YnOEXH7sPm0Ke4Oa5WKVmRd1RkDjnQJHDaF+o=; b=Ftu7dtsJqZ5Q+1wguhd4Qrj4foBe9eur9+ADNhIVUYNHEf+RcENvYBWsx4YpSPpsqU IGzhX4McmOm2hNRp6S+M5n3QKDtYy26MMZQpRMXnRUJ87iGBWL6v1jGJUiIBpzRsKk4p 0gN1o06QdzcP3RUkvkKhT2E1Fjzob4exm2kFZ+x2HzIhNPTe1dyShcIGilgO36SBFV0v ntGsFE7UOPJKVJdCAAZnnVeYPUi4/evzJ2U1lJZAKfWALrULSIrQaaNwb1UIfoS6zqij iWvtq7zf8RwqoHrUhyjqHWmQm3pXbOE89T1tuP9boWamvW12o28qwFA9dDFcYLVt4aKH JFag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386855; x=1711991655; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bN9a48YnOEXH7sPm0Ke4Oa5WKVmRd1RkDjnQJHDaF+o=; b=k2X+DcFUyKxJW+yviidl162aMq96jrZyyOE4vQWGzP2lDllogixchsxA410kRNYMeI YBLzEEUxxKdCgbFkUnP5dRkXfzqtHFZb5WOzUFJaZhqfDIt8QJ6Yv/M/93PUIJrPTZXH DlXmqB3BwcI6GijrJO09hZpT4QbX/dWR0fp3v4KOa66f0WHkbSiKjbwUfY2R/Rayrvz9 HdYU9DiU/McQCnKR14hrinv8Ll2Z+F7ElRd19LM03w7WFZZRk7xNGZNRfMxrwSfWttk+ rElov4vcJiexhWFOwXnmtTZWxxitGc04/HCkXwytK6TAwCJKsqsz8XyRJFMCr8wpDsH9 9jUw== X-Gm-Message-State: AOJu0YxW06e5M9bTZMkbw61uFAZG56CUBzkph44l4zrNSF+b4e77n/HV CEjtNrulp35R0CCSq0lb+n6gZGIT8SivRc7nyUuFzf48Geqqj1FHgPhRJceTojA+ULFc X-Google-Smtp-Source: AGHT+IEV6m/IQGGduQkAd1EXsYeOBd+/asYh5Dr5GFoEqy3C5fYnTIRwAYULWlk6rk/10OQ+ts/pmA== X-Received: by 2002:a92:c883:0:b0:368:4a8f:85f5 with SMTP id w3-20020a92c883000000b003684a8f85f5mr7282293ilo.19.1711386855551; Mon, 25 Mar 2024 10:14:15 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:15 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 26 Mar 2024 01:14:03 +0800 Message-ID: <20240325171405.99971-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 39CBA18000F X-Stat-Signature: jah66ko78z1senq38tyhqdde7fhaoyj6 X-HE-Tag: 1711386857-602783 X-HE-Meta: U2FsdGVkX19vvoeY2j+zE1cIqmcStjWWz45X15arwBr2qsrI0GQHI1tSi8cfy8Uoyz6T5PSECLZwvTOq3eRuYU1d8mLsD6lniyKXc89MrsmVgd2FjP/yKMIQYMbTZ6I1HnOyy+hlpWL96u3TD50bL57k+iNz5oCY+SEZpWhUNyVFSGbz+YJzoXOww1Yg+XhvconYnUHNSKgvt+UedcJPp1q0MR3L0CinZghG1OzR17fa5v/1Rk528aA7BzFdSmQ3hlB1wH3vKcEVEHGo/YOBm/xkO/YK4gW6Oa4it6A2+zmd2KY/dejLQBs7/p1IPIUyV1uHAHFsdh8rnNl5Rl6i0FUXzXoEgdS7bqAn1Fl4guA8OgBPGPW0QerpSw/x8+dHyB0mZ+WcM+IvLtrgJ8c2LopaIS6dXRkl50TuClwLGA/HL9CbQVdXtphycFXSIMZAYzKOky7X8F+LzhffAjGtW31u4LiE/axfSkjG6yI8kKX+ojA6XZiQqo1/orBZhYJyCcCBCrdW4cknzAV6EqHtGMuMavymv+6haSYvOuZaOS29WrGOzJvGQThUO77msLbSu51W+YocAvnnrxOC5wQU53d9TLOxtyKoA/7QxPTpA+JxvEYrNuynDfQjDJd2Kc5T0w/2FsMmZBLOJpU/TDe8GNz7kXqoh+F2qEeQTHMWkquQnG0xvcYcp5GtZJuDjSJqn+DlGaOQznBgF6gnMe9DStX5w7wpQrUdCpnB/Qz1QVoIZTCmy04sLHdAC69FZe330lUOdp2uXPr+jZU4HPmyXVPuupclRnsPYUTsRabygHvod1p5Ev9aWrI++92RJr4Y2iS/ZGLCqUks1cExb+TrYowSwOG2lg9mekpRwAahSxlS0wU8fJ4zO9dUcruTIAXYYHTZ9k9HBG5lsq/7eIBX0L0DNAp91ZIo5ENAd8wzWfJCdqbWjp4zuSE8NvvTeRUNKsiQmMMjohvmxkY+yI4 JwN/ZmvD KJyR0cNNsvgnFU3SzJKkTvov46LbggNny+JkLXtdpwRgWUQrJyCCd1ghkNB2wSrUQBHQ7RkCEh7pTMuhZt5ck+C8irPZ7pEf1grfhHWjhHOPcsOPGaygNeCYFICfSSTqxmLnZUeLYVJ4EUpt/7vRh7w0ey66ImbXpyGzeZX41qgzr/JXpvo1+MLsF4Ce2Xt+if9guCZbraGRmkJS82SkyFLlOUxMJRSyqL8f7SQ7X6CJACoT5d8ggnquoQWur5Uu2CGSae3x3M/A+rFz54kcPVrOSAJn0AG7YOC8mCEc2PxcESJgG9PsWIlWIEfc7wbJXBzQ2ChXozXbPbw/jzDbHHqbdh8LFmvzCeBglU6plL6OZE35MoE0R4CcGv2rswI0ovCzRSgdlxokcPmdXqbh9fMdSmxhuhyBzKUw57TSB/B7axPceoUHjDyg9ua7RoJFapglYN3i+U/mbNwv0JsjdzdNv8R2pxG57DMPQcNMbcGnjIWATpeRaHNe1BaxSAjCGnLgk4qlWwsVz/JXsdiejYVM31KVGnejii0lYYN9tt6qxeDGUVZz34OLHsBwMaLNZVP0myS4wicW6ud8BPsbsfhw+k44MFWvhNvIhd7voH40BFU58yC7RxBBSfltLsRfK7l6ay+h8onQ0LT3CrRlPchUuXw67CgMgsT2RxMRWcRV0c+Hv7h2o6308Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000940, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f07ea0b97698..6bbec8783793 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge = folio_test_hugetlb(folio); - bool charged = false; - long nr = 1; + long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); - if (!huge) { - int error = mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged = true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr = folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping = NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, void *shadow = NULL; int ret; + ret = mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret = __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like From patchwork Mon Mar 25 17:14:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13602534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8392EC6FD1F for ; Mon, 25 Mar 2024 17:14:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 181986B0098; Mon, 25 Mar 2024 13:14:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 095A26B0099; Mon, 25 Mar 2024 13:14:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDC166B009A; Mon, 25 Mar 2024 13:14:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CE8166B0098 for ; Mon, 25 Mar 2024 13:14:21 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A2CB1A07B5 for ; Mon, 25 Mar 2024 17:14:21 +0000 (UTC) X-FDA: 81936209922.14.ACD8E3D Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by imf01.hostedemail.com (Postfix) with ESMTP id CD3354001B for ; Mon, 25 Mar 2024 17:14:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DjnEwVbg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711386859; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6sDmtk9HBtK4D63m2Orr58T2syAmM3af2Tjjpj9AWXM=; b=P5qtfPeA6/hsAvy62JyDbwbQ7b8/ezBCqqbNEX1MA8LqvH/gnvs58hHVzpqO31aNKAZBRH vpIiG0RejfgnFNl3KOz0fzxvgeXvShyosBsdRvWzeWJLuPydrXXYJzr2FK71pTlCy40NOi QyqNdTQg6xxRgm2gIkgtWzw3Gtjhy8s= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DjnEwVbg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711386859; a=rsa-sha256; cv=none; b=hDOr5P5h/P51cgrglCQ0L6atkH6sMU8ullqTsmlRKnfqNhaXlkl0RlYLn7+ZyeuTJSVq74 Fk3A83yaaNiFVRENCst0QWYlTHsAwcZ9Zw8/H/BzZPBxxA0nZQD6jach7gkX35LWBlwi1k hbRbm3gYtriTVCpT5LAxishVWy1eL40= Received: by mail-oo1-f52.google.com with SMTP id 006d021491bc7-5a5508007fcso572520eaf.2 for ; Mon, 25 Mar 2024 10:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386858; x=1711991658; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=6sDmtk9HBtK4D63m2Orr58T2syAmM3af2Tjjpj9AWXM=; b=DjnEwVbgvak9k6M/KigCvaZM/pdDc6O6WV8hS9mIxODItWzcUvWl2AQwFDVjpCjwD5 ukaxd+10DSDlqPU9OqxZVO49csDgrceB+xjea08XwCcpS6JETeuPPnf87npzDLMJb7bi oSV3r2qtqzh72CbgjC2dnOL/jGdljcop67UfM8vLRQnKTnqmEuM6Khq8Rnj+wZRqDvij dQfv89N3ByxWpRpCRTMmsZrYz/+rhB0mYmfjD3jPT1zM1dP3Y7JQM9vc+nx1+JoA5OGb WENm2XK2sOyYbj7Uv3LM+UjWk7fzBlWBgeeORCdfknp/xrr5QX7ej/H9pUCBwboCIAY0 Tx1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386858; x=1711991658; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6sDmtk9HBtK4D63m2Orr58T2syAmM3af2Tjjpj9AWXM=; b=dsvAFUYvUAgfHBTuWOCNy/YgeOPwetiInoYyJfxjhiYDRcReFOmtDlMjnKkdjKOLC4 mSSeVCQeH9r9jSd9S8gOqulTJ877pnkONCzv9BwSp6FivEniWGS+7IJpa0a6yXwJpEf/ NNVg+9R8TveOfn2Nov3ZbryY4lW8bI4u/170mN53ssevTPWyqDqMj1D6kIgAD7rlv9c9 gc7cBE4J3x1bZJiJ5Y3M6+6WuMlxAGKseSOC9mxOpvBy3Jkan7bR/IktfGo8wvnm+Dfb IHAaevcG8HUniIfj15YNihyBMXL3I3WH+/9Zk5b98ZT9tCdscfYu31yPUtjH/yVwzHJl /TTg== X-Gm-Message-State: AOJu0YzkIjNm89UXb8EztZEhHTYypgFPwYyPKrYYhRFqldka2sUTSstc sp+hnchPwq4gL3B0XLjRjroxYvyhqHn/XNuXye9hefnoROoFElTq98tEooMbdKuQfgV6 X-Google-Smtp-Source: AGHT+IH812g8lEhOIH3UUXbXCSKLPkSjpe82QohMIQkOyOH031CL/t5qi83Q7yr/40ax5hVVYS2Ijg== X-Received: by 2002:a05:6358:3996:b0:17e:9ca7:c3a2 with SMTP id b22-20020a056358399600b0017e9ca7c3a2mr9245069rwe.21.1711386857956; Mon, 25 Mar 2024 10:14:17 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:17 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 26 Mar 2024 01:14:04 +0800 Message-ID: <20240325171405.99971-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: CD3354001B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: naoxczh7tnjo9ztxdnhqcpmu1zsucrci X-HE-Tag: 1711386859-978621 X-HE-Meta: U2FsdGVkX1/Za2VtcPCbdFtpU/FE4Og2opc1PAVw7vXwnHe+vjIeEnhTy+Luc25TLNJgjbBE2qvLFl8YrfAQdgcSoORK+0QBx7+qVJsHRI0QZh8b+HvxyqRHLm6bEGb8E+L0VY69dXMMnt78x/y6K7y1CmTT1BoB1J39gsPeipMK1cEiIIg8vvzfXA4AujybB+rxUu04dARlMZp4n7BP28DZeT9wsLlhJIY8lFKqDMpFZDM2QPMvUhh0cvX4oMghRtSubZmtuTRg3QZCRCE3AntyIIuT2f5GviDZfTkjKSCIP2bUAkC8NkWrJEDDXOuDPVLVqwdIdZyxy2hB7q6AsZqb05r0wkNcgRwBm+8/lD9NuLMTkaEKDK4uiFr3lLxf1OcIAJbCqOeKWHCUeh0bdLrXiWPsNeUT/KnaC+S8vuLxWo85x822M113oEX/ukSv5mK2mgtUbE2ts1UOBkSnXNCt7qgjKfTx0FsSSgM17deZqy4+4UiDKDl9b1goXPgMAazaD35UJr/SgsHiUfMVW91EoVrbNj5YTbU8lPnVAaDg8+BOm3koF22vyAgJPbAxylGvHBVsGD0khtqisfKe2RX8f4qg3OUpZdqB+epBksWq59hfsbsRkG7WGXcecXThcpN77QH+Elch3TGce6Nl7yWWKWHtMbpWOiqxt8woRVRlpsPamCH6fLRdFo3/r7yHcWxylxOrwToM7izs+JMGGnp5Yml44GqIZ0rIMWeBxktP65gGXmGXoNPdM4pyM/T61CYtJosM64louwsY6+92b1jeKr4aVvWO0+7IVM5auOTfXXvXWdEnaAGWmHUym/pXbLj+uwHCTH3YtNz3eohQW5+7sANug4cmvwBkealZIQy1YeRjYH1ISovDZboaTiwzwmq6ctb31YaGFjXNcx/QtCpr8Qx56bYdqz1Vd4wzv8bpu5hFlL/nVPnB1TK1wscMlqIV2hgATFSpJ4IjaTl fZeyUFMm ktmcMx+C7wb+K9HAGYlPrvW++4+bJC2U38ejMFcexB7m1U2wb9hEZiVQmEXgyaHDg0XKj7/2+LS+fS9sibcG9yUoqDySvqmv/ifN6Yh2ZOETQmLd062DOx7XiKzAUyyaJflPpOILh1j8iM+uN4Y/84AqqR2cPgVzGAPYB2B5453z60RvaXXMy0fXRwaHFn0Ito+9GNLvSgvY0WOmoXVD2H3LEGdmT5+4vSaSBBOJoWwzmVykdy8hUqVZTsUBow0zqCqY7vEtf90Jm2OmIN4IG2G+pfKNFnJgZzNNwucPH+wquXtud0iX69sCv7xzgBrGfNOKC4M120jZ8muLMwQT8Gb6Ebi5URoWcAAtjGD+W2fj1YlDpkR3bB8Sig9t5dNcOoQRfHeTFdMr5XHlohdTXzi2HUSmkZWiXLoF6mW+XxEAJBuVKcD7jcP/ckxN1W0qlG571ljP1POcJI1/N0lY15kljPt9utFtzlbGow6fIZTs40PloqeYuMHYieiksWiSApNM8AZ2H+llcLzl+5MduUPBJLCQX9z0/QOHLbh14iAyOTad9dxPPdotf4ldxpAVLIx6cwMoqDf89Q1p6ekaR6bIGTy2Sk8Q0ZNzLHokEDBY2zsWj65a51WuLiDkNQpG4UdIl6FTRq8QtzlBCGdcsjqMjsxcLSBZt6aI4u3imQxSuanE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Also add new test for xas_get_order, to make the test work we have to export xas_get_order with EXPORT_SYMBOL_GPL. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 32 +++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, gfp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, unsigned long index) return 0; } +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..26e28b65d60a 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,37 @@ static noinline void check_get_order(struct xarray *xa) } } +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + xas_lock(&xas); + for (j = i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + } + xas_unlock(&xas); + + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2066,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..fbf1d1dd83bc 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned long first, EXPORT_SYMBOL(xa_store_range); /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order = 0; - rcu_read_lock(); - entry = xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; for (;;) { - unsigned int slot = xas.xa_offset + (1 << order); + unsigned int slot = xas->xa_offset + (1 << order); if (slot >= XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } - order += xas.xa_node->shift; -unlock: + order += xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order = 0; + void *entry; + + rcu_read_lock(); + entry = xas_load(&xas); + if (entry) + order = xas_get_order(&xas); rcu_read_unlock(); return order; From patchwork Mon Mar 25 17:14:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13602535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 426B2C54E58 for ; Mon, 25 Mar 2024 17:14:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C565D6B009B; Mon, 25 Mar 2024 13:14:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDF416B009C; Mon, 25 Mar 2024 13:14:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A59C96B009D; Mon, 25 Mar 2024 13:14:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8F8A46B009B for ; Mon, 25 Mar 2024 13:14:25 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 64CEE80790 for ; Mon, 25 Mar 2024 17:14:25 +0000 (UTC) X-FDA: 81936210090.06.E217F89 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf30.hostedemail.com (Postfix) with ESMTP id 6F1CE80006 for ; Mon, 25 Mar 2024 17:14:23 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=J1fzzfBL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711386863; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bxkVFzSJVoLggCSIdplAlTRqteX3zHBKMrMrFYGST98=; b=nmPhlwlW/x/9Cvd9LV4GbRgvdyasMsCjJgxp8OLaL6Dt8eD/6rR+geI34lJIsnuS0vpvga XAI0/gNy/b9dcm7m8HNBSRcpL6rkwqiQWqWPr2aEnMRfbNLQGo2iErCUA52uMHHY6S9Gcd S4bmK33X6ZIysRzpkiA0bdtZiVMHiF8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=J1fzzfBL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711386863; a=rsa-sha256; cv=none; b=f+1wlXDxqtjaigd4okQSyJ9nwRlMYPiw+ed0e/GIbZZyAgHaJjMLSRcLfAoXqy0ruRDl0G 8uWFiJO83bgb+v07QmCU4b616OjIH3Ca6XFc05mnoNRQsFxiQl+pDspf9Gh7R8rY3wcWpl p5LupZVu8433pZWzDsyLSmuai0zSFCc= Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6ea9a616cc4so1226265b3a.1 for ; Mon, 25 Mar 2024 10:14:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386861; x=1711991661; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=bxkVFzSJVoLggCSIdplAlTRqteX3zHBKMrMrFYGST98=; b=J1fzzfBL5R8CeJNKTACab6RJKPe/ISIFWuBrBBvO4EbHx09XO+4UpiJk0yzCkWV8+S 2NJCDyRaImA7AZ4QNwPCT/VsyQVcKCHXjytIk19kqYLRNJ0nipnzVCpcPWiryMYP6rnH N9myNSvp2QlsIAIglb2txEB8amnpO6tzeOO1bHxdGN6EshD9N7gRwDsipG9wHSOOFmAc I0yU3KPvBlfiRu1cw+68FEUIoIF2mUqXeSxFYASYaNAmwDqdYzbim5NVl0pbk9as5clR gX2Db3C8Vss3cxjWqPcAE60Q0lrY5ez5kAtNuqqc8URUOiQSom4mEeImvv9hEJYGA6FB c+gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386861; x=1711991661; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bxkVFzSJVoLggCSIdplAlTRqteX3zHBKMrMrFYGST98=; b=JUbaxtgGNu/8+SkvAPeVsIfzqz6vvpVBoDm6UYosL7gZMrFV4vkFpGQm15UmwY4L1o D8t3SXsYOr6p7siRnH3exF4w4vBBj2Ml4CS8RdzYHZsL9yOrdC6vP+z915HyjulYGGV8 mGiS/XY8aUYV2k397Y7+MrjHCnRm1RAVELCouG8FyuwKxe6uE07PE58xPKeoCfpoyCut Ulpk3pJEB0+PbXaqSCiBz+w0YEwtqpU/m4BoN0NCm+s0vyMcPCl1Vjv3JreiWtU1lEhZ ty6OFBAh30uijoNoznFAckEcdsI7PAdZGflil/GGRxvROg9gEFr1MPjsxQRDv4mDQq60 UYTA== X-Gm-Message-State: AOJu0YzfbpvwFc0GOJ8tP0/5mLFh/j95Ggs3WOw515FI3F1bRSL9K1k4 V9D5k5Wu/NYBV0r3ZDo2Z1FpKWYtYHtCXMEdHoWWpkdOKRXl+lSskwIvlHZMyu2zCCkA X-Google-Smtp-Source: AGHT+IHqt40rBnvxQlAYOk3f3IH9W03fIAB5lQLk0ER2F//3YE3Sl2ab/cKHdSL/dz1j/62/GlTTNw== X-Received: by 2002:a05:6a00:399f:b0:6e6:8b59:1bad with SMTP id fi31-20020a056a00399f00b006e68b591badmr8163470pfb.25.1711386860921; Mon, 25 Mar 2024 10:14:20 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:20 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 26 Mar 2024 01:14:05 +0800 Message-ID: <20240325171405.99971-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: p81wjybn3pbiffdzk9tt4cttjj75bts3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6F1CE80006 X-HE-Tag: 1711386863-86882 X-HE-Meta: U2FsdGVkX1+yqwD455rYqUV+ZYMizNrNniS25V/vfHwozDhaGNd6TLEZl/dhCweHfXDArEimSdFfrgnHJ12N2ZDKmcGY8yDs5kCVKwDWzfvE+9zysgQT4YJVdrOTXQrV5d5O4hd4dWbsDki0jWgO6YVyVBPvhx5k6eOsVscrzFux5JgHXR45W+0y2phWgaAgQ5pYco+Vun99h3YVAs2nchUYA8qyAEKSBXSDfzRzs6g2j9ylJq7v+3gHqwIuYVynj1bFkgoBuWs+0VdoW3swWdLYJk311QJv2cGE+ALxIXbj0vthEqh0chTIMBG2JminczI6KlFjQgZXaE4h6xITumBWiMSUCqpcK3gLpKc/aCPF/ciMjW09CYQMCMPKtK/Ozu++8QXsdnv/z76h8Xyzff8CzFtU1F19wt6g3oVG35nJGRra/1uKNor+oz/LVx+RBfbvSz8zOMRmKwIvptbTsIsBHPVD5vAtsMIXJvNRDU694NgzYtfsGUP5YEWg7SY7Jn5AFLfSxV62+L/keqMny73RXLv/bvZoowLgj8yvQA91E7rWDF28jiBTcPhu35P1iPnV+YpAdEVkV50ifS4kkKh2y+7+plf5cvRFtjEeQq24UKsiQsT38ohsbTxfUMeLH9WXe2e7sLsqmQjkPff86Jfms9I28zRjvBWT9wYARcGxTF+dFBdlqs9e7fOih48KVdFiPhWrIVGcnJ7K/nhQa+aqrPhydncSA0VPj7w24k1zFTpZR8CII8cJSoxozOd67FdQDCNmWM3eji6AiQhwn4JRsc4DryyrBjNSVV/8f75yQ9AClzDChJqMMn1FqqmsBybSPhSigisyW1osz+IXUGuX20gGx8CJuPiWoGaZCnIChkjeEb1GIDZ1ImmvaLRHPDbnUr0QnIdyFunwG+wMBrOER52+ZQDCpp6pZNYThbjF80vzh4VOH0+M1/YBnnRREYDYHypiZi5hNzQfYom Am53RtrM 8+S0tGXzlT2mFPJ7Xl2tBLLNI4OOS92J0QQ1aBpH0Tme5+XMc9620CPBhV4tRaTtsuffReew5cJEBluE9BNN5CLJ28GIZnLi5+5EVdFljzzyv0sFiVgLpvzZy5QsRUPp0NanpOIRb5gnt/sIrm+Qz22LJkB2Qe2zJPwmLTsF5VLnA1/n0MmpSiJcyR3NXXSo5o4D9MMyfUpSTThGFK80sfmNet8wMEMHcqwo7I82Gpx4f1RXD8PnbY9kdhsuC/K2GkSJguwESuMoYuL08FdmJZmReisUsmLaoyH4NUJIDCpxwxRYYKdLO+VdsvKXtO448rvj7Isx8loDicxeOevmOv7XYECfamUFrjvHlQWUOssaBKsVK9pkkADV5XGjjItJZ1t4g6ZVXwu0oN8PEZVmLVJXnMZGxSaRN+L2RWrW2iqFH0ZoFIWdv7EvNcN5t2mwZKnvfdLJDv9VWdRMMU0TpXRxDBrFENqSiMoJ6xjnOQs6k10RMu3bm92IiGxZEi/l0ac67g9Pda+P3u7oJAeUUPLJ3X4Ho9VwHDe2cjDn2OnacejGBRnxKXDmrRUvER5AlqDZXZVldR6vxHGND2nvjUQsaz6puXXfyfHU4kyHjy5PHiwJw+p2peL46eL/eiCURIkLRFukmyAkMKkd3MyPigzNoAt/pnN4dyXLY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691 iops : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691 After (+7.3%): bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976 iops : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976ยท READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec After (+12.5%): bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 +++++++++++++++++++++++++++++++++------------- 2 files changed, 98 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 26e28b65d60a..f0e02a1ee3d5 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2015,6 +2015,62 @@ static noinline void check_xas_get_order(struct xarray *xa) } } +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j = i << order; + for (k = 0; k < order; k++) { + only_once = 0; + xas_lock(&xas); + xas_set_order(&xas, j + (1 << k), k); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once = 0; + xas_lock(&xas); + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2067,6 +2123,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 6bbec8783793..90b86f22a9df 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge = folio_test_hugetlb(folio); + void *alloced_shadow = NULL; + int alloced_order = 0; + bool huge; long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *mapping, VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge = folio_test_hugetlb(folio); nr = folio_nr_pages(folio); gfp &= GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space *mapping, folio->mapping = mapping; folio->index = xas.xa_index; - do { - unsigned int order = xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order = -1, split_order = 0; void *entry, *old = NULL; - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order == -1) + order = xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old != alloced_shadow || order != alloced_order)) { + xas_destroy(&xas); + alloced_order = 0; } if (old) { - if (shadowp) - *shadowp = old; - /* entry may have been split before we acquired lock */ - order = xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order = order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp = old; } xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow = old; + alloced_order = split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } if (xas_error(&xas)) goto error;