From patchwork Thu Jun 22 08:53:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 254C9EB64DA for ; Thu, 22 Jun 2023 08:54:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8E658D0006; Thu, 22 Jun 2023 04:54:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3E0A8D0001; Thu, 22 Jun 2023 04:54:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B7A68D0006; Thu, 22 Jun 2023 04:54:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8BBB08D0001 for ; Thu, 22 Jun 2023 04:54:14 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4DF0940B18 for ; Thu, 22 Jun 2023 08:54:14 +0000 (UTC) X-FDA: 80929772028.24.CD2EAAA Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf17.hostedemail.com (Postfix) with ESMTP id 6D9B440003 for ; Thu, 22 Jun 2023 08:54:12 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AuFJ2ff7; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DFMKKDRv9BEw0wZ8UUzCf9pHUtxe8HkFUWg9bN69VRM=; b=ty51A9sfxV/d8M6dPILIMNq8S3GdZQO6jgjvSYDemR0AvNJhe68RVsfEQpTi1Tphv7C+le 2u5aeN7B+OBV1mjpV+WSpKh37zl55t8NZaSS5yAI6ioNREJoAt2tKxUsRxI4QJVic6uapK BtmaInFRbVbOE657rGKsoDFWnNXPVnk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AuFJ2ff7; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424052; a=rsa-sha256; cv=none; b=IvBLifgTNwiQL7kK0cmtHDw5IkAZ7cCm1cDLub05DYuQGJli6TH5ZiFngbNmcm6p46s7pi KZBNCUqswUHJyI8pK7Y1kOZdkd+I2ncPdSmbB4cfp114dOWl2gGk2+GUgtTXNzyxx752Rs 8fhBxoWksYmii6Tzwphx4vDmeOEwC5A= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9651715ad.1 for ; Thu, 22 Jun 2023 01:54:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424051; x=1690016051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DFMKKDRv9BEw0wZ8UUzCf9pHUtxe8HkFUWg9bN69VRM=; b=AuFJ2ff77l/LXEOtYzZWEupZaAfbwt3boPeShk25fcGMwTsb6VlfqfAhMOVgIPnKgi eqcOSF5WVHBBVioigqwmY9UAoTi5JEhiUSGdMReu4dDGNOFEVgxVIc2OKO3nltUsVDLn bIiXH0DnqlXam4z90R2tFa6XgNVpJSLaIq6Gbf9RXLnB8BOaf0v/n1c1hG3kRZPk0MVc mjQpq7kQFhiCgtBijMi8fR+x4TAH8zuouOKbSAqWM+GHeJnchPRb4XIBJxswbiSbm3XO sXBcqfmZ/ARe540iNA/E7F7ZKvP/VvSbnbTILDPrN3l0DanjIGoHjsT1U12ioP3F05dG yTtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424051; x=1690016051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DFMKKDRv9BEw0wZ8UUzCf9pHUtxe8HkFUWg9bN69VRM=; b=cqGGLIpMo/hjvqXizFC1OUb4JyB8iJAMf2aoBvnUU9n+UPN23mcjRobovlv50H6jux 8QpCv6UWg0UCHoYKGph3m59o6KZ6VL+0PVPQ9wTOZX3HUhHVDU2+/mCutunjywUZY+jO VAlgyE584JSB++A1NhjvqL5prL67S6DJ09tvKuXo904tB6c61U5xbP5CCKYT5VNSK6du dBTpcHPa2AREgzXvrhVk6SYcpGRNBwBhoCdFX+e7UDdOcu7oRIEFOXiM+/uN8At3tEF+ 96TrXjE+bczyKe5iQis5lpxLTbZjFnRsrxj8b6mdbDyKmq/lXOZicoMhY9zEyFGGF5ph 84Ag== X-Gm-Message-State: AC+VfDx3OwPKRCSTlb68OQljcreyzZ07IO2BHn8FwuRPeiIiZ/lRzljU G6XV8eAezv30/945cMslk0PuHQ== X-Google-Smtp-Source: ACHHUZ41Ov0V9PIJeBixnqaY8MSMJrvGTOtmFbuG0vAMaQAXgKytEj5uEpxgQHkUs9eKy8wwGzA30Q== X-Received: by 2002:a17:902:f691:b0:1b3:d4bb:3515 with SMTP id l17-20020a170902f69100b001b3d4bb3515mr21827814plg.0.1687424051106; Thu, 22 Jun 2023 01:54:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:10 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 01/29] mm: shrinker: add shrinker::private_data field Date: Thu, 22 Jun 2023 16:53:07 +0800 Message-Id: <20230622085335.77010-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: wfymdtzdp9hnxu51gw6bxbesuna1y3t1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6D9B440003 X-HE-Tag: 1687424052-788289 X-HE-Meta: U2FsdGVkX1/pucY2Aqo47vk7uL+VTJIe0Z+X15iSbdmHmw9SAB9U5H27ac3pmzAH/kFjQiM7m7VRLXxCf8/of0SYxTgETcqOUDiadEgbWjvglSXRFhIOQgMWAxeiuXRMOgTsUdypWallN76Fyercf+pZnN0VwTYkbc+c5MDVqNwNdAo3HvgIDB41Thav/LtjJJBVp0MLijA431YO9JpDb5428ItwOpMuMdGii1jdjNCRzAJWr56BTCBkyY2UOU9D5HPkMgSrBUfkVjjkOEP0JGZb8AHidlm6FrY0tUtw0AOOl/kWC99oN+YvdvyaB8Pz7ghUYDfCI8/ZVYoZk8DSDwiKULdZwFSQu/eUSdDT88Nm54QaffBKj9knNCLctWeIsbTidrTXBQEBiwDnaCSoztvfxC3z0BdBUwxsyccNJ5EygrseYEdyT6a6xcMixD9snmDY2vjxo2yT6oNPhKVOeJcLEsqiPPu1A3gW4YkIhmIi6/DyHbnko6nYZaIo18SYq6KtXO5vU+cxVabZwxIgwB5M2EnTzm1ON0qZYV68rbzEZ67dsUf8uF5NDDlJZ/lrqJRPb9NVPUYMW1ZcfTttnHAbIo99nQ9rZ9+aR5I1So/B1KHGgF79AaU/e0sDB3GOmjpRMwoVB15MHZdi5fQ9IkGXEacNEEG7vea7Jj/PzbqpEZkK/CJLW3pXdrYa6CsJEXEV7eQvdwlU5p/ZJYQvmhfX95YGrsRyBUpwCw//JQABPU4uukrdlEFxGPx6dl4CZJwSkbp+f/G9CJHZukqwVBP0grJwWZGbFEDg5hJ5NeYvgdks71lpuzfWOTmUfVHv/NsAU/7OIM5z9eiYQDr1Z0wngyDAuRcYsRjHt5Ysp98VyHueCkFjTufw2r/rmMGpapU0oUA3vdtUSp1sq3xprucW1WdNIQI1+obJWSt/G5OoyCo1USR9Pj1KbB2F5rOXFIWGfRFj9v171ebfG5U V7exIDzk EEtc8o4oIyJV72tL/Q1SyLznEAhwRGnin/nTcQmU3xB0Lba12ed0xdDUGeRwoDSAjbds2tmLn/PnzX66Pzwh9aX0alRrtohvMHCHogg8dcNtMB+K5559e54rpao1ryctuKYrRM6KryI6yG+uktGgFM92Pde8/KzTwgAoABKj7wLJybOGCHiY6kYrzhbTgyE2pMr67ena1iPrI6ct/lJW6U29kOweCv5lolOCEfIrjk2jKgPVSIIXkekyBUJs/c7DQWV5jf/IIDtB+Dwl9oCG3GPxeHyBeBMoRTg6kjzO9kFrVBp0bSz3rFGt1ue/F78QanRMsmpFuBW51E8BjB/uxL2dM1r/UD+8g4VDwT1AR+t/xCh4QPigjVII+L+qHqy1kY2vHl6IT5NQIbCj0BJfhnB/s1vgeKZZxSDvq8i1k64vmSbABOGyVV1wacgEDss/4oLiYI5JT6Xo/++s9g/srnvKyDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the dynamic allocation of shrinker instances embedded in other structures, add a private_data field to struct shrinker, so that we can use shrinker::private_data to record and get the original embedded structure. Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 224293b2dd06..43e6fcabbf51 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -70,6 +70,8 @@ struct shrinker { int seeks; /* seeks to recreate an obj */ unsigned flags; + void *private_data; + /* These are for internal use */ struct list_head list; #ifdef CONFIG_MEMCG From patchwork Thu Jun 22 08:53:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0949EB64DA for ; Thu, 22 Jun 2023 08:54:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F9F38D0007; Thu, 22 Jun 2023 04:54:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AB698D0001; Thu, 22 Jun 2023 04:54:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 571E28D0007; Thu, 22 Jun 2023 04:54:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 490AB8D0001 for ; Thu, 22 Jun 2023 04:54:22 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 22D64140149 for ; Thu, 22 Jun 2023 08:54:22 +0000 (UTC) X-FDA: 80929772364.02.9BB3DDA Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf29.hostedemail.com (Postfix) with ESMTP id 5671A120006 for ; Thu, 22 Jun 2023 08:54:20 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Sk9j4D1M; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424060; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4+KB+8A1a3nPwmndVeoDnF0UUBrNJStZPfOFWRboZRM=; b=VqBvHcDO1ac2T1+zFANeWh7WmH6SsVXsO6isEU2HFN2QVTK4CF3JVu99meL6tX8Ma/fL0J 5c7/WCvHPCIYZBt2Xqy9LeG+RIljhVv1WiOVVuMj5p2o8mak4NgBCqpIh7AGxxJh8jpzTc h9odSCJWnbp3vF8SgGSwBwbC4KLS9vk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Sk9j4D1M; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424060; a=rsa-sha256; cv=none; b=yOMxC9KIK7vZOi/N+dC6MvKXn+5P9Z7i4DxCTT4v6CrDzJME6ycIGgirLDezU5OYQWoe9k wAd2WNuA6+jTASRjjUDtsDXwhHQbuJJfyQUwGD8IBdCtHlfW3lAokYB8LbfnP2uDkZSpHH zSv2vHpxdPjLXbDtV4kwQnhoGBH6C6c= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1b693afe799so2463745ad.1 for ; Thu, 22 Jun 2023 01:54:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424059; x=1690016059; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4+KB+8A1a3nPwmndVeoDnF0UUBrNJStZPfOFWRboZRM=; b=Sk9j4D1M64xyAQoy0v1SMiRRuiugSGVsgHhv7vYv9c3IikKaUFVDiFJczRjS+t8QJh 8bRZCGyYvkRGHDm8uUJh3JrpsRJfZFHSfxrg783G1zZ8ag+Nrop0sZP8BnmbcU+0S25q ZIKtGKgKPIH7cGpK2fHrvYtY9p82kt8TosN3NcpFb8CLbiW0zQyLzdahaQYLKutpL1Kb iV83VCrLSsoe4tZLXCgr7bzLBi5xWr4Iu6AgSr+ApJLZMKUxhfto21PwHxkzor+s4sBV FrT1a+LNzQzeAZGp0p/ozv8VRhpBFeAgfX/MFGdkWk3iK2t7SrfDgjAEjXYPkdmChJzo lkUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424059; x=1690016059; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4+KB+8A1a3nPwmndVeoDnF0UUBrNJStZPfOFWRboZRM=; b=gqRbabf78plOFsmHc8t4sLggrWfWzFcbazidcMUe67jmckHmaT+xsMkZGdL+YUis+L Aowb8e1QZq1Q7W7ebhFupHkqZpwIoBBoHxHcjq3rcAQj6/s16O7CQO13/eblhmN/gDv5 eWFNrCPmHW+iLtG1hC4xpvKiui2ft+ssqGzyYI5BlbANblGHuSnwE9abfbI6yylOOi4C ASGGj6CbQaIjpptn/37hvaFx2+0/Bdgs7sgaXTnu51N6sXY1157OVd+my4bBS1vDefci JQ6QAZpAyzJE7+P0HwQibnD+0Mc8sGIMZHWVYBMUBdQ9rInk9kUOg49QV5zZtwkZhOAC bDRA== X-Gm-Message-State: AC+VfDx6DhwUFeMoMxTHAq0Ljxrwwz8kOYNP6T4AMbOxLRj4h1LOFOpc VO+W541JW4wo5Qy5ezXaj0sUqQ== X-Google-Smtp-Source: ACHHUZ4ysdKcTcW8I0NFLAt2c3adAhLn+VAaFg6uutJFhO8zbaZNgTMf0Qt9rlrj2Rh11Ffv9G+1Gw== X-Received: by 2002:a17:903:2451:b0:1b0:34c6:3bf2 with SMTP id l17-20020a170903245100b001b034c63bf2mr21537674pls.5.1687424059079; Thu, 22 Jun 2023 01:54:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:18 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 02/29] mm: vmscan: introduce some helpers for dynamically allocating shrinker Date: Thu, 22 Jun 2023 16:53:08 +0800 Message-Id: <20230622085335.77010-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5671A120006 X-Rspam-User: X-Stat-Signature: q5f14xqykxefiugcm1dmu6sxnu9ncsgx X-Rspamd-Server: rspam01 X-HE-Tag: 1687424060-947350 X-HE-Meta: U2FsdGVkX18emWl84JCetwxr+NOXGlFhPiHPXQV+XUN2YrQ29yYyih/K1Hf3uYKqZOfwN83NyShJBvKDKF43k0K3A2Pgn30m9IISg4mOcb8kae2xMi1BnZ+gKq36ItaA7Wuf9Iwa7v86PictD6s+Ype0ShDUlkEoVfuepz7v8Z74kw8PEbF7SL3NDvvxjJpBJnzD1bQYnYZolJrWSDjCRIkIFlHjU+SoIittz8zXqYnmwga+5I3tpdNFYDZGbZrtlX6NLsyOgRvWjCqukltDnmVABrmjjPaMQLg276jyhhXD3L1T0zfIg69gBT4/X4+Lpy3i9CnxMlRXq0Nj2xx+5ELNnKCsmGjEQKOSrOlsgFPaWTrSUZjKPs/IfpEP+Y5xzJTaR+XDVUi/H/EYQDOQMDOCl4jNx3O/YOzw/tceghCcvVf217S02p5CbeTFG1yxt8ce89axePrY2md5LECohNUmVFl++v6GQq86gfwk/tBbCXLGKd0fpQmrp+wwWS9d0kq5xbEkwhCamgvJgeZdEa1Sbof/8FkAuu/15PhYqcuIU69jR8JHzd5k+s65HqliXeoWIcKk5liM3Wap9Cz5zQV7QRerFCd0PLyL9Kn07sL7AEmvvg1er9gUGFpakcZEI5JEokrG9FdfrsqvS2E+AqLGlGNFQk1uPhjNkVsbTd6XTE9CAj+a11i7bYgjsz3/3y7axtEYmSPqQyUtWsaH4W8QnF61TVI1gUPokEohfzD+bXYpVNpHMsAZ55dYfg3bg0g6aIto1Ra71lcCiEn+/Q7wZnkUY6PZS5tpZJMvnPJ31R+oHiAz895l4ejWlcfkG2RTNHLCByg9zN0ucFEHxOF5yHvEuQiagwnxrU3WT4FP8HqUIIfPU3PWzdUPtFh+YZVL6kvSZSrYBpN50eSmxLt1Ipk0XxJH7CaAMTLbbTuP1T7GGmixOoyg+GT9INl+7mGxxf67R4mGwUZ4BDz ZGKg9f0I OtozSBwtJ4UG7BwyVcpHIbCSEveDUf2v/yVsQxDV4NdeLOJIQ2nPDsb6N8hN8bo6cC/oCVslZmRAPkmsa2HYBvFcaAPTjc4ybq63wBAbJX/D980Py5WAjCHGIZupTya0PGgAbanLdV9peI9r3NBW46jqr2144i2S5fSXx+JAiluTSAMMzqjuJaXy5TVwnN6oOu5yIBEpq24Qiry9ZiO7rhoq/0d5Mv+rABMGNiKTulwMpook/f5BT3lMQy7o7zF1NPuGgwLx2aK2quDah/5x2To8neNPKz40RAVSO0UdhGn1rchyZqbrSKElhgAiavY4LN3hNtyfpcEsL9q8J/vmxf9GT5h1QJQldizXSOvVnBV1gqjawAox6apxgUi5Lg73zD+6CYM3uhnJQd6Gsh72XOZ7G8ydcXCgZhuTZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce some helpers for dynamically allocating shrinker instance, and their uses are as follows: 1. shrinker_alloc_and_init() Used to allocate and initialize a shrinker instance, the priv_data parameter is used to pass the pointer of the previously embedded structure of the shrinker instance. 2. shrinker_free() Used to free the shrinker instance when the registration of shrinker fails. 3. unregister_and_free_shrinker() Used to unregister and free the shrinker instance, and the kfree() will be changed to kfree_rcu() later. Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 12 ++++++++++++ mm/vmscan.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 43e6fcabbf51..8e9ba6fa3fcc 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -107,6 +107,18 @@ extern void unregister_shrinker(struct shrinker *shrinker); extern void free_prealloced_shrinker(struct shrinker *shrinker); extern void synchronize_shrinkers(void); +typedef unsigned long (*count_objects_cb)(struct shrinker *s, + struct shrink_control *sc); +typedef unsigned long (*scan_objects_cb)(struct shrinker *s, + struct shrink_control *sc); + +struct shrinker *shrinker_alloc_and_init(count_objects_cb count, + scan_objects_cb scan, long batch, + int seeks, unsigned flags, + void *priv_data); +void shrinker_free(struct shrinker *shrinker); +void unregister_and_free_shrinker(struct shrinker *shrinker); + #ifdef CONFIG_SHRINKER_DEBUG extern int shrinker_debugfs_add(struct shrinker *shrinker); extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, diff --git a/mm/vmscan.c b/mm/vmscan.c index 45d17c7cc555..64ff598fbad9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -809,6 +809,41 @@ void unregister_shrinker(struct shrinker *shrinker) } EXPORT_SYMBOL(unregister_shrinker); +struct shrinker *shrinker_alloc_and_init(count_objects_cb count, + scan_objects_cb scan, long batch, + int seeks, unsigned flags, + void *priv_data) +{ + struct shrinker *shrinker; + + shrinker = kzalloc(sizeof(struct shrinker), GFP_KERNEL); + if (!shrinker) + return NULL; + + shrinker->count_objects = count; + shrinker->scan_objects = scan; + shrinker->batch = batch; + shrinker->seeks = seeks; + shrinker->flags = flags; + shrinker->private_data = priv_data; + + return shrinker; +} +EXPORT_SYMBOL(shrinker_alloc_and_init); + +void shrinker_free(struct shrinker *shrinker) +{ + kfree(shrinker); +} +EXPORT_SYMBOL(shrinker_free); + +void unregister_and_free_shrinker(struct shrinker *shrinker) +{ + unregister_shrinker(shrinker); + kfree(shrinker); +} +EXPORT_SYMBOL(unregister_and_free_shrinker); + /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * From patchwork Thu Jun 22 08:53:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B18BCEB64D8 for ; Thu, 22 Jun 2023 08:54:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D7338D0009; Thu, 22 Jun 2023 04:54:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 487488D0001; Thu, 22 Jun 2023 04:54:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34F9A8D0009; Thu, 22 Jun 2023 04:54:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 264428D0001 for ; Thu, 22 Jun 2023 04:54:30 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F2C0F160A3E for ; Thu, 22 Jun 2023 08:54:29 +0000 (UTC) X-FDA: 80929772658.11.DEECD84 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf15.hostedemail.com (Postfix) with ESMTP id 2452BA001F for ; Thu, 22 Jun 2023 08:54:27 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cC13yiK2; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424068; a=rsa-sha256; cv=none; b=JOQavfSEvHsM3q493vVyRj0/aBXRNrgpxLvLZ8WSUlMqH1yNRDQ5EhLGld1yeY9mGQh3jo i97Itocdn8x4Yk9pA+c5C22lgH5Lc+/rrMEDVTqLbWkM1R7SMZaWUwfZMrM7+gSghmzGut GQbSFZUo8mYtfyNYbre0uXpsRX9iuYg= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cC13yiK2; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424068; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qIt8zXdDCaGLXCE6rKF2JMKvp8G2AK8nuTQphjV73ms=; b=QKgN/hQxs7N/tqDlnSZ+XGv9C5KyP3yXsMITJl/ofiTcTz1PFtEPO8vrARAjtsuvXB78B2 HaIMRyaOEBJqsH6WJ1tPlYKl+dh86Oy7D+RzmoKlJCYVNcNSIJfARAlKEg6JFYScMcHHzu rYOyYldRXIyE4j/z24WF2eZAajN+mgI= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9652365ad.1 for ; Thu, 22 Jun 2023 01:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424067; x=1690016067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qIt8zXdDCaGLXCE6rKF2JMKvp8G2AK8nuTQphjV73ms=; b=cC13yiK2b2HqEC2WYanx92tTHG9ibgrmYLbrgzYcUarLdydxeypI53cwuyYhjaQzJp onJq4vBBmMajr1poWh86FQTLq2y1xSvoknQDzCFUhqc78tS5eFxPwQt8ueUiPUlQcGJ1 NoUMAyqrL7rvWv1XaJ4CPnN6K7xK1sRx+XiYPW8paVbE3vb5Wkc4IFamUXhI7GXmfwho QPeLUm4YlUE0P3+vsMp1UZZv6g44Pla+Jp2DOFJOg6O2qS7xw748UZa1YIoXHuTtcdry A4esVq8CH+pV+6r7bJr/mBJgAfFPjM5J1TM9ToGnDJSCePlm9yunBGrnuwbojkh7OG6q rFSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424067; x=1690016067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qIt8zXdDCaGLXCE6rKF2JMKvp8G2AK8nuTQphjV73ms=; b=Mkvl80SCfadx9TUSglND/Z2z+RwCoVH1dYN7r4AOjx6+Z94rciqRKHDxWYzQtfIonr ta1F5PgyZLbKW43hkJez2sgIm9ezMfamT0OT2A+ocnmknZ6OX6EhmNXVeWy83Wya+ILl xvDArrKMU+M0LIHWj7gCh+njcTq8ZZisq5zIto+2JtCC09p2lq7xumoMJ+wpC8yq7+pm /tu0st9tEBewOt6Nw9wNgnMOwUwvQriOTPEK6Ir67vHjAOfzrUChKQnPnEmykMUEHjqQ LEkSdY9h0wqDmynv+AvsCzU9O4CKPXmI5+Uq4yrxHU3Kpv3nLIPMsrbKtEeyYktcwvVJ TIbg== X-Gm-Message-State: AC+VfDzFwQY0h+//ZOXZayNZoBuQuzBg2kGbBvRwMJGbrjxQGTJstmKU Qghv5lY8tzP03/yBVwzsi6dVzw== X-Google-Smtp-Source: ACHHUZ68xppE2OzenI0HziNaJw5pBbhsLnOSkDmr33RiuwGHaNU1efiGcJgTbt3uCyHHsLierWgKoQ== X-Received: by 2002:a17:902:d489:b0:1ae:4567:2737 with SMTP id c9-20020a170902d48900b001ae45672737mr21909019plg.2.1687424067136; Thu, 22 Jun 2023 01:54:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:26 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 03/29] drm/i915: dynamically allocate the i915_gem_mm shrinker Date: Thu, 22 Jun 2023 16:53:09 +0800 Message-Id: <20230622085335.77010-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2452BA001F X-Stat-Signature: x5bggp454jueadfp5z6x18wc9qapx61p X-Rspam-User: X-HE-Tag: 1687424067-400971 X-HE-Meta: U2FsdGVkX1/PgaF4qMZvtbYtIhh3kz7yj5sOQhFsXE90Fh4PHS52HN5IkiNuiMu3ssOm1e2dRtiWjZmzuZm0+mvrOfPNs82aiSc/D/8Gqah7AQYiaWQ48U2IQAykrCV3udbg7XqwwSanvyoV6EnTDNEiaSslmLOzMnsC7UjkxzIBsO849Pt623+605N0bdMp0WXqHHRVelBk8JpWfMbDmYlenW/1gYEaG4QEfw6ilh8nxFx7D1y78P63SQDksf/zCeoAW3uDlH9uyrfISHW+tAzmhti8A6AEk2sdWVhXzFjpBZ/j5e7WZ6R4xwrNMupEcHjKz8/WXupFjHHYK+14QQ+/CWStWr388IdUDDA4WMwc6ATEWXYOjvIUoh5CUmqibnPyuch7HUXmPe3hJZIUImRKkYz3g39sxEeYDP8fVolcsZd9WQZtuc7tyVnRWeG2vFjC+W/zjOPN2lPvfnRk4S11KYvbRdyMsGs38seI6nfnubsv2ctTw7qOFv5WcGnVMXmmeeZ5XgDSw0l66LKLjnVtW9q8k5coiEW6MhjgtWxEzFKzYEz8b0+OTVaeisNY5Wka+Lge1YZwssxKjGnRtUbKniOeUaOkLX6FR3Gt7S7oJcFV8hPj4sA2movNpciyBn9kJBun4xLtFhxe5N80yrsS/NsTgn0deejy0W+HHQld5JOkBzQxA+qGW9D3kjHfwCIgZmZtBLsJzXk1Arwzh/QRSYmLCDou/r3LDobyilU639fAEm9HiI6qemfSbsccAJOtHplkWKVHqRrIeu8nFY3bHaqQJPrXz+u2myflDj5fHz+cN7ri1xoiWh6jUNBEQXtf79hw3wcDzAsT11sHk12xhjfLPWcAjJTUDlklFgRP14+I9eVcJohfa45s9UFUEUgoSGnsPwtB6xQqrMcna8wvWLDrov2HZJCbseFnk4ko4V4ovgR3/U8BqMRZu0vvTXt3G5M371fNJICo/Nl QzjszAXw WGpk2ely4GeCrHjfNfKUJ/lEm3NewNpWQT3kUIycD8+4HuirrheHRvCoGLKgZUaHdQC9jokFx5JUyyTgi38b4WBT6/Y54cEXUlHhzbuW5jWpyoCd5P3+EiQgDlQYr3SAWuxUnmai1u7nTwDqquO4fLB0udG0PDgDeM91GkA8kSNrlas4gmfmPPWo3br7Q6+LcIQ2VyuGPaVg7AVmrZyHtP0Xhs1ImHWl58NxQB8ptD7iTTZ+7CBr5/pfgSWJIT9bGbIqC4gfQaIT8AZBdJBWN6wG/AneFNJdxcZ8mvJOeNw5/6fVLvM7YOHgsxFsWgIJH+cKbwTMm1vqqo+EUiNL0jSuhZC2g44njI2JGMdb7KyR8ObVI9nbtAY8+3HUc5A/5nA+6nnT6vqHtgnGPgiUxDpimkYwSR257EkPffB2ocGFOWH86LNuiOCizgDGMLT+B8cEkZtAbS3m9dZcxggi09sURoQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the i915_gem_mm shrinker, so that it can be freed asynchronously by using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct drm_i915_private. Signed-off-by: Qi Zheng --- drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 27 ++++++++++---------- drivers/gpu/drm/i915/i915_drv.h | 3 ++- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c index 214763942aa2..4dcdace26a08 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c @@ -284,8 +284,7 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915) static unsigned long i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) { - struct drm_i915_private *i915 = - container_of(shrinker, struct drm_i915_private, mm.shrinker); + struct drm_i915_private *i915 = shrinker->private_data; unsigned long num_objects; unsigned long count; @@ -302,8 +301,8 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) if (num_objects) { unsigned long avg = 2 * count / num_objects; - i915->mm.shrinker.batch = - max((i915->mm.shrinker.batch + avg) >> 1, + i915->mm.shrinker->batch = + max((i915->mm.shrinker->batch + avg) >> 1, 128ul /* default SHRINK_BATCH */); } @@ -313,8 +312,7 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) static unsigned long i915_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { - struct drm_i915_private *i915 = - container_of(shrinker, struct drm_i915_private, mm.shrinker); + struct drm_i915_private *i915 = shrinker->private_data; unsigned long freed; sc->nr_scanned = 0; @@ -422,12 +420,15 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr void i915_gem_driver_register__shrinker(struct drm_i915_private *i915) { - i915->mm.shrinker.scan_objects = i915_gem_shrinker_scan; - i915->mm.shrinker.count_objects = i915_gem_shrinker_count; - i915->mm.shrinker.seeks = DEFAULT_SEEKS; - i915->mm.shrinker.batch = 4096; - drm_WARN_ON(&i915->drm, register_shrinker(&i915->mm.shrinker, - "drm-i915_gem")); + i915->mm.shrinker = shrinker_alloc_and_init(i915_gem_shrinker_count, + i915_gem_shrinker_scan, + 4096, DEFAULT_SEEKS, 0, + i915); + if (i915->mm.shrinker && + register_shrinker(i915->mm.shrinker, "drm-i915_gem")) { + shrinker_free(i915->mm.shrinker); + drm_WARN_ON(&i915->drm, 1); + } i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom; drm_WARN_ON(&i915->drm, register_oom_notifier(&i915->mm.oom_notifier)); @@ -443,7 +444,7 @@ void i915_gem_driver_unregister__shrinker(struct drm_i915_private *i915) unregister_vmap_purge_notifier(&i915->mm.vmap_notifier)); drm_WARN_ON(&i915->drm, unregister_oom_notifier(&i915->mm.oom_notifier)); - unregister_shrinker(&i915->mm.shrinker); + unregister_and_free_shrinker(i915->mm.shrinker); } void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915, diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index b4cf6f0f636d..06b04428596d 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -163,7 +163,8 @@ struct i915_gem_mm { struct notifier_block oom_notifier; struct notifier_block vmap_notifier; - struct shrinker shrinker; + + struct shrinker *shrinker; #ifdef CONFIG_MMU_NOTIFIER /** From patchwork Thu Jun 22 08:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAE5AEB64D8 for ; Thu, 22 Jun 2023 08:54:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80B8F8D0005; Thu, 22 Jun 2023 04:54:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BAC78D0001; Thu, 22 Jun 2023 04:54:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 684C18D0005; Thu, 22 Jun 2023 04:54:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5B2C78D0001 for ; Thu, 22 Jun 2023 04:54:38 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 250308067C for ; Thu, 22 Jun 2023 08:54:38 +0000 (UTC) X-FDA: 80929773036.12.E4701C8 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf13.hostedemail.com (Postfix) with ESMTP id 1E7A120006 for ; Thu, 22 Jun 2023 08:54:35 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AF1n71Ib; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ftOXXP2bfTIIt8St3T9822C8Mz+yI5OAEWwVCqdxqi8=; b=kboO1sSzg+ZdZJoqPVtWctbkFwnCmY2MadwUIF9IgQkIhBKUbIC3LQfMoaYQHzlIpyxB9l AvvLY2sY9auvqxkxxxPbPB+1yzKCFtFG+Eb+oVht++Rj+6L4+ozw8zXFwAQv43pSvJs9en WdHdL4w+kmRQVhd4Q5PQDxZSrq3nFJY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AF1n71Ib; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424076; a=rsa-sha256; cv=none; b=DetwN51yF6UrW0OPM/c6G4BWchLUqT6LRnNSbyfejTnohWOrquyZbjTiPq5V3GuJMQ7T2G P7yRmIvSf0EyKzR6XcKWdbI1vzkV85Zhewh6DWDy0NED/28RZ6Le5tjGcq0stzgUZ/Xm1G oQj55Un6NrnXBqVQU1OJwn+ahWALvEw= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9652655ad.1 for ; Thu, 22 Jun 2023 01:54:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424075; x=1690016075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ftOXXP2bfTIIt8St3T9822C8Mz+yI5OAEWwVCqdxqi8=; b=AF1n71Ibzw+/eJzW8ZfFCXm2Gb4L6rcgcO3MWUfrJXSOCg2q4/6sdbKki+cciYNMm8 euETZJp/Yv9okGd+5XPvMxEgndnOAmqccUN3sNROxfnHbIeAfCrF3CESQGiUHVaKUyC0 MpSE/83T59YTQuiU5NMvjWSDBOsWVTepqqBeQE1tIy9JZFyOQuULXF0wbqj90XyCa0gR wARDSKvxEvJy+BtXj+ipth/FF0Y38R/9qPz9cOOh0wy/h6cCvspbbaNHGZoc8L1j5F66 eXHRgIUKfoLp9EhNKNNczQL499XVYbOwcbEp2A8mZxNkm++j1bZaQj9vcnyBlktuqhCK 186Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424075; x=1690016075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ftOXXP2bfTIIt8St3T9822C8Mz+yI5OAEWwVCqdxqi8=; b=VS5Gy1AHu1mv3xKh5gmJ04BFJAv8P7BLNH5n0DVAHadXBENs55LO1ImBCNqTPnjk3H Dq7PRKJsNOPpXgbYcD1U7hiJzAla3uB19/YNhLbVuGm9B7orIjKM+hVTOMZP4MIi6aqx KXaana79Hnyi8qt3OUfr3jzFitT0SE/xrVR2NOODszlkEjmQXOD1Og+q4n27E0sBEqFB e9xEDulOTgZg+8jghVzIIsFcolRKOCzLAqcx/vS4nw/DRgW39ueQAYEYk14sN51grV0a R/2/Z1hXEdF1fFKVb/J48eYvU+SGFK4Sw8Yq6rZhZlxT7XCLw8lt1WQXxYNkpDMKBl2V xsSQ== X-Gm-Message-State: AC+VfDxeHpNIy1qgNJRhNfekWAsxph0GlgdNvG4fD/lHnRudWGDpq4k1 lTQda10XQJWzVodtidpz/p7ORA== X-Google-Smtp-Source: ACHHUZ5hV1GhQPf78RPEQj8dLuXzVGGtbX4uDKxYqhUuZR/F2WM6CQGQNWMXHDLRseslqPACLVPKJA== X-Received: by 2002:a17:903:1246:b0:1b3:d8ac:8db3 with SMTP id u6-20020a170903124600b001b3d8ac8db3mr21942648plh.6.1687424075082; Thu, 22 Jun 2023 01:54:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:34 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 04/29] drm/msm: dynamically allocate the drm-msm_gem shrinker Date: Thu, 22 Jun 2023 16:53:10 +0800 Message-Id: <20230622085335.77010-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E7A120006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 1jbbu4xbqy9au4jmef3kezeou97m94kz X-HE-Tag: 1687424075-411365 X-HE-Meta: U2FsdGVkX1/+b+eWb7dJKfChHL+jrHUstRKPs/Gkb3VIb/6CgG2sEdCgPxmWxZeSzrggrmzHd2WUTzgNjJuDokQpushIvqbMIS4KbL3LGCTZYc8dL7HISmbUOQXTyf9l+U7twokt5owE6JaAiVdqNK9d/3HqdCDUhGd+VwvZqWOfCEim5hqh2aeqm73TdKE8fuRwYBMRazYzPSj+qaZm1M09E4ZlghOdDKgaqXtGeRPUMwMEX+/IImR7iu8RO0SrJzCGbEbSq35sm96d/cJmX3v1DuGDlKClLp/KqplgLpk69NTRHR9GKa0U/BhwsV2Vb+nWp8v2sodwf2759oW6yWqvyHLSnVRClhZqE/HK6I2ldpwrFHfbOvnBqRF4M6uLH4BsapeQX5OO9WB1i4OBwd4MXvkXxdVqFF1czpnBP5/Srho1IyXi4UKBTXdAsX3M+9GL13KgHIqdwqrvVe39z0i3i2xoui0qAuVE3xqgkSqk51bHI9feKY+je3CdsfHb+GV+nkYc8A3/z6Fz3gyiCxVlniM3X00IL8uref0eN0L2K0x6ekdb0T4S1r6VBJE+HHuniu7mFBwCfVOjl80TgHIdnmPVeY8An497C9s8QeTgy75c531McDQgNiZPBQGOHyh3O87jo31WvS7CPcltuNjAovH1yraljfaioUbx2Yub5Ys2qYOqihVkm9qFcLIqm+8kWxfRMsDRqGqRQoWdN3cN+imc4DWsIA+EKXmZnY1ooxfb2pu3dYEGhJ7C5rFFMZrV3dWOyHCF5w8cmCGlmpt1UBls0JRNWw3KBwPX6OWbmPF1zNR6qKcUwZdzmINbzCVTdbxWVzpx36AF0nwA1rFikOAwIxaPk7UDLPAJrPxoATDcT9jD4WFOGLSo4tsEjx5CPNVSWxo8mVfiyWMkZsAQAD/yx5tscy4A8lUrZhkCIzYKcaYxNCu8GlthmjILFruPlLAY76uKR6MZJLq 1oOiABTa s5w57J7/AK5BX/KdKBYMmicKHqUbARRk58mOCCx6n8CDvNpoJRB1SNu7E5LOIoNmNGDpJLxComle2XfAKDA8SHZRUBL33FTElK7WqhM0pqFSWMwUsK0dCukJY9wFZvUwrc3w53isq/vtZd33rOeCyaiYSiB5DT3mMGQDq4z+XUbbRPh30xy9sIVDtKjqSc+txjGXYwwDWNLVNrl3Yn5kVYIaIn/ewbxs01enliWMb2afRVltFgTGowtGDLOIe5V4fKm3vriqwRYSMzpTelhb3Mkv/FP2CNfTehKcWQBAE9ymFxcrehrb746pGnD4G4T5Cn3XaT+SOu4MM/a7OvEsN+RRDsvSayR6UWMiG/3HgSJhVFuQ0zmU0toTIz6L37z30MOO17/tE9YrGKXNRC7Lzpkaec3nrKaVvccv4Wjaf0KogriGtgFNduQX+LVJGi4ZTxq+7LuuTZyHX7zn36Thd/X+A0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the drm-msm_gem shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct msm_drm_private. Signed-off-by: Qi Zheng --- drivers/gpu/drm/msm/msm_drv.h | 2 +- drivers/gpu/drm/msm/msm_gem_shrinker.c | 25 ++++++++++++++----------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/msm/msm_drv.h b/drivers/gpu/drm/msm/msm_drv.h index e13a8cbd61c9..4f3ba55058cd 100644 --- a/drivers/gpu/drm/msm/msm_drv.h +++ b/drivers/gpu/drm/msm/msm_drv.h @@ -217,7 +217,7 @@ struct msm_drm_private { } vram; struct notifier_block vmap_notifier; - struct shrinker shrinker; + struct shrinker *shrinker; struct drm_atomic_state *pm_state; diff --git a/drivers/gpu/drm/msm/msm_gem_shrinker.c b/drivers/gpu/drm/msm/msm_gem_shrinker.c index f38296ad8743..db7582ae1f19 100644 --- a/drivers/gpu/drm/msm/msm_gem_shrinker.c +++ b/drivers/gpu/drm/msm/msm_gem_shrinker.c @@ -34,8 +34,7 @@ static bool can_block(struct shrink_control *sc) static unsigned long msm_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) { - struct msm_drm_private *priv = - container_of(shrinker, struct msm_drm_private, shrinker); + struct msm_drm_private *priv = shrinker->private_data; unsigned count = priv->lru.dontneed.count; if (can_swap()) @@ -100,8 +99,7 @@ active_evict(struct drm_gem_object *obj) static unsigned long msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { - struct msm_drm_private *priv = - container_of(shrinker, struct msm_drm_private, shrinker); + struct msm_drm_private *priv = shrinker->private_data; struct { struct drm_gem_lru *lru; bool (*shrink)(struct drm_gem_object *obj); @@ -151,7 +149,7 @@ msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan) int ret; fs_reclaim_acquire(GFP_KERNEL); - ret = msm_gem_shrinker_scan(&priv->shrinker, &sc); + ret = msm_gem_shrinker_scan(priv->shrinker, &sc); fs_reclaim_release(GFP_KERNEL); return ret; @@ -213,10 +211,15 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr) void msm_gem_shrinker_init(struct drm_device *dev) { struct msm_drm_private *priv = dev->dev_private; - priv->shrinker.count_objects = msm_gem_shrinker_count; - priv->shrinker.scan_objects = msm_gem_shrinker_scan; - priv->shrinker.seeks = DEFAULT_SEEKS; - WARN_ON(register_shrinker(&priv->shrinker, "drm-msm_gem")); + + priv->shrinker = shrinker_alloc_and_init(msm_gem_shrinker_count, + msm_gem_shrinker_scan, 0, + DEFAULT_SEEKS, 0, priv); + if (priv->shrinker && + register_shrinker(priv->shrinker, "drm-msm_gem")) { + shrinker_free(priv->shrinker); + WARN_ON(1); + } priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap; WARN_ON(register_vmap_purge_notifier(&priv->vmap_notifier)); @@ -232,8 +235,8 @@ void msm_gem_shrinker_cleanup(struct drm_device *dev) { struct msm_drm_private *priv = dev->dev_private; - if (priv->shrinker.nr_deferred) { + if (priv->shrinker->nr_deferred) { WARN_ON(unregister_vmap_purge_notifier(&priv->vmap_notifier)); - unregister_shrinker(&priv->shrinker); + unregister_and_free_shrinker(priv->shrinker); } } From patchwork Thu Jun 22 08:53:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10025EB64DA for ; Thu, 22 Jun 2023 08:54:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2DC58D0006; Thu, 22 Jun 2023 04:54:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DDC68D0001; Thu, 22 Jun 2023 04:54:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CE3B8D0006; Thu, 22 Jun 2023 04:54:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7CEB98D0001 for ; Thu, 22 Jun 2023 04:54:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5156340B1D for ; Thu, 22 Jun 2023 08:54:46 +0000 (UTC) X-FDA: 80929773372.16.5611B00 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 5F209A0010 for ; Thu, 22 Jun 2023 08:54:44 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bLkRMM1j; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424084; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TQaNNe8oXaUFeFpLguh+yca1iGAODo9pDL1sZw6pJSE=; b=PZ10biIzn40MRR/7ze4EyxlOXlfDyvrt+bahb0p6wl/FxWuStzDy2LObvvJmzFy+0IGpTv CDB+QPipr3ZPUIcskYowgbb50sreC+gt8cTmHDqRVjfa1kUsPWvIGao+HLod4PO2Y60BCF hno78iZXcDxTUBW+2qkwc5qMT8rZXr4= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bLkRMM1j; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424084; a=rsa-sha256; cv=none; b=dtNDKYZTiu8Vjqtfd4ZWzsxVLxWvgArEfiUqVjXAMf/89qGudgbY6PfcsWOYDedABxN1fq iCUFeuVQTnHj15s4hnnmaAI3kSQA012Ie4LuqS+EOBWEa0XVI5ziyjFXyUekN9cJTEDKCG q54XTa3IXuSPX+FJgfUGDv+Ikr1ZisA= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1b515ec39feso12619155ad.0 for ; Thu, 22 Jun 2023 01:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424083; x=1690016083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TQaNNe8oXaUFeFpLguh+yca1iGAODo9pDL1sZw6pJSE=; b=bLkRMM1jSdqXsEty9KLAFC9vFSq6Mb8s85EPujNxx1fG7cL+WhObUy7eJvjIIVVT27 Kiy5rsYxFdvQkjX/Hn7QfCF/H515sUldxTAV8THo6pKuerK5LdnjnrVU9ezaWAUUfrz8 rV0yX1ewF3SUu27Tytlc8UIGxdmoc2rf5KwQXmt4TN0xZENUrrBpDEpOBdYckFOGmW+/ CnWb/ghzLgegjrSPuJYmDsQVFRVYxO2XxU7X4C2jxwq5osf5N5i/+CYLyq1fNxhJsVCX q//piIl1OJNIHjgKTj8Fx/a4slkfZ5A+zdUvxvyRJBR/16JSIqSsZUTa9z9xdPFzC7nq BhVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424083; x=1690016083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TQaNNe8oXaUFeFpLguh+yca1iGAODo9pDL1sZw6pJSE=; b=YGjYEE7PIODV/JEsFqA5PCUkBFydfyHE3L3j/qPm2GAX4nHmryZ2wFZOX7LkDRA+hg xztULvHhdWvpkjqydErjbHaNlB6XZ8D63nDhMwPBIHtRCqJmHts/f6fv2yR1yivROi2g SxmniDDlg1v/VQt8MeBD9y4S/M9Qmsm/XHXq/hydBPVNbiBOiLET5E8VgxNAp3FP6bOY cjYehsLNzJDyynFDGa27vRjNATGK5SWOpo4sszp5pfxXAhtHA7QVU+gDOHSHmvfVKAeA r5UROrOWyytqBEbJA5BDD6fcvPpp4XJkBFJEWb7cMZkV3LdqhwdKmjFeUpW141aL8rvZ c8aA== X-Gm-Message-State: AC+VfDw5aXukhy+WDkQagIdtzsq7cF7M+2o4VDjZRphnfD3Ew/l6N83e yNWeZ3cDC2VkCL3HZRomPdM1rA== X-Google-Smtp-Source: ACHHUZ6gH/Pv0NlMjwBsqv6k+8Uik3ghEf6lWW7kovYDQCKB++KLEA1PTcfQABA8HOR66RClg4yAXA== X-Received: by 2002:a17:903:32c4:b0:1b0:3cda:6351 with SMTP id i4-20020a17090332c400b001b03cda6351mr21555341plr.0.1687424083229; Thu, 22 Jun 2023 01:54:43 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:42 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 05/29] drm/panfrost: dynamically allocate the drm-panfrost shrinker Date: Thu, 22 Jun 2023 16:53:11 +0800 Message-Id: <20230622085335.77010-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5F209A0010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mhym1wndk6ctkxu75qg5mr6ipsnjas3g X-HE-Tag: 1687424084-356487 X-HE-Meta: U2FsdGVkX19cMqr1E1zJVc76M5HdM4YXaqqrJZkptXyBV5kIu9y7Mfga7zK5buYnGvtqx/un8iad+PAw6WkN54YvfTjiiLTb3JSGb7Ig5Qbu3rhWnGHq8Dhce6HwJOyCzZwMvZQ6BjTgC2YLXn3ObGscyFFM2h1LLk+0ZtbGEj1qTq0BtvNcMqZlv0gm2rRIZd2/Wd10+1RGTWY5O0rZTwTciEn29HOIuqgk5gJysWwAFrptad116tQYhvV8EAto1wmouQblSdxrwZ2VM79ye7498RYwRW0lw/RZd/zc2ohlpo9T5ZK/jcRwBALQQYs6pxl5WMCbLBJ7mQBZhDkDfkdgR00lbgijX5mE5/7vM3qD5TjZPWnPdE+4VEPeNQGj32oehVXzYR13Of/UQEhU/Puk6PSpA7InujYAX+IP/F/HAzPrWSt5xe7CAzXN5/D1V6SmPW4Hxyg3rmOShQ5YDo8TVFjyh0B32kww2K6/FxxoPkF7ltguAVlpbNEUZMFm+IsLQE0T5ruduDSJaBERVlVNfdelUtSC0/HjzcLFKHBon9Q/36fEZNBrpG9gfA4Re2FkB2mO7caqGGms3sbbQV0ALfOFt5nAMwh4cUNe8o1vV6wO6zeEYI+7f1GzXBkWwDon9JohjvWF6ZwVbXk7sd+ZiOLEMqA3jGl55es9P040gumNTxr4MmHJRsvdHdvRrUsMoPx/70syaeAwsxEe8rdsBm34bfFSC7wFnE4PsOZrizwIU09Mrifwm2YWRWVpAmUctrNP2SowPjleoQ78zT6oo2ykucy2sPnRYz3t0jG+UdvI347f4siDOhtPrf/RycDOaeNoEwDn0R/Hqz3ub8Fb79fS2lXyaPiPdvnboFi83EhrlfrTdN1qVSReK0xlc99qjxKW0IfEKEb025/gana5C7UMTSauRXtuywTIZcvLxK0hW9A/T8LPq6PM2exVEUEikSN5FwbK3qUu4LN rONcEaLj Nqwm2X8v0O0bK5pIDcAW6ptm4Du/KgM1QgDXfJJOdOwB1s0wJe/rjAqfguVqHO7gxd8TXcZRi19A2FiMlo5bqdIec8BUU/yc122jrou0wwztoW/b2mb0eSIWWOpUizfhsEgsAVOJW/SBvkl+i9dSDwo64B8Upi5y+SHFj0oiaEKJ9XQDLejdvl4+M1t7YNJvlRD1ktH9A5gENZoTct6Gb7FNvc66trWoZAXKAnd8CnPdYFRfvZthstgomQ850bs3veCVu3mrucogO21YZI9Knn+YpjbhbRvc0+V+rfF/K1vFROu9tPdtf5HIIFsuKqrmIgAjnndYTfPAbkCgaJYyWVcVgkFfOnYe0yc4clk3OuFAT9MdyeZh3IRE7RMcMsOkxmQVOI5hfPcKzcc5hlFFS302ApcpGoi2DfTkc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the drm-panfrost shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct panfrost_device. Signed-off-by: Qi Zheng --- drivers/gpu/drm/panfrost/panfrost_device.h | 2 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 24 ++++++++++--------- 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h b/drivers/gpu/drm/panfrost/panfrost_device.h index b0126b9fbadc..e667e5689353 100644 --- a/drivers/gpu/drm/panfrost/panfrost_device.h +++ b/drivers/gpu/drm/panfrost/panfrost_device.h @@ -118,7 +118,7 @@ struct panfrost_device { struct mutex shrinker_lock; struct list_head shrinker_list; - struct shrinker shrinker; + struct shrinker *shrinker; struct panfrost_devfreq pfdevfreq; }; diff --git a/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c b/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c index bf0170782f25..2a5513eb9e1f 100644 --- a/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +++ b/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c @@ -18,8 +18,7 @@ static unsigned long panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) { - struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); + struct panfrost_device *pfdev = shrinker->private_data; struct drm_gem_shmem_object *shmem; unsigned long count = 0; @@ -65,8 +64,7 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj) static unsigned long panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { - struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); + struct panfrost_device *pfdev = shrinker->private_data; struct drm_gem_shmem_object *shmem, *tmp; unsigned long freed = 0; @@ -100,10 +98,15 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) void panfrost_gem_shrinker_init(struct drm_device *dev) { struct panfrost_device *pfdev = dev->dev_private; - pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; - pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; - pfdev->shrinker.seeks = DEFAULT_SEEKS; - WARN_ON(register_shrinker(&pfdev->shrinker, "drm-panfrost")); + + pfdev->shrinker = shrinker_alloc_and_init(panfrost_gem_shrinker_count, + panfrost_gem_shrinker_scan, 0, + DEFAULT_SEEKS, 0, pfdev); + if (pfdev->shrinker && + register_shrinker(pfdev->shrinker, "drm-panfrost")) { + shrinker_free(pfdev->shrinker); + WARN_ON(1); + } } /** @@ -116,7 +119,6 @@ void panfrost_gem_shrinker_cleanup(struct drm_device *dev) { struct panfrost_device *pfdev = dev->dev_private; - if (pfdev->shrinker.nr_deferred) { - unregister_shrinker(&pfdev->shrinker); - } + if (pfdev->shrinker->nr_deferred) + unregister_and_free_shrinker(pfdev->shrinker); } From patchwork Thu Jun 22 08:53:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12C3EEB64D8 for ; Thu, 22 Jun 2023 08:54:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A07558D0007; Thu, 22 Jun 2023 04:54:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6C18D0001; Thu, 22 Jun 2023 04:54:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87EF58D0007; Thu, 22 Jun 2023 04:54:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 778248D0001 for ; Thu, 22 Jun 2023 04:54:54 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 507831C8CC3 for ; Thu, 22 Jun 2023 08:54:54 +0000 (UTC) X-FDA: 80929773708.25.9E1BC5A Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf09.hostedemail.com (Postfix) with ESMTP id 6DFC3140019 for ; Thu, 22 Jun 2023 08:54:52 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VsUY2aig; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424092; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=161EjO+w9TmzA/a8qJytM8fXp/e7+Gx/R5PIlDH2uAk=; b=uyU80OAs30qpLj2p8MW5lt89Q9ue+dvIwZs0HMO+oC20409OWAXs3NEIdibpbQDyFmeW0s o0YfgbZGlDTrDyubsNZCJe1UxlAWVHO6KN/Fejgq96rgVxy4sK97Kl1N5TI/kpK3mRgHSs Z7NO8XkaY+sRVHSsPP3XwyF1k8EJ5XY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VsUY2aig; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424092; a=rsa-sha256; cv=none; b=BQVb1yHNKJxCCJQhS0lK3gBfKo2ATrxd34XrlW0JXHoeVZBmWsSUsb0fGYGHrS+MeS6e0H syIg/RWLF5gdDz9cY7ddEr+XIM397YTDsSENK+cEW+K59XWr/8DG4lgGeJrPYF+ltoDBLU QdDTJ3rHlPfh/dNieGUw14claEcBiX8= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6664ac3be47so1328444b3a.0 for ; Thu, 22 Jun 2023 01:54:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424091; x=1690016091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=161EjO+w9TmzA/a8qJytM8fXp/e7+Gx/R5PIlDH2uAk=; b=VsUY2aigV4ZcD/zqH6R6qf5YjeDjgcBdr5DWNiFL+xj8FWQJXfxqjN+JjtQQqPUTZU y/hvwjl2otPmZYcL1mDMWeSCzXx77cZKx3CoOlix2I89OTP/XqbmazxkdkRcxT8IoFUn 2QEonOktxP7xKSbe3Xt0JkWeJ0vXY7bYmU0ntULjSwRJaXbaN1TAoEufMFnqYH3EWbZ2 epggxRJm4wrl127Vz5WoCbm7/HM2e7h8o7U+VP/mEnC0Icgf0KIm3PI/NPbIYz3ozq3C dpiJUcJ4yXe7UOxEIQJgQOohxTSbyOV1lnJIMgAbmswsZtfNIYimqUElkeBCjB3tvm0/ 8Aig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424091; x=1690016091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=161EjO+w9TmzA/a8qJytM8fXp/e7+Gx/R5PIlDH2uAk=; b=EEWUeho78nqhQm0Wxat4VkHLVgxxz75eLjF7PwwPpmxRYHvIkDcTKV4xWdFQLd1JRG CfDPOcpnPC88ICcTqht2A5C+hgKorjVPdvJgaWvzN0LWQNzL+/FpkOBVc7SCBiLfHjDn uG0Ds2BKNqHeAL8x9EXpk9zS8zFWgQ7jR/YJShKt7R4vMuxrsQlbCd+NuGEom1VExbN9 OTNMwX5/cfdhY952wRwS2NglQAU3bbQTnly/bFeDKMhgNN5XpHOSUnBl0k1ITWEbeS5D L+886kfUCMzLDqws6EOPXw2WkG6unWxv37w2W+ybXaC74BPkppOSLI66y1bPVMPAhPr5 qJ8Q== X-Gm-Message-State: AC+VfDxL4EOH7pB1OkC0JYMHQ/NFpAzA6Psg1uQDjOrRPuD7XATXv3G5 k6hI6D2Ef8mmeVu2WkcYCj2N0w== X-Google-Smtp-Source: ACHHUZ6Zp8ahuhiEYJqMKGH6XvbaL5lpWamc0XC/4KRVYsGWUu1w4+GFAcXsiV5/9t8MrDqgvh99eg== X-Received: by 2002:a17:902:ea01:b0:1b3:e842:40a7 with SMTP id s1-20020a170902ea0100b001b3e84240a7mr20959635plg.1.1687424091188; Thu, 22 Jun 2023 01:54:51 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:50 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 06/29] dm: dynamically allocate the dm-bufio shrinker Date: Thu, 22 Jun 2023 16:53:12 +0800 Message-Id: <20230622085335.77010-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6DFC3140019 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uangt6bq5n5116os5g4axb68yr8eqqdn X-HE-Tag: 1687424092-945829 X-HE-Meta: U2FsdGVkX1+i34A4IPivhtlUZOhLie0+UaJbxMIGquEnwquLctymG8LwaMYzlXyLamBqeLzUTLzG68JQpgJ22crl9/Ahqg2etBm6K60F6XhTBNtUqT80fLP8hSHYVYV/ISxHWgNIzuB3MH3gDYfQ+HxiB1LHxk0NNFqhR14hCm0GcPk/nSheSfWNDSAL5wCqy4QjHOJKqngRDNxYhbGEKlGeg5EQmZquAgrMCt7vP2lpiTAKYPu0SUlStgZL/hPg3vcBWtMNFo1FT6aO8xpKPUj1OEw1KPWathks2FRpLd/RH+0y9S29i6IVUy/9CE8OsPDbchHPyvxGGj3AMdJtAxuthCU8b+po8nL72SXBOhX3YkuPPrddZz3cNNWbXmxljx8nD5mrcWSgK7Cl3414SaqWSQhABR8TFLDjTuZK7bb/OuPema4zJZc/rDAz8YaoeogV5WMG5giJM++VCvQWdJNwydu6goTmC5y9dFkDR9iTXa1lpcZPKp4cJJFSbwCDtwhR+MYbvu2YHN4kx0ySTzsBj3GulsBnpbJLASlKDkfUfHCeDCwGpcUXlCOm+r6Ye09XaE+vep0iNnsXQwkke9Boa985YUy6WY4YTOhhxFcuKBifA/YmE7h3Bu9phk1NUUDJOYWzvfriqfcv6u0t9WISqTmJnoUO6hCQNKreILIZnx5NEEHTiNrs2GEP8GyHTdO55PbvquZp18tb+mqUpuI+X6vrFRbiSJvnFHxm8zGxKZFQR1p1yLyAnpEztjvHfyVBZAa5c0t4nVehRU8TWXk7LsSJbuSoUufAIPAB17kS7IVeevmA2IyOOnIjfdvzrmM4sq4o4cRbHyrhWxWP4RN4lYQZP2mpjQ3TUnFxLjk7OkC+sjq+iCJhlS4pD3tiBr0PjjEOxkgzX1NaNKPhWaqY2DmUl5EQShFMbTBnM0w0BUTaQqLh2Cq1TlpQcwOnmMUWxMjKVd/8Q/LzDCj To3O0kyq NogQMRIiHVm2+uCw4g3kxtMi6BZ7yXOUGoz671aSqEE0nAeuouOfl2I0w2G+6tdO5/pw5UR4INQ7xgfhm4m+shoOSFiylotfpqIzFd9yn2zYU6UXpwwq6Dtx15jQuTvPcj+PUd3ppXM8B+07LATvp50wy4yCN3d5QAOZIFGAPWDb3vgnGJW8/oqxyihF0SKXKIqLzDu9ecU5hODcgVHcbAZ3xDTGEr/FH0tYGidegCkj18ljJZqTn9PrSXXjqE1SvLeU95d+J7YWsmhkYbNo6a1CZHHUP5ISj+3sJYH6CLXxbORhcORqjd/7njF+zYlrRp4C02MQQPRFtqroh/l17Z8WY/Eg9WfI4siwURw6GzV/GbAOagTcySC8HEvquut2k97tRM5rsYY5C9xsuZfKaTYCbWQI+wiBTpCQT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the dm-bufio shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct dm_bufio_client. Signed-off-by: Qi Zheng --- drivers/md/dm-bufio.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index eea977662e81..9472470d456d 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -963,7 +963,7 @@ struct dm_bufio_client { sector_t start; - struct shrinker shrinker; + struct shrinker *shrinker; struct work_struct shrink_work; atomic_long_t need_shrink; @@ -2385,7 +2385,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink { struct dm_bufio_client *c; - c = container_of(shrink, struct dm_bufio_client, shrinker); + c = shrink->private_data; atomic_long_add(sc->nr_to_scan, &c->need_shrink); queue_work(dm_bufio_wq, &c->shrink_work); @@ -2394,7 +2394,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { - struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker); + struct dm_bufio_client *c = shrink->private_data; unsigned long count = cache_total(&c->cache); unsigned long retain_target = get_retain_buffers(c); unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink); @@ -2507,14 +2507,15 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign INIT_WORK(&c->shrink_work, shrink_work); atomic_long_set(&c->need_shrink, 0); - c->shrinker.count_objects = dm_bufio_shrink_count; - c->shrinker.scan_objects = dm_bufio_shrink_scan; - c->shrinker.seeks = 1; - c->shrinker.batch = 0; - r = register_shrinker(&c->shrinker, "dm-bufio:(%u:%u)", + c->shrinker = shrinker_alloc_and_init(dm_bufio_shrink_count, + dm_bufio_shrink_scan, 0, 1, 0, c); + if (!c->shrinker) + goto bad; + + r = register_shrinker(c->shrinker, "dm-bufio:(%u:%u)", MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev)); if (r) - goto bad; + goto bad_shrinker; mutex_lock(&dm_bufio_clients_lock); dm_bufio_client_count++; @@ -2524,6 +2525,8 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign return c; +bad_shrinker: + shrinker_free(c->shrinker); bad: while (!list_empty(&c->reserved_buffers)) { struct dm_buffer *b = list_to_buffer(c->reserved_buffers.next); @@ -2554,7 +2557,7 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c) drop_buffers(c); - unregister_shrinker(&c->shrinker); + unregister_and_free_shrinker(c->shrinker); flush_work(&c->shrink_work); mutex_lock(&dm_bufio_clients_lock); From patchwork Thu Jun 22 08:53:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5716C0015E for ; Thu, 22 Jun 2023 08:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C4CC8D0005; Thu, 22 Jun 2023 04:55:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3753B8D0001; Thu, 22 Jun 2023 04:55:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23DCF8D0005; Thu, 22 Jun 2023 04:55:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 151E98D0001 for ; Thu, 22 Jun 2023 04:55:02 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E636A12010E for ; Thu, 22 Jun 2023 08:55:01 +0000 (UTC) X-FDA: 80929774002.05.7C17C61 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf21.hostedemail.com (Postfix) with ESMTP id 1DE211C0010 for ; Thu, 22 Jun 2023 08:54:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=E+tFDKMS; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424100; a=rsa-sha256; cv=none; b=XDBOfINTtbtW2cCHkNvdi71abPlBUZb0qW1rEdECqyoNVvBZW9bfI3ycVkbnR/bYrrVqOo 5aM79Bbxj5cbCA4RF4qzVLaHO1MSqcvAyYu2KlM9sFu4yJXWI/wsVen3bGuZpUpeh74vtV 04Jt3YcepQ6vflTu8b7NGQDCPnyF1+8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=E+tFDKMS; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8pXD3DmjPUfWk2+jjA+m+4zKKy0fC+8AwUsZhq2lRmk=; b=k+G6dsxc6ZE3GFRDp1YCWvqMdnbf92QaRgx9kO2OZWlnNRe/OyNPwgXTxStXd9wgBTw4Ap swS4tz0MkpU68vimHg1IErzh4LlNo2EBrqusy4/V4uhuNhXxE9izy6frMnfjLHzNclXOEo 2pDwHxMagx6ZbIuMqKf3RsQsu9RIYiE= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1b5585e84b4so6827195ad.0 for ; Thu, 22 Jun 2023 01:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424099; x=1690016099; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8pXD3DmjPUfWk2+jjA+m+4zKKy0fC+8AwUsZhq2lRmk=; b=E+tFDKMSki037PBhfmhvWVloKXWuzU3Ri56xXQW9hCkunzqr3/MhlEZSqiNZ+tNI9q wXEuj2hNk0ysqFsBunC7adB4hsOSpWoOkzyNHyQickBfIH13Ob7zUzdQVkoowJm+gwHf GqLtSyQYvEY7J6FdTyYJ1xVnKRHg63OezBYT/3UycGb6EQo3xYiC+CTrGWJfn3US3UhJ dbGuvQxkFI38s4idyaP5iHfWFNKc7ZWCaOV72DK+NmzkQO6lFrqKljjPnoN3sVHURF0C M8AH8Js6rxjyGoBC/sNAUYyKjN9tqWo3Nk1SQH9TmJx7zV/vF6Qb8YX+sV5g8XxbHf9I nLxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424099; x=1690016099; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8pXD3DmjPUfWk2+jjA+m+4zKKy0fC+8AwUsZhq2lRmk=; b=Ds2AcdULCpy4cWBsQSSrttbfMnDQpdKPFYF1pLJ78ehfzAe3BQSxtFGojg5yVN/Ifk ymLn1XOyMDs/J8bSFAvP1zT7KLV9NHZ/9ONEKwmrY6lBOb+whehczr0MqzzibcPMS2js Vm1qmne2aEV/GtaMEddAhGuOx/G5mxdBDDAbxa37eXOvgxVtrqe4iElyRHVCOriluAkk mpSWflZfCaViePidrqjau4Tt/MZAIUWJe54JUcRLVReb67vX+JHe4RLd423EpRUUttpA qwzGken73W0tPyLJz325GzbdU1b/nyYVM8fqKC9YJxFMKfQb2xcxckhb5LUTizloDyBk kpfw== X-Gm-Message-State: AC+VfDzZtyNHFiJS9YHRjMagosN9Ikh9eXjkQdZWkWdnvkfZEEKyCiVX 5rWS3lB33qqBcyAB3Ctf9gGyFA== X-Google-Smtp-Source: ACHHUZ6uW4WhSYRgUY2aQCit4ogoH8R/r57rghu14Eyu6d6WG0AvLb+rcmWHNr1Voea08aO5tIYo3Q== X-Received: by 2002:a17:902:ecc6:b0:1ae:1364:6086 with SMTP id a6-20020a170902ecc600b001ae13646086mr21600250plh.2.1687424099018; Thu, 22 Jun 2023 01:54:59 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:54:58 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 07/29] dm zoned: dynamically allocate the dm-zoned-meta shrinker Date: Thu, 22 Jun 2023 16:53:13 +0800 Message-Id: <20230622085335.77010-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 1DE211C0010 X-Stat-Signature: 585nxf63xtwb8f4y5sa5643gykzryb9d X-Rspam-User: X-HE-Tag: 1687424099-785984 X-HE-Meta: U2FsdGVkX1+IXODUVS4R9xB5u1XsuBBykxzT4ufuo5woolnEO082Fc0ABtTPwrbc/1VLgDTCJsEeRtGcOH47QAPA/zBj/AsjE74Dc2LBWwJvO9nubSJRLXV7kYNNkUphg3L5W4p5Sou1cb9htftu+C8KLZelBATdwGhczwYAXAeu9Teq4oxBIRVXLf8jzhzOh2q1zh9u3G+ayHigy3to1q+rTlv+uuWyjd1KEgPG/Q6feTUlYlpK16cy1TmgcWV+EMo3JEMb68R9dkeX36SZT00EVzQXIt5AgREiNvIkkaDRz2DE7q81bGBGIK44Xiz9iAOqc2toVmb+TWn996FzgWbYLDBMqqJ1NarEKUZDdE4YQdb8WmSv8AoLZ3VeNKhgMUNGwMn8Am4Bc7+xjJ/gALHPlFVU+monXA1A6E3l7xocK10M7YMBJDUvsP551IvUiYfzoXg2hJ+LPmgf//M3mazhzb1yimlxsz49KNrUv7fp8nHLOHOfMUbgohLTDZu6qcTaN2GDee9/2HvlD+U0VzRIOLqtnpng6+OHNXa2jD9zMavpIewFl7KHC2P7NmW48wiaI27HED7b7/wFrP/g8Xga00PQH4ULtm4GhOoK1RGBIL8sZuznX/yF+QkrYafqqhnh8VGvaOpmsPwLvB7IW/LUa03erGqpjdUo7I5hoTqamsePXC8zS5+P1drlAF6oJA3iMIC9Qn9/noE6UfgiPk7l0eXCaeZWP7qOrLPNNNFOhpA2bNbWb8IfHKKSvshxEbCqo7OAIRAe10i1dLGefEv5PvBc2h8GM8koHyS3sTJ4+PXN7gXfR4RrwPNb1oVgeoEjtrRSoyd0JMyc+RrrtDHrVmL1DLVliGqlkCrvujicG5lHbAGihTIHxUByujGXM8M4vgDSo6EERAWblQSF6vbWF2wid9+CMuzcQagjLYYtDLbXfg0ffeC4rKzDXDW+FA+5cSekrsaLWEmF3Ki YlulD2dC jtofaMVD5TfuOXpJQHiZfdRtNAT6deQT6TIsC64sr0geY98xXoZA4q4PtGl7sRUkU9vA/+XI23tojP679FYGRBSvQaFYF9o5IKpwtE76HTRr16Gsb0OIdAZwxq0zKvKbVgijIkuQVEKIHLNVHu/NZQGZi3BOwqiHeR65Sk3VIgr8Cbfwz7aUD2Nah7+5p6wvTykVL8eLKye9TyGncefs98jj8eHbw1VQt4pP0XfcrOq1tk666InAGpUfUsgaGwGO+wvBa7zMO97EM8jl529wewyeI4gv6y8bGYrSsXqrG4YH0O1GBJKk+F3ERXBN2xEYqQGMkIwpMq7Q+kA8CBoMzJ+pPcThlU1kjMaClfxSPjCQEmgvviP32sk81BYrn4e6Wg9pnw9NK6Kl6AJij25982eu+bWmgbGpS+ZUG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the dm-zoned-meta shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct dmz_metadata. Signed-off-by: Qi Zheng --- drivers/md/dm-zoned-metadata.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/md/dm-zoned-metadata.c b/drivers/md/dm-zoned-metadata.c index 9d3cca8e3dc9..41b10ffb968a 100644 --- a/drivers/md/dm-zoned-metadata.c +++ b/drivers/md/dm-zoned-metadata.c @@ -187,7 +187,7 @@ struct dmz_metadata { struct rb_root mblk_rbtree; struct list_head mblk_lru_list; struct list_head mblk_dirty_list; - struct shrinker mblk_shrinker; + struct shrinker *mblk_shrinker; /* Zone allocation management */ struct mutex map_lock; @@ -615,7 +615,7 @@ static unsigned long dmz_shrink_mblock_cache(struct dmz_metadata *zmd, static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, struct shrink_control *sc) { - struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); + struct dmz_metadata *zmd = shrink->private_data; return atomic_read(&zmd->nr_mblks); } @@ -626,7 +626,7 @@ static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, static unsigned long dmz_mblock_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); + struct dmz_metadata *zmd = shrink->private_data; unsigned long count; spin_lock(&zmd->mblk_lock); @@ -2936,17 +2936,22 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, */ zmd->min_nr_mblks = 2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks * 16; zmd->max_nr_mblks = zmd->min_nr_mblks + 512; - zmd->mblk_shrinker.count_objects = dmz_mblock_shrinker_count; - zmd->mblk_shrinker.scan_objects = dmz_mblock_shrinker_scan; - zmd->mblk_shrinker.seeks = DEFAULT_SEEKS; + + zmd->mblk_shrinker = shrinker_alloc_and_init(dmz_mblock_shrinker_count, + dmz_mblock_shrinker_scan, + 0, DEFAULT_SEEKS, 0, zmd); + if (!zmd->mblk_shrinker) { + dmz_zmd_err(zmd, "allocate metadata cache shrinker failed"); + goto err; + } /* Metadata cache shrinker */ - ret = register_shrinker(&zmd->mblk_shrinker, "dm-zoned-meta:(%u:%u)", + ret = register_shrinker(zmd->mblk_shrinker, "dm-zoned-meta:(%u:%u)", MAJOR(dev->bdev->bd_dev), MINOR(dev->bdev->bd_dev)); if (ret) { dmz_zmd_err(zmd, "Register metadata cache shrinker failed"); - goto err; + goto err_shrinker; } dmz_zmd_info(zmd, "DM-Zoned metadata version %d", zmd->sb_version); @@ -2982,6 +2987,8 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, *metadata = zmd; return 0; +err_shrinker: + shrinker_free(zmd->mblk_shrinker); err: dmz_cleanup_metadata(zmd); kfree(zmd); @@ -2995,7 +3002,7 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, */ void dmz_dtr_metadata(struct dmz_metadata *zmd) { - unregister_shrinker(&zmd->mblk_shrinker); + unregister_and_free_shrinker(zmd->mblk_shrinker); dmz_cleanup_metadata(zmd); kfree(zmd); } From patchwork Thu Jun 22 08:53:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D246EB64D8 for ; Thu, 22 Jun 2023 08:55:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DEED8D0006; Thu, 22 Jun 2023 04:55:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B6E48D0001; Thu, 22 Jun 2023 04:55:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC06A8D0006; Thu, 22 Jun 2023 04:55:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D98F08D0001 for ; Thu, 22 Jun 2023 04:55:10 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AE16212010E for ; Thu, 22 Jun 2023 08:55:10 +0000 (UTC) X-FDA: 80929774380.08.A58575F Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf25.hostedemail.com (Postfix) with ESMTP id ECA31A0010 for ; Thu, 22 Jun 2023 08:55:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=dSY63dBG; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424108; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VJl14vOz0XQFz+azMteX8g8XWQLmhD27loPyzQxCyWo=; b=ruBJd9Sevd225DCVHR+02DzcS5aHDPH+beVGhPPjK65e8MaCMJSgoAtzVOLpR5248XS3Wu EWIBC28h3x+j5Lb3SHIauea5GKL3npaV8Ai6x0EGK9h3UGd8MvFGmGpKdSKg5IGHdpPBhe hLXLk75nOi8iPoBxDcSFVOtika8iwug= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=dSY63dBG; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424108; a=rsa-sha256; cv=none; b=vuQ9XrnCQkBbNP55k4dIGw2x4PPle7T93SOXe2R2YlcN6RHjyVHGikkZuM+V89uNU/7ree om5TYtqhv9ROejBfaHT2Nt8dd2gW0phM9gZSd0kRQQX1EdW2rH/SnHifsfLWtETUooMWlg LIDsVnDSS+IjgT6MliD/SN9qkJXcyXk= Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-54fd4a7ce25so1096066a12.0 for ; Thu, 22 Jun 2023 01:55:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424107; x=1690016107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VJl14vOz0XQFz+azMteX8g8XWQLmhD27loPyzQxCyWo=; b=dSY63dBGkLY8Soj8dXVS98dEuYYMZ3kYNqwn6f5//glPXLQFLJz27NFKmb5CQWTM2k APxbCFally5Ye2wcKhRjXEzkXpY2MMujAr7sHxpWVpY6Tw+UW8TdBughq+A6CDzAWoWy +AmwZXYTUtj1EVbssFnJPqE+UVXszSk7I45NYzxDje6ZfgHjeGWZHMg6aUv73LqWwWwW 5itKACc99wkxtDt6xsJWBL2urtctKzIlWYA5VnKnCf1Z8lrwitknq6fV+Y7yJzzApX1O ankK3WiQYH84OGQwTOkmZRg+avhtQPcT8KVvO5N8KslMNgzr0qkBoxHulDKJPx12AE+X /x4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424107; x=1690016107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJl14vOz0XQFz+azMteX8g8XWQLmhD27loPyzQxCyWo=; b=hXhUwE+FDaO5Mzc/6VDpy2VFziaMPTNL0rCrAXRZqt4v/C/dAnmo2xkNLf/GnT7P8r rt0h8imKlNQ8omBmjaOTDLuTYYQu2vJR6Le/K/RsaFURtW8Wy6JQQ+suHEZDGn+bHp8M uJa1yxz15HBCy1XQ6qUK64uJ1XjPKDj+/6kc/e+NVhCI+iAiej+GVoEBjBipMefeW3dC YuuarPLQCRwlezjjtxvnIf++94EtuFzRULQnzZEaOA0f14HHzS9xOrmhzcTVdXe2vZbl fVcHoMzunnr5O9F8K+zh7BVnW3KhFBnqIYMljGst8Bpd0Nc3f9TpEwCGEk8feOqJW8EJ kpng== X-Gm-Message-State: AC+VfDzKGGOowr1PjFPfdUJRlC/fiOg5JFXrOqkKN3UL+8eS9ekLPfXR dH7P6t/KeXMXMBZiYmfI7NtIrA== X-Google-Smtp-Source: ACHHUZ4GugeqE/IaG8xLwh4Bzf5woebFV8d7i9CEhVh9D6pBJJ8ySrAWkjhYmHqxBxnT9m8efiqyXQ== X-Received: by 2002:a17:902:c945:b0:1ac:40f7:8b5a with SMTP id i5-20020a170902c94500b001ac40f78b5amr21000858pla.3.1687424106926; Thu, 22 Jun 2023 01:55:06 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.54.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:06 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 08/29] md/raid5: dynamically allocate the md-raid5 shrinker Date: Thu, 22 Jun 2023 16:53:14 +0800 Message-Id: <20230622085335.77010-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: ECA31A0010 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: gt3kpifnktegfogwmn46g3an8qzsgpup X-HE-Tag: 1687424107-314282 X-HE-Meta: U2FsdGVkX1/GYucdrDnCJJpZVynw/PxGdmE7UgFWsYYunzbA2tZaXdsGKIwqkkuNR5lvhu2i/2E1O/eraWDtptHlkd+UFuC+ZC1hgpreh87oNdljTukbKiJCiZvEBnDIq8JTuMfKE+mxFxmvH6+1HIWSOSP74du18TOiwLBTECOfBNS/AOnYn05hh8EKVe7/0Tvs1PfLnTmF5agXk9nUOyVasroJtic83s/wtiyjNx0lAi5NORVOucaOkCjAkhb19VOxcXZ0DS2Uix+MHS/yT8TuiqxbBO1tT1nO6Fp8kE69lLOgQJ7Mlm29azf3QMuTaLr5j8SBwbiDgIx1Ewj2uT569IvfB65iTrH1h1KRpEDzMuWfWxUewACIXQ7O0WGQyPjQ3y/cyK3OsvpKyAh/xyZCHmx1F5SzWngdwhG5RYMgFqNUw7DnWNIXDYvyGEWLMmpxH48D/rNAC+C339Vuhq314y21J3bj/qmtCWy3L/ROKWR1hH/pEe/8R7m+e8T2NrVDDD3J5l+k1/rFdJcYJd8LLbCwsrWLK/XXeNtnBrqz2swqnCuNdKcLfJ9FX+u83XrY6/3UTXPRbFk/xrrfXqKoRhj95iUBM45qVWbkVW0ArpiY6LoA5KPBHKWKvgz4CE1x+k1op4yTXNazWC67bU+HLZILc2vdXhSNQMAnN2v5OfDjJvRCB64F5uEfEx+rOStP/e8h5hkNDHEapS5Z4oYFC8cWChI2eL8hrCtKCncN/XEa1+6VdTCT5kQkM4ja1v+TT9g0bhaNmyuOIf9x3NOxx+CMpmnTQmKUSgIsyJhzzoHOrRWWu9hZD4qo3QhfdCl2Z+vBlPYAEBAhGMoXLDzb42y9v+Gc6n2YvqKJe8TLoACQ3OqaNOXE2rBVxGJoTRpVAg8CTLa+NXpxGrUcrZJea32nqFMJzES8YVs3u3rcYuzCxnmxr5h08EUBLHZl7CDNWTLsM5z6bhdgHWp dKBmz0VB j2JurCjX3HlaECniJMFOGOaW/7NYYBGqGaBOyF7+VVE+Hjm7ABG5pK1A5sWpIXC5shrg8wuJUHEwtVtQVhxda34VYZce8gR0MZ+mKH+cEuDZhFl32YX2nf7K+Huk4a5QoyU42Ary35rExYkMQJbffdxtVvIq/F+/8tJvI1VYdIXlBpX29Vkqaww3iKtiLMBw8BlCa/0w4MZYvyTDa28HktZDf2Dnq+PmK37GyrKylM0ZcBK2AVbzUbYU0+3Kzvxm3UMHlKZZ6n6hcXZ1W2pF0RBL8FunCp2yu4Ig+PLlBL1f/+uCz42vlaWzwy/Id8fFb0K009IFGi0h+QxBlv6X6xCGnDhxBh/CJqTe/6tLvyoDUF4A/yBocYMENXGfJN45AEeEDwn2J0ql148Q8GE3mYxnMzJ2N3s2R4oqG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the md-raid5 shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct r5conf. Signed-off-by: Qi Zheng --- drivers/md/raid5.c | 28 +++++++++++++++++----------- drivers/md/raid5.h | 2 +- 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index f4eea1bbbeaf..4866cad1ad62 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -7391,7 +7391,7 @@ static void free_conf(struct r5conf *conf) log_exit(conf); - unregister_shrinker(&conf->shrinker); + unregister_and_free_shrinker(conf->shrinker); free_thread_groups(conf); shrink_stripes(conf); raid5_free_percpu(conf); @@ -7439,7 +7439,7 @@ static int raid5_alloc_percpu(struct r5conf *conf) static unsigned long raid5_cache_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); + struct r5conf *conf = shrink->private_data; unsigned long ret = SHRINK_STOP; if (mutex_trylock(&conf->cache_size_mutex)) { @@ -7460,7 +7460,7 @@ static unsigned long raid5_cache_scan(struct shrinker *shrink, static unsigned long raid5_cache_count(struct shrinker *shrink, struct shrink_control *sc) { - struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); + struct r5conf *conf = shrink->private_data; if (conf->max_nr_stripes < conf->min_nr_stripes) /* unlikely, but not impossible */ @@ -7695,16 +7695,21 @@ static struct r5conf *setup_conf(struct mddev *mddev) * it reduces the queue depth and so can hurt throughput. * So set it rather large, scaled by number of devices. */ - conf->shrinker.seeks = DEFAULT_SEEKS * conf->raid_disks * 4; - conf->shrinker.scan_objects = raid5_cache_scan; - conf->shrinker.count_objects = raid5_cache_count; - conf->shrinker.batch = 128; - conf->shrinker.flags = 0; - ret = register_shrinker(&conf->shrinker, "md-raid5:%s", mdname(mddev)); + conf->shrinker = shrinker_alloc_and_init(raid5_cache_count, + raid5_cache_scan, 128, + DEFAULT_SEEKS * conf->raid_disks * 4, + 0, conf); + if (!conf->shrinker) { + pr_warn("md/raid:%s: couldn't allocate shrinker.\n", + mdname(mddev)); + goto abort; + } + + ret = register_shrinker(conf->shrinker, "md-raid5:%s", mdname(mddev)); if (ret) { pr_warn("md/raid:%s: couldn't register shrinker.\n", mdname(mddev)); - goto abort; + goto abort_shrinker; } sprintf(pers_name, "raid%d", mddev->new_level); @@ -7717,7 +7722,8 @@ static struct r5conf *setup_conf(struct mddev *mddev) } return conf; - +abort_shrinker: + shrinker_free(conf->shrinker); abort: if (conf) free_conf(conf); diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h index 6a92fafb0748..806f84681599 100644 --- a/drivers/md/raid5.h +++ b/drivers/md/raid5.h @@ -670,7 +670,7 @@ struct r5conf { wait_queue_head_t wait_for_stripe; wait_queue_head_t wait_for_overlap; unsigned long cache_state; - struct shrinker shrinker; + struct shrinker *shrinker; int pool_size; /* number of disks in stripeheads in pool */ spinlock_t device_lock; struct disk_info *disks; From patchwork Thu Jun 22 08:53:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76913EB64DA for ; Thu, 22 Jun 2023 08:55:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A2428D0007; Thu, 22 Jun 2023 04:55:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 152918D0001; Thu, 22 Jun 2023 04:55:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01AC68D0007; Thu, 22 Jun 2023 04:55:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E93558D0001 for ; Thu, 22 Jun 2023 04:55:17 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B4941C0AA0 for ; Thu, 22 Jun 2023 08:55:17 +0000 (UTC) X-FDA: 80929774674.18.9D6EF5F Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf18.hostedemail.com (Postfix) with ESMTP id E223E1C0009 for ; Thu, 22 Jun 2023 08:55:15 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=D5LVC+4H; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9Gb8UpD3lZt4bh5cbhFKOyQevww2sfofkVlmbPNwxg8=; b=CmJTvmlZzPt2Fi2LTRV+5AS6aQLFlrj9BIOKAx6UHyEp7RtJR4PZtTTn4Kxnsr7WxiTudL 3jGNBdPU+GfTnf3pYqOADKnEgPJOIf4a8jYk3wwxKBdQNvRCgPthUcaNPl9nhl+zZsA2xl 7zKvHsRvqv1PbMdnLoDDqk1D+zF6wo8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424116; a=rsa-sha256; cv=none; b=6KIxSCpR251f5AOjnficnhc67ouaDIyS6voCXalwbAYboQCu3rE4NP+AKx8w/PssxIKB0l aFeNOs0l6WGh4NCTG8LthBbiVIR1cyLahfx7pX7LupVN806EhbrL5RS7FEOVuT9SsBvaKQ gzPXEGUzikDgOLMU/7i5lsAW8UXUZ74= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=D5LVC+4H; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1b5585e84b4so6827455ad.0 for ; Thu, 22 Jun 2023 01:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424115; x=1690016115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9Gb8UpD3lZt4bh5cbhFKOyQevww2sfofkVlmbPNwxg8=; b=D5LVC+4HokJCz4FPeXcY+dsaTssife3Qqo2QFad+aTpXncu7uEYIb5Esgo0X9193yq JD9ShpMbmoNV55hSp3a3PNB6eqKE4aEEgjXjfXMhn0CWCq43W/gk5ji4OA6QFiz5YEbm Of780eCZQIl3oU3nzqHMjitXTfpqsvwkQAoJ3/aSYN48pFnlXdDicr+5TJONhzqSsJ8t IgcwK0tiB9rdlMicFxhfFEkVC1N5FWb0hV5n3c+iSTxmBlXa0ZlHHEAwcq0BLcrHgmll f7y7d2fR0/LzWuGi/rCVXp+lACayEzs+7pdEprt9XcBuFLw9Vycb7reaOybOjmFS0TWh I1Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424115; x=1690016115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Gb8UpD3lZt4bh5cbhFKOyQevww2sfofkVlmbPNwxg8=; b=JBACSM8CGWQLdcns5piT/jpvUePysVJEtGafsAn5z4qZ+hLWsf81PZSpdg9Wv3PHNX PWn+LwhN6IvWD5PJ9Q2szKyZsldgeXZv1H3G1goQ2Np0JfipOW7wPk2uqou+w2/4zgyE /OgWcJVbUlG721sMY3pHg/NFswkdrf9M73vBCH5PFyqZXIGbR1/ZoG/2uasbJF9zdFFH MRsWNYXdBPGprG8TRjRNFGnYrN6Lhe9VKwxTyuVqYvoxjvGFZ4MSr15YyuKfBGV+3s6T vFoi3ZHRZ+AeovddMo23xMDEFqVrl1Xm7pt8FIAOkzGcW3+P7GyLzsVrZ5CCwMIplIjd VU7A== X-Gm-Message-State: AC+VfDy7SHwaceICRMcicIf5mURJXtd1Z8tQupeFIC12HHlNnKuGML9g KHfihNO1xEvkJpB2I4uq/oLqJg== X-Google-Smtp-Source: ACHHUZ4eMYIbb4I7cyZSyRLa7A39+u7jKZFqr72/TYsRg3oKUy2dca/PQ8JGq6DHBe5p8nrFI+JLcw== X-Received: by 2002:a17:903:2451:b0:1b0:34c6:3bf2 with SMTP id l17-20020a170903245100b001b034c63bf2mr21539426pls.5.1687424114850; Thu, 22 Jun 2023 01:55:14 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:14 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 09/29] bcache: dynamically allocate the md-bcache shrinker Date: Thu, 22 Jun 2023 16:53:15 +0800 Message-Id: <20230622085335.77010-10-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E223E1C0009 X-Rspam-User: X-Stat-Signature: xucx6mjjgfm7rbyozgt4cuqj33cekep9 X-Rspamd-Server: rspam03 X-HE-Tag: 1687424115-304983 X-HE-Meta: U2FsdGVkX18xePxgfffSflV9gHeGXW20oWgQpzTA6S9x9N87BoZYN2qqq3O28E9AcUAdee2ilguajthUujjwM5t80ImebnFhk2LgVoImTJfYjPgUOhUjtR/h/IGFUwYwU64mvYAByUKd6eoJkLCNfthYn3VKBTEI/AxFHB29le4Ft+hIfZQyRwrSsl8gG1Z8Bm3jxgkWG0O9uF4d24OGfSJbL8SJiiCj6GPC34ofr5PlYbu7YbKuMMwi8yDHu2osx+Nc+rfy1r/oMh/jMmYknEnFBeet72A1e/c5v70/RN+acGIYWBsmV7EpgOB/hF7Aruc7pYB0eyD3CXXy4pQ53kqL5etl47/1YuymH3orabIyqxvcxOby+JmgGh8qBUC251DgXLySXEfoxj2QrT2A3W+anK+MxH1ffWN+dKZJ9vxfPYep86/7Mvv8MMUvIjx1H+BkG1GVK/9Ju9TyFQk9dilFVhHTgJC66rf09pa5wYHIpebgOpjqF/hFpraWtsQEdLpIMIyfqqkeKw4XSxj1AJ5roVxh+WB+J+VlLj+Xt7vxTG4uOEBCiqoDffAWGUf5spr5DSzcww2CCIyB3WRt77ka+lLVcEHAjrLqDHNoEvFGtCAGU3nDv2UFv7NMbKUoUetPWzeCf/8JtbyuNe0WFzdxQL9zPmKct186JjmSkqVA0syuoDUtMsqiUbov4VfcZsSJu+leOpt9lKZRSdhIu435m01Uc3lEWeZvyjcosozDuy5iyKnfF0DdfQu2jlm1IF0nc1pzmEVuuS+JwlORyGPQAwsPAvmbzfuSCR01mCL9l+UhKp5JUcT4e9/j3aZ8zh+4Cn7YNDlFE+hauHtsR+dzWJ493TshmSz41ugedp6DCDOW7Q739T/mbmTrJu6nS/nmDjSdyEPTaUXmNNqcfJUZ+3A7vcq79dwxlyKxY0myL5N/0rtvzAaqzC2jNPIpd6Nuf3dZGMEOY8Dno0V dQXbDrTJ nQtr3SvJufuszHB5DP/7yEpR/0Qk+JE2nQ/JAoCYv+YuRdU6xRdSbF7DwNsZnNZ2lUE79nnpAzU5UMFpxwXwJwYN5Wfn6hp9fecFCDsS5XhtQ1Glsa6A81ityVNZTWToO/92QfGPQYMdOt+ujLT0q26y6DFBwrKLClzzPSuz0bCJnC+qcBrcXOrQjDi4ShqCg8mP/sL+rQ+GFXsyoL8vajIZfDECliONoUjBKRCut3UPI6Gce9OEMgDkEaWpBH1wmvrp62GjE42OJM+qXkhTcoG5CAXu5FHKxSI5yzIr8SxHbCuBTkjSB+8A3UxKbNcJREtMFsqOJsMkterDkYE8aydyN8IPtyPpecG+Oo4HhrdbZjj2t1i1e/FXUrZknCpjze99VesG0G6al/x2ysy3zNGgIhgpRzBPdzIZ9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the md-bcache shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct cache_set. Signed-off-by: Qi Zheng --- drivers/md/bcache/bcache.h | 2 +- drivers/md/bcache/btree.c | 23 ++++++++++++++--------- drivers/md/bcache/sysfs.c | 2 +- 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h index 700dc5588d5f..53c73b372e7a 100644 --- a/drivers/md/bcache/bcache.h +++ b/drivers/md/bcache/bcache.h @@ -541,7 +541,7 @@ struct cache_set { struct bio_set bio_split; /* For the btree cache */ - struct shrinker shrink; + struct shrinker *shrink; /* For the btree cache and anything allocation related */ struct mutex bucket_lock; diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c index 569f48958bde..1131ae91f62a 100644 --- a/drivers/md/bcache/btree.c +++ b/drivers/md/bcache/btree.c @@ -667,7 +667,7 @@ static int mca_reap(struct btree *b, unsigned int min_order, bool flush) static unsigned long bch_mca_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct cache_set *c = container_of(shrink, struct cache_set, shrink); + struct cache_set *c = shrink->private_data; struct btree *b, *t; unsigned long i, nr = sc->nr_to_scan; unsigned long freed = 0; @@ -734,7 +734,7 @@ static unsigned long bch_mca_scan(struct shrinker *shrink, static unsigned long bch_mca_count(struct shrinker *shrink, struct shrink_control *sc) { - struct cache_set *c = container_of(shrink, struct cache_set, shrink); + struct cache_set *c = shrink->private_data; if (c->shrinker_disabled) return 0; @@ -752,8 +752,8 @@ void bch_btree_cache_free(struct cache_set *c) closure_init_stack(&cl); - if (c->shrink.list.next) - unregister_shrinker(&c->shrink); + if (c->shrink->list.next) + unregister_and_free_shrinker(c->shrink); mutex_lock(&c->bucket_lock); @@ -828,14 +828,19 @@ int bch_btree_cache_alloc(struct cache_set *c) c->verify_data = NULL; #endif - c->shrink.count_objects = bch_mca_count; - c->shrink.scan_objects = bch_mca_scan; - c->shrink.seeks = 4; - c->shrink.batch = c->btree_pages * 2; + c->shrink = shrinker_alloc_and_init(bch_mca_count, bch_mca_scan, + c->btree_pages * 2, 4, 0, c); + if (!c->shrink) { + pr_warn("bcache: %s: could not allocate shrinker\n", + __func__); + return -ENOMEM; + } - if (register_shrinker(&c->shrink, "md-bcache:%pU", c->set_uuid)) + if (register_shrinker(c->shrink, "md-bcache:%pU", c->set_uuid)) { pr_warn("bcache: %s: could not register shrinker\n", __func__); + shrinker_free(c->shrink); + } return 0; } diff --git a/drivers/md/bcache/sysfs.c b/drivers/md/bcache/sysfs.c index c6f677059214..771577581f52 100644 --- a/drivers/md/bcache/sysfs.c +++ b/drivers/md/bcache/sysfs.c @@ -866,7 +866,7 @@ STORE(__bch_cache_set) sc.gfp_mask = GFP_KERNEL; sc.nr_to_scan = strtoul_or_return(buf); - c->shrink.scan_objects(&c->shrink, &sc); + c->shrink->scan_objects(c->shrink, &sc); } sysfs_strtoul_clamp(congested_read_threshold_us, From patchwork Thu Jun 22 08:53:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0B5FEB64DA for ; Thu, 22 Jun 2023 08:55:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E4618D0005; Thu, 22 Jun 2023 04:55:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76CEB8D0001; Thu, 22 Jun 2023 04:55:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60DD58D0005; Thu, 22 Jun 2023 04:55:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 516608D0001 for ; Thu, 22 Jun 2023 04:55:26 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 31CD180761 for ; Thu, 22 Jun 2023 08:55:26 +0000 (UTC) X-FDA: 80929775052.22.C4E8B86 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf22.hostedemail.com (Postfix) with ESMTP id 4BF6BC0017 for ; Thu, 22 Jun 2023 08:55:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="f9niG/1n"; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424124; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u/vExFBLW3qTniAw4XeP9KQ69i9zow/4wwrpInqlwjM=; b=JP07DisFoDa/vWCMd2yUfGwzzqM94sGZz59GpmZsB6tc6w6aAma4rNqHMBI0lkFRWiY7ll UivuZhSdeZfwjx52Cm11sPMx39+mqNspdbSXZcd9b23nTEL1gM7PW1VgmaijVo8Y2tZujr fNM8SxIiup96iWIh/Aj3n+DACmMrjbY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424124; a=rsa-sha256; cv=none; b=eBD83Q1i4AlrvJmB0OcwSCVSF0zgjvam4uc5VcBWuPhClSuuoABLLrCc4pTnlHgxmPTmca hV3EbC2fq4zbBVD+w+JYnQaNbe3ehqoIbrIiisVCD9vsmAltRrdaMzwplNRl6IZeETlDf3 +zncEWiJhTU/YmJM9GKeJa4e6y5zDRI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="f9niG/1n"; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1b52418c25bso12032625ad.0 for ; Thu, 22 Jun 2023 01:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424123; x=1690016123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u/vExFBLW3qTniAw4XeP9KQ69i9zow/4wwrpInqlwjM=; b=f9niG/1nDZCnLyGAdmXOPnr0UhYRgdnlIknKHTp5fpAvQQctnSzqFRWkCf20Ww177d QG9jeeQWMuTex/IwaFDMA8vXlH6fpJPO6Zn2dpmhMfbdqmxsLU2UIGO2NDOzTGYWiDw7 ywIPFwnvUTjEHuZeccmcjrjEkLsXSYquHahbHd6ZdjnzqEP2RHdSMNMWgiY9L8TVZ0lC G4aVTSqjJ5NrWtnVCJTDf1mbR/bEFK3aai4gwzFQmPA34Ikuzxs8i6IqNfr5Nxc5N0yO /Hq8+LGyQI4XHPCqHbrNX4k2UTVR4h1zk7vD27aNaHUnKMeOkCqJba/XQtzAGJZToHzN Aj7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424123; x=1690016123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u/vExFBLW3qTniAw4XeP9KQ69i9zow/4wwrpInqlwjM=; b=XbZ/YUBKLe1eioC+a7tyAiX4A6bIm7Xw3CGutwtClPANOiCSM3Zpt6Bhwoez/2t18Z TDUXqQw70jBxr7DLILofbod1d3Tf6mhuR3PrMuO4A1VdG/t2dE7HYqE+HnmlnTYDY90m FZGiOYQaRPNWtMANNSoVnipMLtujZUDCFjXccAFWI6l2vIDQZ39773Bs6K5/HEr6qdkC uW6x0rTG8/qxi2Uiy+r6PjlQTIUKmrop4i8ROL4g2246UQNgxBSc45vLCgfEGHt5iXsU LgiO8gA7MpKlMJCw/FvV5vMXkS6mnPTSOc8MDfEEpZWTyrWq7ewlFFtF/+ppqIqQe1Et CL8g== X-Gm-Message-State: AC+VfDyVtDZzbMg2972AFtBitJVl+ofSJtdU/Fz64YZq5ZiGVS6cI0ia tkBAjS9m3dENGqsS/GOEAHyBGQ== X-Google-Smtp-Source: ACHHUZ4R6XsCch8DMbBn2ilzIIAPskFMWcDdECndDXpbpwroXAZTFBoXbn8TaLTiXdt1Xz94/St+OA== X-Received: by 2002:a17:902:dac6:b0:1a1:956d:2281 with SMTP id q6-20020a170902dac600b001a1956d2281mr22035085plx.3.1687424123198; Thu, 22 Jun 2023 01:55:23 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:22 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 10/29] vmw_balloon: dynamically allocate the vmw-balloon shrinker Date: Thu, 22 Jun 2023 16:53:16 +0800 Message-Id: <20230622085335.77010-11-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BF6BC0017 X-Rspam-User: X-Stat-Signature: tkw3eyurjjion1eqgf67gr717jbu9qxx X-Rspamd-Server: rspam03 X-HE-Tag: 1687424124-718913 X-HE-Meta: U2FsdGVkX19CjhVa93HxTtha6vcbGCN+UW4bP2oYSJN+tO3l3rbqZ9tsvigCCEbE/uf7RagY4US1lGI8wLXqSdFqTLwH3fgFO9MHuAvUOReCr7x5i/s+j0swkDurgQwoMiVZx9l9FqCmjDyWuIWBV0yHPEMxev1ZKfKPBdymkLerZrVsd8ZQd2ck5Cixq2iWNEvnxP6lJM2LcUKhIND8T6tZXkvU88IbZ+XdzhoAhh50rwhO0WDYuRP2dj3D8fFG91XByyqlHdvGIXKnFoC1O8u41vcNFzMYROEnQvBw6sGSm7NbKfMOhOsSUExjoATLcglIGuAF2KriRXYZxneN+SskFYa2uLXxLLkjSknCz51AkyrqrzFwBPf4RO648VPi0E3Fvocvd3dopSvMaZSaiQ+4ymZrSICg9fHXWEsE+70wJnOJPx0Dxpvu6E2P9lTOJnIDWqRZ9/HAXrJZWvcCEhRoiYyVKVSvbYF9lunY+ZLqRfQbjDX2YHtSr4A3a/z0EvcCdR6+6e2Kpauc4GAg5eBW7D2cbJ81M29S2jsCcEkflmWk48tZP+TPxHcSN0lMHP64Ozyu6CseBeF0c4UhX3KpkLFDWuoqshI8v4uYSHXhgcyrtW0rdfJMhdMUZCZO2Z9qfvvxKym+WzXRLsvcJJWfICfF1nWHMaxrUbsql2ymbECEifAlhSkiittGWVmvhpstnl+0bKpO7YPFbrpT1vCnuIdi5kiykNBXhFzsEaizH4/Xx3fc/W91ZR0BxayLde4F4vwPwWNbfRiRX5xRU8gS7BLLraKFJlaOPzUPIhkoTMdF7KfGvfB9bGqA6FHW+RVfDucHsnX/h3kxLzbOsVulAwxK3Dqi6oSeCGhEKxDOa9SzqT9pMwdwsdASoD6HoLDKH0HrryYADeS6dkEkY4usEbBoqJtbG9ocWWWmZljkSp6CdjhEwEhFcZyemPQER1ov0L+3i2pFLMtqmIB ZSKipqem OtIby7ZiXoMG5YOHWtFOf40oJMq9zA8LJJ7z3Ozc43bhNfQIbqZx4ayWeGflnSMJXL0glziDGMngK2o+ji7jd8k1Vj13frIHZcoRPDeljRSzT8B0lAgOC3/nZg7gtE0SkU13myhwRnsm/LEhCKkXAvYTI1NrDKcvImO4OE2cFzgSv5f83qZgGEHP1Qb0CY8l9srNxZmSxORqtl960ojIPRJbj6S4rck9vK0CoETEBcIt8N6Za+Fe0Ylj+LFBjFIbT5mr5rP2EMt/kloyoYsrGji8foozXLWt+tAvrmZiTTb4NwlS+Lrr8qveYQLf30WSeOnTCFvXtuOplDMfDDnUO0OsLt67z14CFpIHgxHlnnfQJEs8NiJPcMWEnMaGQ3xYLgE4Zi7RfteDMGdPRCwoea86CdUf9+1wGOvSiMxt4jV64XfvGrYinsErRhpM2Ji+nGhx5Ki1I0pSKly1BoeY/zt7fJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the vmw-balloon shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct vmballoon. Signed-off-by: Qi Zheng --- drivers/misc/vmw_balloon.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/drivers/misc/vmw_balloon.c b/drivers/misc/vmw_balloon.c index 9ce9b9e0e9b6..2f86f666b476 100644 --- a/drivers/misc/vmw_balloon.c +++ b/drivers/misc/vmw_balloon.c @@ -380,7 +380,7 @@ struct vmballoon { /** * @shrinker: shrinker interface that is used to avoid over-inflation. */ - struct shrinker shrinker; + struct shrinker *shrinker; /** * @shrinker_registered: whether the shrinker was registered. @@ -1569,7 +1569,7 @@ static unsigned long vmballoon_shrinker_count(struct shrinker *shrinker, static void vmballoon_unregister_shrinker(struct vmballoon *b) { if (b->shrinker_registered) - unregister_shrinker(&b->shrinker); + unregister_and_free_shrinker(b->shrinker); b->shrinker_registered = false; } @@ -1581,14 +1581,18 @@ static int vmballoon_register_shrinker(struct vmballoon *b) if (!vmwballoon_shrinker_enable) return 0; - b->shrinker.scan_objects = vmballoon_shrinker_scan; - b->shrinker.count_objects = vmballoon_shrinker_count; - b->shrinker.seeks = DEFAULT_SEEKS; + b->shrinker = shrinker_alloc_and_init(vmballoon_shrinker_count, + vmballoon_shrinker_scan, + 0, DEFAULT_SEEKS, 0, b); + if (!b->shrinker) + return -ENOMEM; - r = register_shrinker(&b->shrinker, "vmw-balloon"); + r = register_shrinker(b->shrinker, "vmw-balloon"); if (r == 0) b->shrinker_registered = true; + else + shrinker_free(b->shrinker); return r; } From patchwork Thu Jun 22 08:53:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D1E4EB64D8 for ; Thu, 22 Jun 2023 08:55:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5718D000A; Thu, 22 Jun 2023 04:55:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B644D8D0001; Thu, 22 Jun 2023 04:55:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2BFA8D000A; Thu, 22 Jun 2023 04:55:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 95E768D0001 for ; Thu, 22 Jun 2023 04:55:34 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6747512010E for ; Thu, 22 Jun 2023 08:55:34 +0000 (UTC) X-FDA: 80929775388.29.F411DFC Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf29.hostedemail.com (Postfix) with ESMTP id 609FA120002 for ; Thu, 22 Jun 2023 08:55:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DMwdTIl9; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rw+THIB7nYtdEoKFrcRpf+DtRSemfk8nQmXyWQCWLOU=; b=BjYMHkfNyfcDN3e6hFIeDr+Y8h0GCilUkfboHNzJ91pNS5n0n5Jk65/ElEMFEF15aQ20Pc TUsNhqdZFjwjeRghA6SZOeIx23mEpCPaTQK+Ngcf41lYHyzoJO/gU0ZVDjgS4Nb1uuYeBn lx3/Tiz9XjRC/NTmR2qN5L5HnxH3EqQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DMwdTIl9; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424132; a=rsa-sha256; cv=none; b=67fPVuEiNdP0K8LGxIkDODrs08HXAPv5DIJQzAclGqeD//SSf52tDnpmvrVkrLnlrI86yk SUwBDZYn/g1gVNWbeixsN6lq6JdXP+gtiEK90CeaJDx9Jcr6AIUwRH6X9yd2RkZsPFI8xm AlOjLaNLJOeIyr0XcDEMYtbWJKW20X8= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-51f64817809so680031a12.1 for ; Thu, 22 Jun 2023 01:55:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424131; x=1690016131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rw+THIB7nYtdEoKFrcRpf+DtRSemfk8nQmXyWQCWLOU=; b=DMwdTIl9g+okPZHJv8qTrb+gYAPEj9v1sxcSDlzK5ch7N+TuIC/BcFHKohEZo55ND8 XU29tOhBmT8qLwPtzV/wEHQ6iOqaUqSy8YTL9GNxgMKyLojaih+JCKFRV8sA50VDaMhL 4ENhUxcmuP8buO31FrzpXihlkiHbj7xjQ5jQCZ/XAAfjoNJ005ZfE8KQlMXJnunKzCcO Fe2zvwkJ/uki5/MCqM+WcH9ckA1bFEYVaXIUmwP+4s7h2vpQxpmZaq/OfODYIqbWYJLH gNO1EGlemqhIu8SDpgbpJo6kXONx1A4ErgZXcMZMC3KoLi5/PQ7Kas2P+1tIIIXAXu47 3GiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424131; x=1690016131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rw+THIB7nYtdEoKFrcRpf+DtRSemfk8nQmXyWQCWLOU=; b=TW6mOwnc3VHZ4XPCQ4+q1hhf2KoNJFH0SVCtxpO5pXx0LzYaXgtJewLEyHGxUoKeFN HnzExF/r4Gf1k/jTGgkAQdT/3W5zWbkeTH+/XR87z9aglenW4qwhhnyAYMVqeu5n44iP ZhlUHyeXKyX/7/i9nJgsBFflTsAEFirlu93j5tWBBcNauywVKyNpj0NPQ0N/Ery7Kk3k cYxcnMivWx/swFAsh9eBHpTICt99hBH6mhXN5aYhknzPkD58ORcJKLSCMcYIM/02ovGC pXzdhb+0rrBE7pyxtIXswIOoN0dYJpLEadl4itNXyCJI0ZDwNFOEwZDErmv00VjxvYj5 pp9A== X-Gm-Message-State: AC+VfDwD2WNDqZNlNEaS3+cck/oWpvfkmebwHKktnCNDVP2gUFUZEdM9 DuTPShn0VJX4RB59Ne/E+fkF/w== X-Google-Smtp-Source: ACHHUZ74w6zUyxxONmkdhWFpbjqcQv1Ri8s1BW02LunWtr1K9sXtmODVbQHvvX6jLBNiniDuaX5mmA== X-Received: by 2002:a17:902:d489:b0:1b4:ddef:841e with SMTP id c9-20020a170902d48900b001b4ddef841emr21417877plg.4.1687424131373; Thu, 22 Jun 2023 01:55:31 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:31 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 11/29] virtio_balloon: dynamically allocate the virtio-balloon shrinker Date: Thu, 22 Jun 2023 16:53:17 +0800 Message-Id: <20230622085335.77010-12-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 4zz4m6pzog17qh85p9d6jfdhwieky41r X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 609FA120002 X-HE-Tag: 1687424132-662456 X-HE-Meta: U2FsdGVkX1+qgqFVVb7WWWzpwWfym/ddJPUP0nz7J8Q7SFeWl8kmxjfxY5Lmh2spJB9G+Cq1MmttQsTCoonBBHcxVDoMtkSqxGO1Xa/drJ8jszHwGsyKsMazEs8WKwFFTGLYJtDuhj4c0vL/LTNd2VnKPsjyOjbq/If0RJNUeTo7J+eUBMNwvZcFTW8a5dMC53UUuJ6+gT0xjLBq5D4gtxKw+hY2b3IpwY8BZRJNBIZqhqvmK68D1bxAoohWOydxWHPf+vJTc87K57cYvudfz4c5P+5V0DJiUrNUkdrCLys0k0vNH9R1Va+ScfNfQlw39cvaAfmG+fV6ur83LEAfqNB9BEufESA+k2swO4e6C/0IhrZKhk8438K2UKlhb2j2B3BfevEZ+z1f3v45GdyG0ZCKHHzJDp7Yl90hyaG2mMaXq3/DMPP2YVxEJdndKYJihqYT0V4avU3KuwIWWBgBD58JatzM8DMa7+6bZs3qm8lX0IQrRbOr4XnBEr6x+39Hq9Lyr82Dm899E74OZl4FFZS3y3J3/p9PtZgeUP0t2NncNV5DnC58VSiXZK93xgz29bz8a1TTJ0Q8oowD7Gp82LoVunpzro2/XpQ4TjPykxG2wb815W/0a9AwggB30DI8w/wXb2YRoDs9vT7oVXjaFoJE8blKzPlRtL57zxZq/AlrGqXes/17M5NGjyw7oQfQt5CFPu43pwkStziEEXatF05387D3uXjOYyucCvwyrZowac1ikX2XyIm3CUhTHhnzoutTd3MBFg4xJQZKZbRkVRSihSApB9nlRw/RxI8iB469G9IrqeIMc+80iWbqzxU5eMa7+qe9SzFdDqU0KcrR29kxUG9HPhccqBHM/V0xAXcw6UqKesUPK5KxKwV7GLZCoLJWcCt+/kREGgENlgjpJlEy89l2dl7jKSYN0xuMYo+mMWhlx8b5yTmlEybLUhkrEzp857xZN6vml6NobBs 2nO8xM7E lJKQEqX5gvTlSSIMAlvHyOth6zG9bekTFdb4wJ73jEK91f+HVxGpDhyVkoiiYCaewrIiAd1HN096ryimvexXwwr1Qdb7ceaoAlyXCT2oqGoe/lqt4ny7Th+srQ8Xx2h4hQlMke8gxb8jKSd8kgBDFDE2FR+0EUK00qDd4ztQaReYJrxe2vz/BGQdPctpGjY2V33svPztSp064Kxdc/EvoUkVipNiJV5k08QzGfYCyV8u+mvoYoAItTLNb/TnbpDNlvAVgoXVQ9YWLKfLGs3lm3TlkzdyBKPD6UYOqLz4NrpyXaymGMDJaezgMhjL6AEH0S26FuMvhE0E+X+s/hVSyaVI6Qma9wE51K1Caq4Ae5+WSBrf9+9XgwbfLa1VYsyItKA459gOUbPmn76NRcScyQN12msiJbrn1I0PJmA8J+b56aYttmP8vnMSomgl2z9ggInWE8idyi0DBV8ASoVbLVLs9Uw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the virtio-balloon shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct virtio_balloon. Signed-off-by: Qi Zheng --- drivers/virtio/virtio_balloon.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 5b15936a5214..fa051bff8d90 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -111,7 +111,7 @@ struct virtio_balloon { struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; /* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */ - struct shrinker shrinker; + struct shrinker *shrinker; /* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */ struct notifier_block oom_nb; @@ -816,8 +816,7 @@ static unsigned long shrink_free_pages(struct virtio_balloon *vb, static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { - struct virtio_balloon *vb = container_of(shrinker, - struct virtio_balloon, shrinker); + struct virtio_balloon *vb = shrinker->private_data; return shrink_free_pages(vb, sc->nr_to_scan); } @@ -825,8 +824,7 @@ static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, static unsigned long virtio_balloon_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) { - struct virtio_balloon *vb = container_of(shrinker, - struct virtio_balloon, shrinker); + struct virtio_balloon *vb = shrinker->private_data; return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES; } @@ -847,16 +845,24 @@ static int virtio_balloon_oom_notify(struct notifier_block *nb, static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb) { - unregister_shrinker(&vb->shrinker); + unregister_and_free_shrinker(vb->shrinker); } static int virtio_balloon_register_shrinker(struct virtio_balloon *vb) { - vb->shrinker.scan_objects = virtio_balloon_shrinker_scan; - vb->shrinker.count_objects = virtio_balloon_shrinker_count; - vb->shrinker.seeks = DEFAULT_SEEKS; + int ret; + + vb->shrinker = shrinker_alloc_and_init(virtio_balloon_shrinker_count, + virtio_balloon_shrinker_scan, + 0, DEFAULT_SEEKS, 0, vb); + if (!vb->shrinker) + return -ENOMEM; + + ret = register_shrinker(vb->shrinker, "virtio-balloon"); + if (ret) + shrinker_free(vb->shrinker); - return register_shrinker(&vb->shrinker, "virtio-balloon"); + return ret; } static int virtballoon_probe(struct virtio_device *vdev) From patchwork Thu Jun 22 08:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1413DEB64DA for ; Thu, 22 Jun 2023 08:55:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9913A8D0007; Thu, 22 Jun 2023 04:55:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9415E8D0001; Thu, 22 Jun 2023 04:55:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 809C48D0007; Thu, 22 Jun 2023 04:55:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6F1A38D0001 for ; Thu, 22 Jun 2023 04:55:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 410C916040A for ; Thu, 22 Jun 2023 08:55:43 +0000 (UTC) X-FDA: 80929775766.12.9709BF1 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf18.hostedemail.com (Postfix) with ESMTP id 401461C001E for ; Thu, 22 Jun 2023 08:55:41 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DTUuH0I9; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424141; a=rsa-sha256; cv=none; b=k293em/gfhrjQU/GmT4wKnQOfuTZ6RC2J7CfrRdanYG9qaGiB07s0A0aMX4xn3kR+4wuMf rjr0iekRRnRHbQDQkFVJ4vM+8uHP1tb0KXRvFgLl//2SYPcMiMnn43r/YYrmPzAOpkGU4e bPHb12Jq4P0mbPEGK4kWlpA75hx00QU= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DTUuH0I9; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qqb5b2GNvTgVTRVLR7iEMYBQ9KFY39GKHYb23c1SNH4=; b=JCEtgunJ3okQ4W0TrudqmXOvQ+0UD649w7ac+KNTdE48LnSgFbNkoZR1/r2FvcrtiDST3I jknODRD+xXbnDYLI3re4hk2BcY9baxBmerF9AnrS1SiuUsYTtsrEa7clSJjqhwVEdVFrZD atE/qM1y63+zH2UVuX+0d7wq9/PZBJU= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1b5585e84b4so6827855ad.0 for ; Thu, 22 Jun 2023 01:55:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424140; x=1690016140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qqb5b2GNvTgVTRVLR7iEMYBQ9KFY39GKHYb23c1SNH4=; b=DTUuH0I9uHLii3LJ9Zyuu9c+EPbxYZC20xJ5+3ma+Kp5qpBNFOFTSMnclhdsEVCqLi IhMw89U+Z8kUvjOX0p1/U1TPShfVW/vJXGuaCCTRG1bGCD9nXSC4/ZiagSGalcNeTARf jURjtWkQSw5PrzWP1wBPTMrMWiplRo47rrBJgWE68ZF11RySU51PM2cKEbziuuvrByZZ Zy6z/i7b8JtNNsvUGe9g7/sl4BzjGsT0dMzcCmwPn8WMxyE8EEE7gXWjStqzol2ehPl/ 69+rSZ9IFQ8XiOW8vY+908jjX5pcw5aN354YWworQkitYsT+fnzRPZItmEL7YHvgl8Dv 6USw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424140; x=1690016140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qqb5b2GNvTgVTRVLR7iEMYBQ9KFY39GKHYb23c1SNH4=; b=Co/MBgRKJuBu1ASXzFDET1tLx51WAVxy9r5hl2uXefH1SF+igtni1mmoqC6no0R0mR oCONQ9RfuYKv+beIgctmvm7ygMGooFlWrI8/bNkqfeyGJYLZ73UAC9XqESIcoYRqbYsR WF8iaCnyy3eKWSYCcp+5XA1ARFGplEJnVXRrbNAOTSDP9vxnpu3sf5WWgyr2n2ZU+/7I Rq6j7vdV/27q/GAjE//Nw3gqmSnka3/ER799ArDhVoOo1QxmQ9Xh8BV+ke4ZdJ0VFZAg RxFPdTjJWbHx7a+JbFeDugz8AT8XmUpu21pOx0gKGynRMfm2YRm9kjhplMNzdi5Uk8hc XkbQ== X-Gm-Message-State: AC+VfDxkFX170gJ5NHdzVGFR51JqqzZC4nz7K4/hZcUm2WdFsSmVRX5b vO1IMfVljUz/41e5EYR61AXmBA== X-Google-Smtp-Source: ACHHUZ4U5RlW8/2AChRXKukLEKek4tD/ljCLASAoWCRPvf/eu3Wdfis/k+W/YQGMZOvsRYh9ePd5+g== X-Received: by 2002:a17:902:d509:b0:1af:a349:3f31 with SMTP id b9-20020a170902d50900b001afa3493f31mr21732029plg.3.1687424140189; Thu, 22 Jun 2023 01:55:40 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:39 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 12/29] mbcache: dynamically allocate the mbcache shrinker Date: Thu, 22 Jun 2023 16:53:18 +0800 Message-Id: <20230622085335.77010-13-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 401461C001E X-Stat-Signature: yghzp9yk7mxbszi1dgjog3xbg41gm4ni X-Rspam-User: X-HE-Tag: 1687424141-18888 X-HE-Meta: U2FsdGVkX189m1H6D0/VI4j9Wyhu5t7oeK253Qhcxhw9zKYohVYq+BBKOajed4deHdzZ2Mm0Ruk7/3jmekjKZCGOe7F4S353ZFf5hXiZSF3L0pp5MoMgg9P93mS+u+jtWA9BpCtV2punkY8abdd79pgXyTRPPXk1jMCbDSxG0QySFTcQZMW8kFhdaU1/eNHw6IWfAvrFguMZ5xwJPHjpgNV8u/B1X6e3u6DRYvyJ3ivkI+d+5kNTW/2FqpJP5MeneLim+GCO7/Ey/7yrDxF+eXLn3rOTvbEpP9D2hH6ZW/NBhPlY4UHnNhkPyOh2oBf3gwJJwHtJqBTlOCgo2qlW5r9DfhEv07ythjkmJPua08/DQ2gp+aaxpkS/C0OyDjqUSxhyYzv+v2IUkkVpvZYHP6W3KgpTmL3eClI65+rhXUvx3fJA/pq3D4W5dkL2MsdycGOzGpy5PfYHznKhv4kRzZvrZs97X9VgPbztBI5JA/kcSoy9Yys5n/YvHSlwPVWbRHelpC8FdOZLDNWsTOpEc9I4zFgXopbXt70NwaLmPFagdXZtMKBq9vUgQvppXKeCov8/wdGY7hw83JZMKgknc0htEheSe5nTH8QhcKgRVX6mqXbtU1OK11rD+8ys509hBpdhF5qTihQTfAtpyzmtslaFDHu2K+zC3/nGTo+HWKGJMv9Y14yB+RDEwREzB9ZmFptVx7UrpnxxBT2SqeQExSpfCDQSBa7amtg16irVitl8yodztrYDKRg+jIn20jAiFhc22KFXEs9E0ZDUdCTt8VOMofOGP+k/jKNF1rQnqKkWxTuBp4Llx5UFP4JGJQZ53HkD+lBrMIlG2DWHt09qfXp2kSM31nL/Aso2yAcENGsVAwkr28JCJl3wQcav10eqiQteKgNE7dShE8nMVMdEvq8NK/ARJPF+a2pwsx6gcQFzV2Ctsv1mPBMCvbY/kdqK8vwidKwrekQWRIgLF16 YdUxga2X qckrw2ngvgB/pNw696pwVFjAJmuOas+BwOdRP+Li69JY5841Crrbyv8Yl+elHiFTekZ3Te1kzzJ8kgSB8HRhl+7CU3eXz5YxZmIC2oXkaY7wQzxlNWgTYkxVUzFeQ+gedXdDFbFgsmU7/PBcl5821AECf4v8d6l3SQL8+7SNYWJG/121oqS4rB2dhP5x+9xQjK7fIhg/mZ7L3OOo0bzAuwZmaKW8vbKlFVkQCMGPZUggVc0Fy66BGMnSS1mKL8GaJGvX/eDn9qmXBpEZ5pRYVwOcEhXkiSr6nbeqTaD3k65m3MrHAMYxCUqAK6mvr+SzWT8qfM80I+4WAjb+Hr4oOPEq3dyNgieODDGGodGgTnNJ6IAC7UJ7p3iZWP7ZTvLHeQbdKM+jZ+9QeQtkxtY4Q/35pbv1fq/fD0abt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the mbcache shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct mb_cache. Signed-off-by: Qi Zheng --- fs/mbcache.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/fs/mbcache.c b/fs/mbcache.c index 2a4b8b549e93..fec393e55a66 100644 --- a/fs/mbcache.c +++ b/fs/mbcache.c @@ -37,7 +37,7 @@ struct mb_cache { struct list_head c_list; /* Number of entries in cache */ unsigned long c_entry_count; - struct shrinker c_shrink; + struct shrinker *c_shrink; /* Work for shrinking when the cache has too many entries */ struct work_struct c_shrink_work; }; @@ -293,8 +293,7 @@ EXPORT_SYMBOL(mb_cache_entry_touch); static unsigned long mb_cache_count(struct shrinker *shrink, struct shrink_control *sc) { - struct mb_cache *cache = container_of(shrink, struct mb_cache, - c_shrink); + struct mb_cache *cache = shrink->private_data; return cache->c_entry_count; } @@ -333,8 +332,8 @@ static unsigned long mb_cache_shrink(struct mb_cache *cache, static unsigned long mb_cache_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct mb_cache *cache = container_of(shrink, struct mb_cache, - c_shrink); + struct mb_cache *cache = shrink->private_data; + return mb_cache_shrink(cache, sc->nr_to_scan); } @@ -370,26 +369,30 @@ struct mb_cache *mb_cache_create(int bucket_bits) cache->c_hash = kmalloc_array(bucket_count, sizeof(struct hlist_bl_head), GFP_KERNEL); - if (!cache->c_hash) { - kfree(cache); - goto err_out; - } + if (!cache->c_hash) + goto err_c_hash; + for (i = 0; i < bucket_count; i++) INIT_HLIST_BL_HEAD(&cache->c_hash[i]); - cache->c_shrink.count_objects = mb_cache_count; - cache->c_shrink.scan_objects = mb_cache_scan; - cache->c_shrink.seeks = DEFAULT_SEEKS; - if (register_shrinker(&cache->c_shrink, "mbcache-shrinker")) { - kfree(cache->c_hash); - kfree(cache); - goto err_out; - } + cache->c_shrink = shrinker_alloc_and_init(mb_cache_count, mb_cache_scan, + 0, DEFAULT_SEEKS, 0, cache); + if (!cache->c_shrink) + goto err_shrinker; + + if (register_shrinker(cache->c_shrink, "mbcache-shrinker")) + goto err_register; INIT_WORK(&cache->c_shrink_work, mb_cache_shrink_worker); return cache; +err_register: + shrinker_free(cache->c_shrink); +err_shrinker: + kfree(cache->c_hash); +err_c_hash: + kfree(cache); err_out: return NULL; } @@ -406,7 +409,7 @@ void mb_cache_destroy(struct mb_cache *cache) { struct mb_cache_entry *entry, *next; - unregister_shrinker(&cache->c_shrink); + unregister_and_free_shrinker(cache->c_shrink); /* * We don't bother with any locking. Cache must not be used at this From patchwork Thu Jun 22 08:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CB99EB64D8 for ; Thu, 22 Jun 2023 08:55:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B18B78D0002; Thu, 22 Jun 2023 04:55:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC7D08D0001; Thu, 22 Jun 2023 04:55:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990F68D0002; Thu, 22 Jun 2023 04:55:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 86A8C8D0001 for ; Thu, 22 Jun 2023 04:55:51 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4402B1408DF for ; Thu, 22 Jun 2023 08:55:51 +0000 (UTC) X-FDA: 80929776102.24.FA2DCFA Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf22.hostedemail.com (Postfix) with ESMTP id 64C9CC0005 for ; Thu, 22 Jun 2023 08:55:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YL01pLqx; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424149; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eFTuD75yazkbkprx1nDK+J2a+gsKcpdTH5uLLFxlsCE=; b=LpG1GFFjtPK60AvFQI/mYq9g/8YQFrYHL5ePWrEgdAyc/8Q0/YjCXCDLKOLnC19g7gucqS ukxdIOgYQGcCpPwNidiJg/pRNkU26KkI2ciTEdzCYTlOziLau9gvGKdVtCxqK6ay6je7F6 yPjczIrYDaZaJoqf5cxPI2S99jiyPkQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YL01pLqx; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424149; a=rsa-sha256; cv=none; b=59H9Z+u2ZStQJ519Jy0sr0BaoBpsSr5XgkmLaWzuGyL1tS56Qdh49b0//4tZ6sSsnyeX9W yu8v8eda4Akj4fumD6DQ3YcGMKbZs251qApJVWJyJskGV1zGkx41fpkoFcXUScg/RbWOWb fiePpq2Ox/M5dUJHFM+FjXDPvIeDmlM= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1b693afe799so2465205ad.1 for ; Thu, 22 Jun 2023 01:55:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424148; x=1690016148; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eFTuD75yazkbkprx1nDK+J2a+gsKcpdTH5uLLFxlsCE=; b=YL01pLqxeJdJ3yx7vlwGwrScHLp900A8hEE3Wf+RWJNlhx4L8kHsp9yLDKLvMKBCnC 3rsy/Bt3MovGZd9xDykjB2GgcKTKrjV0UKR/p/LUcqeAgCcK1oeyqL4lKiu0nVi2NGBN TaYF2c0QLLlLRjtauaahzpKEMrC5bmEQ14igH5gpZVZzZ5PZmJEXxyLsPSab+WZLJtP7 soRQ7v+HKrW5nh3UOzeGEOAr+30lPAAddvMLxo2KnZ7MK/KtFrltdlmvaznhvWw7dzRl oronsxPEPE0DLPq3etiyjPQ96cDhygWHhtXaurPBhnnq3MU+faq6ga287Qti+jNFsnPy uaEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424148; x=1690016148; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eFTuD75yazkbkprx1nDK+J2a+gsKcpdTH5uLLFxlsCE=; b=SkGaudaLiOQw3Xnfh8ip4UFx4KH9KWmdNS1nEeV48oNaRz8zHBGaNaoDgDvrdcE0pV 6SWoXitFhK2XbTNoH3c3WDczllc6w+RH0e/0md7lGxlC5rMhFRjVkyGI92nWBlJeGINQ ECGFEwZOwbW0hHHVflmOS5CXURuX+aCbNqJvfb2FzbZDttjkDuYlBc4ecLQRsqb+3hmw ShhULBuF/7hq0kZ5l16HzfxzJHB4s9+Jp1eH6B1SLbGSl8Ue6YboxUYPX6zSSAiLX6Q0 IbvmZV43t2DlBE4UlI3YuFH6kNjwi2wL1M1A1qEV1f6BYWho3LJ6bNj6UoOP4TT1DljW Az9Q== X-Gm-Message-State: AC+VfDwFPzyqNFrxTw4RW25TSp+VlvSQcYSovePduLSdt315unv8c4H8 FfZ9hdR3G0RuiZOUMDx/tDB97g== X-Google-Smtp-Source: ACHHUZ6jVuCzKKZsdSS1vKWkjhi+wc7lz2Kfe1cJ6l4zUout3MX707Lbn9nWcxBb7Ur8w7srUJ3z+g== X-Received: by 2002:a17:903:2451:b0:1b0:34c6:3bf2 with SMTP id l17-20020a170903245100b001b034c63bf2mr21540496pls.5.1687424148210; Thu, 22 Jun 2023 01:55:48 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:47 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 13/29] ext4: dynamically allocate the ext4-es shrinker Date: Thu, 22 Jun 2023 16:53:19 +0800 Message-Id: <20230622085335.77010-14-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 64C9CC0005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: i6awsff51ucx85y1hxzypzipkkf6xyu6 X-HE-Tag: 1687424149-844419 X-HE-Meta: U2FsdGVkX1/Zkh9EGa1FO97qOdSyveRYKog4lUrENjNc/Cua3OGwKTjIQrcd4rmmNFRwm9ouxK4zOeg0tPIrs/QeRhgPOidsiz/NCBWP6ozl1J4+6pl4VxZWidHKEaTXaXy5yixvZ/ep7hLCDhOatdvJuAGaNkBHKe1szD4IdJfLyVig4kskHB9CEgG/onDuR0IkF1l8S82SX8Yr5huS5IIw5HBVI79doOMBh2U+Yrsoz8N94BTL3UrtEZASH+x6hfcJUhB8So+rdrGL8VZFqb+0XQzrdghSFK0TOzwqT3XlwgpoOly1OPt/5D2ZY02zNKbS1z/fMq21NSIx5WS3HMm5S8PseTJfk/em5HVcu2oUeFciyc9tP5CKWWg/26xOIEUjmi3U2MN7rEiH3pPcNktQkSeqYKrx80dYcOpHxHZAPhy69uSjWrDbK0tFr7NLcZsgLzbThDc26L7kqU3AbxvvSGHI7fIudb1mpwwiaRmJZjXC2r8fyd/ZFx/hOonpb5XBBKvcaWjExyrd9bIy9oNM2JJxqjOwedGxkYXjir+zKVJXyhZwiBnntJpNQH1ZXMkww7cNZhA1MhgXyh2kIvM+G761UbInZ7luyKnfFTI950EfYsiyzRZhjxH077uhCyOxXwD5+1oBpkzJcqQg5552WI3iim1ubQNPao7cMHfgT23Q+f3Hspqt439UK6rU8c+k9hSl1BvjoZtN6JwOKplEDUUe54IYGi+ydTP7GatF3Zws5cRTcCHYO8AYY1qpcfdOU4/rfDMZ+vBp3aB4Qt5oizcPPzslXUmWzVeVnOYi/eGOid5V0JonAzDQQXCs4mclZ7ddDzo1szRLhdafKHRstQxhRiBxvCa2xILySQ/5fQSpY4Vy89eNy3ampydM1IVrSGvxADLC6vduZCyug/ijDYoshZnthpxRMgAM1ayT6dD9pLwCuwGlIr+paJXZT6HYMumiecw+O/Z4p3k qVFYMVnN H3+KuCWEK1PBSedQ5Ob+Uogpwbfcg1YAJ7mHv7ppa134w7IsMzxEhq/guCkfLgAZHVJlTP/4e4NCS1VR9BijENGgMHvAL/znjNLIUXGHKTtoObNC3wax7xUdmp73wx5yn2U4sLfJnGVotMd5ZDPF3KQdWnEmgzzaxMH+q1hWB1q/HMRBYgmmMqEgXs7CGkDUVXp/vz4Hx5TnyaDTNQDmZpjp3P4muanzUiTJLKR+aNt2+BNeD/kTbedAVdmgMezaiVPDEx9i88dpVFvVpuok8ntRZSG/hlVm3V6KsJZZmXSmgGOnZwQMlskBUaQ93lQG5agJO4Lr/0GjBXN+8GiAHr3RnBB4wCpyAkUbu72DEv6P2z/WvDrWf1yCZ9v3oPkkWsZ2m6cEFUmmt+qCbxWINFY6wC1yoq2EJ1+Lr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the ext4-es shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct ext4_sb_info. Signed-off-by: Qi Zheng --- fs/ext4/ext4.h | 2 +- fs/ext4/extents_status.c | 21 ++++++++++++--------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 0a2d55faa095..1bd150d454f5 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1651,7 +1651,7 @@ struct ext4_sb_info { __u32 s_csum_seed; /* Reclaim extents from extent status tree */ - struct shrinker s_es_shrinker; + struct shrinker *s_es_shrinker; struct list_head s_es_list; /* List of inodes with reclaimable extents */ long s_es_nr_inode; struct ext4_es_stats s_es_stats; diff --git a/fs/ext4/extents_status.c b/fs/ext4/extents_status.c index 9b5b8951afb4..fea82339f4b4 100644 --- a/fs/ext4/extents_status.c +++ b/fs/ext4/extents_status.c @@ -1596,7 +1596,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, unsigned long nr; struct ext4_sb_info *sbi; - sbi = container_of(shrink, struct ext4_sb_info, s_es_shrinker); + sbi = shrink->private_data; nr = percpu_counter_read_positive(&sbi->s_es_stats.es_stats_shk_cnt); trace_ext4_es_shrink_count(sbi->s_sb, sc->nr_to_scan, nr); return nr; @@ -1605,8 +1605,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, static unsigned long ext4_es_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct ext4_sb_info *sbi = container_of(shrink, - struct ext4_sb_info, s_es_shrinker); + struct ext4_sb_info *sbi = shrink->private_data; int nr_to_scan = sc->nr_to_scan; int ret, nr_shrunk; @@ -1690,15 +1689,19 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi) if (err) goto err3; - sbi->s_es_shrinker.scan_objects = ext4_es_scan; - sbi->s_es_shrinker.count_objects = ext4_es_count; - sbi->s_es_shrinker.seeks = DEFAULT_SEEKS; - err = register_shrinker(&sbi->s_es_shrinker, "ext4-es:%s", + sbi->s_es_shrinker = shrinker_alloc_and_init(ext4_es_count, ext4_es_scan, + 0, DEFAULT_SEEKS, 0, sbi); + if (!sbi->s_es_shrinker) + goto err4; + + err = register_shrinker(sbi->s_es_shrinker, "ext4-es:%s", sbi->s_sb->s_id); if (err) - goto err4; + goto err5; return 0; +err5: + shrinker_free(sbi->s_es_shrinker); err4: percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt); err3: @@ -1716,7 +1719,7 @@ void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi) percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses); percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt); percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt); - unregister_shrinker(&sbi->s_es_shrinker); + unregister_and_free_shrinker(sbi->s_es_shrinker); } /* From patchwork Thu Jun 22 08:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2E11EB64DA for ; Thu, 22 Jun 2023 08:55:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 831968D0006; Thu, 22 Jun 2023 04:55:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E1E28D0001; Thu, 22 Jun 2023 04:55:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 682E78D0006; Thu, 22 Jun 2023 04:55:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5A6A78D0001 for ; Thu, 22 Jun 2023 04:55:59 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3EDD4A016C for ; Thu, 22 Jun 2023 08:55:59 +0000 (UTC) X-FDA: 80929776438.02.ED6CB78 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf14.hostedemail.com (Postfix) with ESMTP id 5BF5B100006 for ; Thu, 22 Jun 2023 08:55:57 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IN1IoI2u; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YV1TxBHonYX3pT+e4dDoFW63Wrih64tWAHWeaspLQMY=; b=hZHLMw7olA49iICfqpmkhRhZVGahJU88KpJenO0s4wI0nUtKIkILcpj6z226djoTGmmUPh tRtLLvvVbL+QDNKQPtrzqXqPzpDKMSljnpCYOb1nXDRxM28+6AtkUYNNMD0RlZAGu0CHX/ KeiWJnMZW15Gvg0k1mR3z5fpUdiMQEw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IN1IoI2u; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424157; a=rsa-sha256; cv=none; b=fpagBVdbP5Bb5fhx2GB3uX/NLABpnT8NsPvl2bi2MlWV5LkECo2yhk9GSRSDzk+3A+8Lhi IXQKOuAFA4tqoy4PFv88F4mnOSNB6LtSvZ3yUm1xq0dXFqPci/R599XzXu3Oyfz+pKQORf nUQ+OQLZ1HLkRItqSBet/MGBViGrPbg= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-54f73ef19e5so678893a12.0 for ; Thu, 22 Jun 2023 01:55:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424156; x=1690016156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YV1TxBHonYX3pT+e4dDoFW63Wrih64tWAHWeaspLQMY=; b=IN1IoI2uOCQQeS9Pc7yMEYFtdbTKp2FDPBFMAyikkB3bBw/FCEgPQpTXBrLGJxS9op OYz9dXXM6oULWR3l3wYKXb5/BI9AkFpryZP4yoepCwR6hfyyNUOfJ8hsO4sBRwfJ49YX gUV9lDqFE5hQ/d7HdmIIwZSFFlm7MsgfCCmjn2SzIPpkrW6AhbhGMDaDZJJIRMhzbnyY 28028P9U8PoGDzKdCZpYkcFtSyvZMpWWRxH6zTertHKnPVPdcLDGdjwoWDesvLV0LnN9 TRCiB1NZo1an6jc3DcyUcaVRTuirz1I2fp4pQQgMnnutBNGuXTEGp5MRQRxr/+etF0+R bUJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424156; x=1690016156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YV1TxBHonYX3pT+e4dDoFW63Wrih64tWAHWeaspLQMY=; b=VmGVxAYJk5TN8umf1oLMRexWQojMGxbUGYG8IJ9HNej2GZcjylyrpthtHVwBrXOwku cp+MAwp5rkSy3SxebXZ8Q8EZXy8AwoSXEwgGnT6+NfGcswj66WIw5x4FqtnTKXl//0m1 kE6wafwRRlsgTeXsQwoTldpNaAhHqwxCIV/y6C+kfkiZzq+CvAI16nJXDPB22XVjLJ21 QmsxuYHE/9mocyiJjNa9/n9n6jqAK4MrjghRkiTPE8OVO2J5AhgFpNE4a9DJ/qe9Bgh4 N/vkP/ATAWYhBEm/Z6PcWjGyvlolCb+8JEIpaBBOY5lpATL1XNrJWz/Gh1GPM8rcOJ5P fHyA== X-Gm-Message-State: AC+VfDxnXyAgWkr8hZl8by58WcVhb62BRvFDxZF7R1z0XzltMFbH+Lhw l9VN50lHQA0c5funJWgMQX8/5A== X-Google-Smtp-Source: ACHHUZ6yC2snRIv6FqrsdGDMg/3uXpE9V6V7JpWahZzREljgaxmC3AuzAW6pVJNk/nQXbwoLHaAThg== X-Received: by 2002:a17:903:2451:b0:1b0:34c6:3bf2 with SMTP id l17-20020a170903245100b001b034c63bf2mr21540756pls.5.1687424156178; Thu, 22 Jun 2023 01:55:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:55:55 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 14/29] jbd2,ext4: dynamically allocate the jbd2-journal shrinker Date: Thu, 22 Jun 2023 16:53:20 +0800 Message-Id: <20230622085335.77010-15-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hsk4888ynrmdnemk8hxaxzhgb3wz6fc4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5BF5B100006 X-HE-Tag: 1687424157-420243 X-HE-Meta: U2FsdGVkX1/FyaokULEyFhqEDE819xrH1LDIvELu3fVIo/UuEQv118EuHmaliwtpAa+djznYFPoDZ3TXrddnx5gKtLjCLUOFG4owVr+cc5QhvNFcgdkTQzm/etaO0iY50RqJii45lS++odT7Arbe+mS0EBANSeLFQAPMQfCeM8Ulo/dOwqDMKJJ1VoKAM01qaUxMKAN53ti7eda2YFjbdUsytBZt0YyyyOSllANbkl0s0TL94xYNmyizQzJEJyTvcp1GD6blJJ7n0J81iRvqNNMiQWKRLmNk5iOWbdIH2mrzNmvurBULo9RE5/fqySxKTmMQcEiiw7cBkQC3A3ypKlYLr331wemNquu/hAZrI8j1T2pk9ieH4Nj6PNC0Cjf1BuZto7HwqNMyHwjTxrxCHRmHxZ5NAgFxDmw5MuMhbJav4Ay0QG3sFGTjksj6Kp3fmD4hvhKu3CB8WTwrWHATguPHyYUm+xdxChIfbJUol4NjWshTtv6vCWDFO0Y07jaf95xC7NeqpW0SHAf6PClpPKjaEdwG00NiZo9OQK7gOjezN5mH9/g76b9rlpcdtoTyCOS57vaZ4eAhU4NwscOZ4VkLwFZXgGhcj8IhaAO2+rv1br+WrP3zn+JXsODGtlxxrrFbLEQKNBtRIAXf2QmbE93qJ80I9F9KVudJaWLBPWRvypmmPdzuBToYKKN0IIMZpcqAzkH9jp1Jp8HUY7YS8UHrMsbU09/R1RsbQ1sdtTNm/NChH2Xz8SjmVOYwpKW6lHqjjSMz/Ev2rbqIe/1LUdqSuU6kY/zAJecFa7tE5hkcvuohMqtfS5I2VwiHVz5AuTmNQOjeb4VCcLSdWhJAuhtz3NtJ821qZKP1iwC/XpOx0uPesYcBHmespHGFkofZXaFFaMzzKDyggO+lsfbRkmglyyMFOgyQyoc16PQBMhVvzLqZth+/TQaIS0xjOWpbzuX+ebFG92u0N3izO4R THHUnx7Q WMxE2HdNp7qjFZfaEUuPqeqXGJhZ2AD1UifgnIVoX32ByqwohAcaeg7NVzdR8fXTbCKjSOE65MTq68UfasPcU0I21qhA9h/pRRbR6VPis7Abza8kjv/7CQGDCz5R3TPlnFeHFUZ41fZ1eRATifn7iYSfpadvxS9sjx3tDPVtejCEhGtrb6vW1kj6kdzi3B68/F5byDeTtfIZAQ7IP7N0zKSteKBs0DqGEms7SbFZ57PuPqJHR2EVvFCu+EMQozeg4Ob86C/0lGpcbHSs1kK/Yvv2bxBS3NsptgWO45Ilpvzpbx9FdBq/61WK5Z9Jhd3l0tNaJHQNdgx11CWZKwWATXJxhR9fIDXqfMfYPZT0e0BTansSwdR+TaPwc9Cy+ucrXLVVg/8IU6zs/iDf6KEpmVwYd5Z/PDUrFLAK7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the jbd2-journal shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct journal_s. Signed-off-by: Qi Zheng --- fs/jbd2/journal.c | 32 +++++++++++++++++++------------- include/linux/jbd2.h | 2 +- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index eee3c0ae349a..92a2f4360b5f 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1301,7 +1301,7 @@ static int jbd2_min_tag_size(void) static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { - journal_t *journal = container_of(shrink, journal_t, j_shrinker); + journal_t *journal = shrink->private_data; unsigned long nr_to_scan = sc->nr_to_scan; unsigned long nr_shrunk; unsigned long count; @@ -1327,7 +1327,7 @@ static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, static unsigned long jbd2_journal_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { - journal_t *journal = container_of(shrink, journal_t, j_shrinker); + journal_t *journal = shrink->private_data; unsigned long count; count = percpu_counter_read_positive(&journal->j_checkpoint_jh_count); @@ -1415,21 +1415,27 @@ static journal_t *journal_init_common(struct block_device *bdev, journal->j_superblock = (journal_superblock_t *)bh->b_data; journal->j_shrink_transaction = NULL; - journal->j_shrinker.scan_objects = jbd2_journal_shrink_scan; - journal->j_shrinker.count_objects = jbd2_journal_shrink_count; - journal->j_shrinker.seeks = DEFAULT_SEEKS; - journal->j_shrinker.batch = journal->j_max_transaction_buffers; if (percpu_counter_init(&journal->j_checkpoint_jh_count, 0, GFP_KERNEL)) goto err_cleanup; - if (register_shrinker(&journal->j_shrinker, "jbd2-journal:(%u:%u)", - MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev))) { - percpu_counter_destroy(&journal->j_checkpoint_jh_count); - goto err_cleanup; - } + journal->j_shrinker = shrinker_alloc_and_init(jbd2_journal_shrink_count, + jbd2_journal_shrink_scan, + journal->j_max_transaction_buffers, + DEFAULT_SEEKS, 0, journal); + if (!journal->j_shrinker) + goto err_shrinker; + + if (register_shrinker(journal->j_shrinker, "jbd2-journal:(%u:%u)", + MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev))) + goto err_register; + return journal; +err_register: + shrinker_free(journal->j_shrinker); +err_shrinker: + percpu_counter_destroy(&journal->j_checkpoint_jh_count); err_cleanup: brelse(journal->j_sb_buffer); kfree(journal->j_wbuf); @@ -2190,9 +2196,9 @@ int jbd2_journal_destroy(journal_t *journal) brelse(journal->j_sb_buffer); } - if (journal->j_shrinker.flags & SHRINKER_REGISTERED) { + if (journal->j_shrinker->flags & SHRINKER_REGISTERED) { percpu_counter_destroy(&journal->j_checkpoint_jh_count); - unregister_shrinker(&journal->j_shrinker); + unregister_and_free_shrinker(journal->j_shrinker); } if (journal->j_proc_entry) jbd2_stats_proc_exit(journal); diff --git a/include/linux/jbd2.h b/include/linux/jbd2.h index 44c298aa58d4..beb4c4586320 100644 --- a/include/linux/jbd2.h +++ b/include/linux/jbd2.h @@ -891,7 +891,7 @@ struct journal_s * Journal head shrinker, reclaim buffer's journal head which * has been written back. */ - struct shrinker j_shrinker; + struct shrinker *j_shrinker; /** * @j_checkpoint_jh_count: From patchwork Thu Jun 22 08:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1F32EB64DB for ; Thu, 22 Jun 2023 08:56:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7201F8D0005; Thu, 22 Jun 2023 04:56:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D0528D0001; Thu, 22 Jun 2023 04:56:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 598628D0005; Thu, 22 Jun 2023 04:56:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4853D8D0001 for ; Thu, 22 Jun 2023 04:56:07 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 151CF40B1C for ; Thu, 22 Jun 2023 08:56:07 +0000 (UTC) X-FDA: 80929776774.16.6757A38 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf16.hostedemail.com (Postfix) with ESMTP id 3B9E8180014 for ; Thu, 22 Jun 2023 08:56:04 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=W9Q0mEFJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424165; a=rsa-sha256; cv=none; b=oQIJf0uP4bwRqAhXgKiLhmLQ9hYV9lgUyr614q3mhFIDnSrjKgxMSEjrQyH1LoCmEdjtOH 53D4hcdU+Q0v35SyreUYPvL7i01bycItXZ7YDtECCvEWgNLGICTfxfv7a8cX1G3Q4iUhVX kWHL1gHFrk0AgkYfE2Y//oZYMka+qrk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=W9Q0mEFJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FWRSgLiVxGu7by+bRdZCMSZgL2CmyzCgjcAmFywfg3g=; b=Y0lKBkgL/Zbdxf2rCafMWOyRWBNXBBKDHrqaCms2wiVnqAhHTaAkzzGNaUqLOCrOqajgNA 4O0kDGKIvDOnRvZIBD9lqjBLcswkz0BlyLY+bA4cYW9Jl06u6uqDvTLYw/+pyEBDO3sb/2 pf8ket5DXcOiBuEDdCNo9qAZ+5bwW04= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9654925ad.1 for ; Thu, 22 Jun 2023 01:56:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424164; x=1690016164; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FWRSgLiVxGu7by+bRdZCMSZgL2CmyzCgjcAmFywfg3g=; b=W9Q0mEFJ4YX/Kxo42ZpDHLFmjt3CYHNqumMzPD6lCEJ/AclBj6ki+9nXx+XP2OFoT9 biokZFjtTO4s6n85EeiNgXVRb8aHb7r2IMBB+HnwNOlBqet/0l5xfJVO+DolK/U7XTWU M277NuxA39AWH+q/lhKoP2vIlehvWLhpgJj/4uGWoGxCITX36OL+YHFtfzDKSSw2iPaX MbP+C5V24hBPOg4LWw9eV/ZehzOW8RyY8zV5pnDOyFwQ720HnrExvRoWQvOw0JMgQMOU XwvMcYncbAVdeNBjwwgU6wHkMwvamAyfgHl3RPNu98UxqyF0R9Sxc8Wx/1hGBgxkRDYX Kkuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424164; x=1690016164; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FWRSgLiVxGu7by+bRdZCMSZgL2CmyzCgjcAmFywfg3g=; b=HgJEogKspnwVpZvGppOeBIehYLkkkMEFbGq60SGlxSNyygydao0YJVfecBns2NwhSf ikLmQGQtITOoElwuulSdhionU7tflLtcBzZcRvlP4Fb2GaitunUkLrXsf8x8ulAcJYfz qAU/oEk4rM9IbiUNA0eodMhF9f31DC+rAxGnJO8WFCS0bqE5VfuXcUxM4xuXWQeWGhOq PB06cxrPGE1EVVWRYO4KLSbs6Fl6WoAZD+kZP0DEylwItDlNmW3571qnNclRPN1jwC5V q5aB60J803OsN2sGqp0/PlEKEAzxiIFgdnHPwjQ/UFbl0WwruzhZkCKbkGKG0Nb6l750 kpgg== X-Gm-Message-State: AC+VfDwJ9sZBnPt3Ze5OwXBNCZ2gyoztKgLDY5gpzfQPqD9VaWLufY6o jRJx47vdn2XvhfSpFslXeuf7ea1dYc9nGxbGsy0= X-Google-Smtp-Source: ACHHUZ4nYFfrGKqZLYUWSywLes9hZrkxqoQAJfwtv5FmoHmc5MLCqSR93ZcAMHhMn41B6ov+vpDAfw== X-Received: by 2002:a17:902:ea01:b0:1b0:4680:60 with SMTP id s1-20020a170902ea0100b001b046800060mr21968987plg.1.1687424164102; Thu, 22 Jun 2023 01:56:04 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.55.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:03 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 15/29] NFSD: dynamically allocate the nfsd-client shrinker Date: Thu, 22 Jun 2023 16:53:21 +0800 Message-Id: <20230622085335.77010-16-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3B9E8180014 X-Stat-Signature: ms3aaqptfm6n4qeqcxaohtz119aiwr1e X-HE-Tag: 1687424164-858578 X-HE-Meta: U2FsdGVkX1/u4HCW+Sp+oCsGPKfQAudE6ws02PTCZyChwByRWRxESB/jpBCrUkbGsrNjRROinahRe5dqYcb+qSwEUUysvAAfcLKKDpRMm7JwEU21Yzg70iuMyG/wsF5+MkQueR62TAbEwIOlM1hWXiFcdYQCkkQboGj2n7SE8cWHqXrraAZaxwRq7zRc7g1ct8CBhRUcXC5Zw/+EiGYzTnJ1YYEVQQo1gjQiE0ujTts/PbY2M+wayxG9ghAVT63dAmCqo4BLY/MzPKrQ47QihYfUiu3FytWCS2W2x9ruZigMunvYvN9tbFoN9LKQVRsyuzB5FX0l2u8+Np3G0mPZEWz2LM0e+/GY0Yuc/jb0rNRl1kMmG+ILYIUU76L+wH51vIMg+Q5zd33OOdMhsaEBNhPbCWghAIdf8D3Rkxp2Kx39TFkXxAsprhZBNyPt0J8ul13gVwo9wWHK6cdhMYyrYCu0g1X6gE07FRNla4iqZtLV622U0PZ5N+kFEOJ6o6jlW5LQU/bbxy5qAeEvSC9T/t3rIzp6iHf03X0FDtWwYYFZrYPFaVIZG4aboHaG1MWY+41FqxSk/uySyYLPcn+hVkGB3F85dKIm3vYU/UQfhKElUQYA8lijJIUmBtolTPO3wpzk7eD6Q0YSQKkMoPxfiH2ImlNGZKYQYzQsatsSdMpS4saaODLXzHkXLgs7YxqNXNGZ73HvAdpCgOTDuZ9evZAHvOHgpwafCvnHgEwYU3+lcgx86jumq+dElsJGbt4U/D8USRzH86/ryUIOzAKEFmofNXmqW4aVRTN12em1mEtRQT8hD20ngKdgzd6OzJ0F5NNkuw+tbgBBD4/LAJjgfc8jPWLoENSn8fSkMZeZw4yZ1ldnoUhq94C/YHruEZRNZV5tunEBMN2290/bvVRpR7Sqm1WhJG6RIy653NElwzWj1V3N9hig8sLoLNK5M9uKTfyKrEVzLTkpMlBSenm EAyJciyn CA0rgLzmsK83Yud3BG+crBgPHj3iOWuFRVxD7BZrjIcuiORzWIIpaCYK6kPQPBt3MpISc5sCw0CpeUyzkGBvYzKCuaYPN0XY0vOru5bBpcj3A42iAyxl3ha74ZUayglgIg+OTuX5q2hqNNP0x1YhH1oONppmpb4iY5wGQWvr3/jA43drTbk6271YML3eQUMcplPNRbMfZPF2GpPAyY6JtZ+oeM9LDKdQvb3BK6rKneQ+MYW+6KUw4xs5f3bRG58olZJKhrtAjykEqQv4H9j1H+1LfKUvR7bziPFu/M7RZ2rx6SOEmFC4M3tr5TCPAwmOG9rA9lO9lcFcil/mWkKkyaNAp9gpl3LVuWbERfpP1YO9HuvH0xxiqe9pCJ8P/XvyVUN87Ok6U0Zc4c7JuKwPtI4AIdIwO97cD9L/a X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the nfsd-client shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct nfsd_net. Signed-off-by: Qi Zheng Acked-by: Chuck Lever --- fs/nfsd/netns.h | 2 +- fs/nfsd/nfs4state.c | 20 ++++++++++++-------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index ec49b200b797..f669444d5336 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -195,7 +195,7 @@ struct nfsd_net { int nfs4_max_clients; atomic_t nfsd_courtesy_clients; - struct shrinker nfsd_client_shrinker; + struct shrinker *nfsd_client_shrinker; struct work_struct nfsd_shrinker_work; }; diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 6e61fa3acaf1..a06184270548 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -4388,8 +4388,7 @@ static unsigned long nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc) { int count; - struct nfsd_net *nn = container_of(shrink, - struct nfsd_net, nfsd_client_shrinker); + struct nfsd_net *nn = shrink->private_data; count = atomic_read(&nn->nfsd_courtesy_clients); if (!count) @@ -8094,14 +8093,19 @@ static int nfs4_state_create_net(struct net *net) INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker); get_net(net); - nn->nfsd_client_shrinker.scan_objects = nfsd4_state_shrinker_scan; - nn->nfsd_client_shrinker.count_objects = nfsd4_state_shrinker_count; - nn->nfsd_client_shrinker.seeks = DEFAULT_SEEKS; - - if (register_shrinker(&nn->nfsd_client_shrinker, "nfsd-client")) + nn->nfsd_client_shrinker = shrinker_alloc_and_init(nfsd4_state_shrinker_count, + nfsd4_state_shrinker_scan, + 0, DEFAULT_SEEKS, 0, + nn); + if (!nn->nfsd_client_shrinker) goto err_shrinker; + + if (register_shrinker(nn->nfsd_client_shrinker, "nfsd-client")) + goto err_register; return 0; +err_register: + shrinker_free(nn->nfsd_client_shrinker); err_shrinker: put_net(net); kfree(nn->sessionid_hashtbl); @@ -8197,7 +8201,7 @@ nfs4_state_shutdown_net(struct net *net) struct list_head *pos, *next, reaplist; struct nfsd_net *nn = net_generic(net, nfsd_net_id); - unregister_shrinker(&nn->nfsd_client_shrinker); + unregister_and_free_shrinker(nn->nfsd_client_shrinker); cancel_work(&nn->nfsd_shrinker_work); cancel_delayed_work_sync(&nn->laundromat_work); locks_end_grace(&nn->nfsd4_manager); From patchwork Thu Jun 22 08:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D85F8EB64DB for ; Thu, 22 Jun 2023 08:56:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D2238D000B; Thu, 22 Jun 2023 04:56:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 681D28D0001; Thu, 22 Jun 2023 04:56:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54A698D000B; Thu, 22 Jun 2023 04:56:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4672E8D0001 for ; Thu, 22 Jun 2023 04:56:15 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0F2C0C0ACD for ; Thu, 22 Jun 2023 08:56:15 +0000 (UTC) X-FDA: 80929777110.22.6D15ED0 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf01.hostedemail.com (Postfix) with ESMTP id 185AF4001A for ; Thu, 22 Jun 2023 08:56:12 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PB56VJEt; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zUNTs2D5J+Px6+8NyTz0IGsKdVlRq3fVdCNXQo6zMeo=; b=hWONiU+cAzHWNEO61XQ0OBTxAZqY5MDOg8SoI/ur075RjQSoQ5S553fZ6Nu3M8HpsoO0Sd MDwiZAlTEis1Ml8MdIMhNEOg/oNP4rZf2oJgnxPWZEcqfM93AyevdZG4zAJbT89aaq7Rvt MNW5HjGUX4090Y7Q+6tTS3naCpvqoNs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PB56VJEt; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424173; a=rsa-sha256; cv=none; b=7E45yINLpX9HW2QqoQeSAsHYqGuUweGtZQV13vGz4gSbVhlb+Tk9Vw4d5IyUdDmugGvT9e YebAbdaqFwTanh1sjghgB2PCXVL/VeB7EgGvVip1DgvnhuR/XMY2ExM8VJsVpVa9Ej+Ae7 IZDfVBATMXSQceSwchxoWsJlusYTNK8= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-543a37c5c03so1091198a12.1 for ; Thu, 22 Jun 2023 01:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424172; x=1690016172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zUNTs2D5J+Px6+8NyTz0IGsKdVlRq3fVdCNXQo6zMeo=; b=PB56VJEtwK39jb6ujoKeOUgYB/PJLrruyxKRjJr7vrytJTdiBm6YHrGem+eoP0b4tN ySO0Fp6iHJx1jYG5cUtED8VGJ9nuR5OzkvfKjRzGq011b3+5HaAEFO0JOZ1JrJyRnWbE zD+J47hPkoBpD2IoCQquNzCqiGwmlgwN3y5YbOJ5VGLqEl2FtVJoITLOXOxAzhFddu+k 6lt/tmVHj10PbE9jdwz/MvuKMpEDpm3abrU9F/6h0dgBlWZ5OjmzxDq5+jKr9MCDVLPh Sp8IUY5vM81KMKjfLSjlTI62E4wEBy9NQ8VxDpQ2xwoZwC4ItSHDc6u/r/EGjzaiDmrG iUiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424172; x=1690016172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zUNTs2D5J+Px6+8NyTz0IGsKdVlRq3fVdCNXQo6zMeo=; b=MUDMGlKDCoXDkJnn6JrQMztp887ceE/EWJiPkzOik4zBbhyqorO+CIT82DJtvs8+/Q HHNh5k0dvvTboeanBKdHMJk2Uw9fQyWUTws52WzBvPWREByiK33Jl33kLKy5/yosIY9O rg7KK1qcboo9YSdhuvNCcKjh+RKs42A+YSMarbxFzQtFRgxs2wtWAZqn3F4wDoTjBO8P 6HGGvWQT7CgJuccoqCKXUoSaU5NOlEWwiT1NTILSMzqp85mbgXgZF4oh8DepAfpCMnDY R9ufU1jYz54frRwpyypF9opiuVTwucIDgHQgbJecxlDfpSLmnr0gXqLgyX3NI3STd6Mg MuEw== X-Gm-Message-State: AC+VfDyVRvenc/VtHQkHQjUFjN8In3szUTBTa0kX4zi8n8C46LFRkTHV Wy4MNyiphtZt0hTBg2Gg315t8xpAVYYdAoJyZaA= X-Google-Smtp-Source: ACHHUZ7RZPPxDHHt9nXlq48w0UysjH5yX+GnVgYNyQdTPpVzG1MBfOvOCSKK+16aR/H5lLV5H8OFJg== X-Received: by 2002:a17:902:f688:b0:1ad:e3a8:3c4 with SMTP id l8-20020a170902f68800b001ade3a803c4mr20681104plg.4.1687424171977; Thu, 22 Jun 2023 01:56:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:11 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 16/29] NFSD: dynamically allocate the nfsd-reply shrinker Date: Thu, 22 Jun 2023 16:53:22 +0800 Message-Id: <20230622085335.77010-17-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 185AF4001A X-Stat-Signature: 5nqaftyj5467yqxufd1nyqmbft4imy8r X-HE-Tag: 1687424172-622356 X-HE-Meta: U2FsdGVkX19BkMRpNN+24NZhdZ+r/LBFn2u8RiWUeZZMqHeTdX2IZ58Mz+PIM8Ar6QHaJ0Jbtp5FYU0zLYP+Ycykd19HQKfs9DJUEBuBMrFMOqME50CetAdlD1ORt6CVbKzlDqCoqBwcz94ht9/D3jM+lCVQvgF9uRTIcRu3NrMdI9w8Ixu0q+KIugeUJv7G+QIJ97BpCOvwz0+GuSSa/ZC/F/4tsYKPWHhfkJXvbGjoy+VoDF9LmzsVL0r7MH0mTqvA+Vo1OOUXH3W9fNTnU4KidnpWA7+DIYO5XNXkD9B9zAJw/EHAVqIQBDFlePg/G/c+BXmXuJX4nl+vM8//2HAsWcnIyMvBUqGW80V+IauppnK9q4pmG01YjRWhuFszl0pg957qqbtMqWTodBDVcoU6b19ak8YkQ/fVX++PLTckyP/kXWnsEv3bwOfLBX7wN/1wXIkC64DUV301qBQQkRSwEaL6ktkoSZyuJ9XwGCW2AwRA2IyA4FZ2UTNH43BTYTwUG+ESKrqU0JBDPXvji3wiul9wTd8QqMEUJs8gU9Xrmd0LZyaADl38lZXPxxKVBK4kIDhX4dhduHesKDhYF5o47eypPSqO+PtgylZsV2fZBQM9tlf8R+Iiym4LASuXjqIYuNBKKlqZr8JxHk074w7HLxWlAhx35SFecABVu4OGHMchOqqL4QnmBwy32rnlylNuta0pmtOuX5h6JfTInoq+ZbRhYFAYTGS1yVInR/Ze6WxCE6RhNC48wNstTkZ/vWD9cQmgPuVPC56I+PeoPIhSiGZUGlXd52ZoxIB7q8pQJXyZ6P6Z8wwLMA2INz7NKKLDK5D4I06N4gY000DtsnZ33IHjwXJxuXQVpUWTPueEnneSwvgA9fp++Vq/cz0l3H+Ma4j+fLcd8oPTAG9Mh0iv4n+0df6SVHSjJSgtLSgu61+EFHwPxYBwvYjGebACZJKj2ODIazCzbV8e7yi mLKS/OkQ EGlVnvmH9B32r/mgr6jIeSpzSBKh3VHaroaQ383x2RXsLGixnL2E+/8GxzEK+FLXro6qJHhdIP8aEBr50W+2ro5IN5p5+jRceoB5dPcBFuJ9Z4xSjbxTFgMT3vhR695EmHwi7TMWOFRQXF6nHePR3rdSwc+BWv47y2jzZfvK/Q4XSMzkoFb3Huw/6C5Z6Va/tgR+AJc1gHAgjrqPpmlFGAZxbQNq7E0ApyGAoqvyDMaNV6SzhuPs+kNM/OfA64JhfZPh1GMrUdili/TdueQzf+UxM4idjDPMyG9OYGyra/Lo6jBF7O6ZUhjBWqcXSqBEAMIGk+J54Ck235VIB77HIc7b1bgW+3qzvYVpnsyrL0Q2Nt/ZwPgEn3TJk1Zy99tRP5jc2iOgEO95psPYPF6bfQiQ4DYfiZQYBTvb3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the nfsd-reply shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct nfsd_net. Signed-off-by: Qi Zheng --- fs/nfsd/netns.h | 2 +- fs/nfsd/nfscache.c | 33 +++++++++++++++++++-------------- 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index f669444d5336..ab303a8b77d5 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -177,7 +177,7 @@ struct nfsd_net { /* size of cache when we saw the longest hash chain */ unsigned int longest_chain_cachesize; - struct shrinker nfsd_reply_cache_shrinker; + struct shrinker *nfsd_reply_cache_shrinker; /* tracking server-to-server copy mounts */ spinlock_t nfsd_ssc_lock; diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c index 041faa13b852..ec33de8e418b 100644 --- a/fs/nfsd/nfscache.c +++ b/fs/nfsd/nfscache.c @@ -173,19 +173,23 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) if (status) goto out_nomem; - nn->nfsd_reply_cache_shrinker.scan_objects = nfsd_reply_cache_scan; - nn->nfsd_reply_cache_shrinker.count_objects = nfsd_reply_cache_count; - nn->nfsd_reply_cache_shrinker.seeks = 1; - status = register_shrinker(&nn->nfsd_reply_cache_shrinker, - "nfsd-reply:%s", nn->nfsd_name); - if (status) - goto out_stats_destroy; - nn->drc_hashtbl = kvzalloc(array_size(hashsize, sizeof(*nn->drc_hashtbl)), GFP_KERNEL); if (!nn->drc_hashtbl) + goto out_stats_destroy; + + nn->nfsd_reply_cache_shrinker = + shrinker_alloc_and_init(nfsd_reply_cache_count, + nfsd_reply_cache_scan, + 0, 1, 0, nn); + if (!nn->nfsd_reply_cache_shrinker) goto out_shrinker; + status = register_shrinker(nn->nfsd_reply_cache_shrinker, + "nfsd-reply:%s", nn->nfsd_name); + if (status) + goto out_register; + for (i = 0; i < hashsize; i++) { INIT_LIST_HEAD(&nn->drc_hashtbl[i].lru_head); spin_lock_init(&nn->drc_hashtbl[i].cache_lock); @@ -193,8 +197,11 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) nn->drc_hashsize = hashsize; return 0; + +out_register: + shrinker_free(nn->nfsd_reply_cache_shrinker); out_shrinker: - unregister_shrinker(&nn->nfsd_reply_cache_shrinker); + kvfree(nn->drc_hashtbl); out_stats_destroy: nfsd_reply_cache_stats_destroy(nn); out_nomem: @@ -207,7 +214,7 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn) struct svc_cacherep *rp; unsigned int i; - unregister_shrinker(&nn->nfsd_reply_cache_shrinker); + unregister_and_free_shrinker(nn->nfsd_reply_cache_shrinker); for (i = 0; i < nn->drc_hashsize; i++) { struct list_head *head = &nn->drc_hashtbl[i].lru_head; @@ -297,8 +304,7 @@ prune_cache_entries(struct nfsd_net *nn) static unsigned long nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) { - struct nfsd_net *nn = container_of(shrink, - struct nfsd_net, nfsd_reply_cache_shrinker); + struct nfsd_net *nn = shrink->private_data; return atomic_read(&nn->num_drc_entries); } @@ -306,8 +312,7 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) static unsigned long nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct nfsd_net *nn = container_of(shrink, - struct nfsd_net, nfsd_reply_cache_shrinker); + struct nfsd_net *nn = shrink->private_data; return prune_cache_entries(nn); } From patchwork Thu Jun 22 08:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CA4DEB64D8 for ; Thu, 22 Jun 2023 08:56:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3017D8D0006; Thu, 22 Jun 2023 04:56:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1A18D0001; Thu, 22 Jun 2023 04:56:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 179CA8D0006; Thu, 22 Jun 2023 04:56:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 061AC8D0001 for ; Thu, 22 Jun 2023 04:56:23 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D0D14140AD5 for ; Thu, 22 Jun 2023 08:56:22 +0000 (UTC) X-FDA: 80929777404.26.8481A8C Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf05.hostedemail.com (Postfix) with ESMTP id 0B1DD100019 for ; Thu, 22 Jun 2023 08:56:20 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="J0xdX/HV"; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424181; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DX6SUDFVJBECNsblFEChPYzpO8xY8cES72+0dMzxuzQ=; b=uf38NhEgUqjHfZhAvsEUWdLfsAc2Rgz8eUVlT5+7UXNxz+yJdJsXN9KabDPoU39iwzOoDW OWbHbRb4gnfKmXr2pf0TdLJ9zJUchwEfTxFHJr3IKU9+JCXHEtEB8uWP+WYutqKbydfU7c fVkPj010uxIcU/Oe3pnnDXvbgSDLz5A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424181; a=rsa-sha256; cv=none; b=sq4oOxPr/NeR5X8WWWCmpCgsrO/O6Fv0+b3vxXICDQZLhq2ntkc+5Z2m6Sh7e6807NX+o8 NIeon//AeQoyxj/xf4nHKB/HU0PveiqD7ntSmuADYB4mG424Lqx9dPcKuAKilMgXtTAGwB WMBjnzJIXmDcYLjamQEIRqO1nQnydCw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="J0xdX/HV"; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-54fd4a7ce25so1096308a12.0 for ; Thu, 22 Jun 2023 01:56:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424180; x=1690016180; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DX6SUDFVJBECNsblFEChPYzpO8xY8cES72+0dMzxuzQ=; b=J0xdX/HVFp5YYG4TfcvtFJvRxLyQLKtNeoVYCgaC2J+nRoYDa/BzwrOrCj094o0cZq 1xReHr0GJ1hygdSXflfCN7O6Wd3k5bnTYjzGehkNWElkKcnI57FpdEY2zo1QIhRivIfW Vn3A6HH2yEtIwY+bYcUHLIMQK61OnnzFACK07hJ1eCTN8lHwm9h5Rchf6mt0xuf3bmH2 YrMt/i+taGwUAGQfmqtCpero+/nADk45X1pPDnlfELgRkE2o8P26K8ugc6xZW6o9l2x2 0OyQF06tkzpnptZeoSrMW5oczymeBmy807qxrTJz07x7dVgWrGVUEsLlXTXBF7IGucek WQiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424180; x=1690016180; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DX6SUDFVJBECNsblFEChPYzpO8xY8cES72+0dMzxuzQ=; b=YfTClnwNXAryHnyrHNlkIM/e53HLZq4rqffUVT134BiOwRM/SIzCZVUtMDslmeA1er 9V/hWmBNnudjXutZHGrCLl5EO/P1PCUj7qQ5onEve05mreRqRu9wL6lltgvwGO9jH7Mi 8Q29hT2rjexzHEix7GAu/Q/tcAxSc++SAzI6FSMCKQqk1PVMLGvwAUqcKj1zd1okSM9F lbxMqAOEJjUR0s9BaPDM/vrxf+OqtaBFUdrf8OdlcQ3omg33wUKOUC0YQGpPZLBQHCFv vLX+POQ67btaNn/2vyho+duwegWPDyJVpc86odLwCwnsT7AdgbMr8hpY+M/TbNjEUTE4 7wiw== X-Gm-Message-State: AC+VfDzbl00deHST7F4W8yknMFyXG4xs3T0gYuA353NYxhodbsXrpc4D zomIe3se6xvXJDV1E9c0DbOmIQ== X-Google-Smtp-Source: ACHHUZ6cRwGFUE7wdLf8WreSOUecoWuLPpHsSO23aKngJIhhKsUdOoFlGAyrZhuJGN3RD5MvXrHaNg== X-Received: by 2002:a17:903:2452:b0:1a9:581b:fbaa with SMTP id l18-20020a170903245200b001a9581bfbaamr20887740pls.2.1687424179969; Thu, 22 Jun 2023 01:56:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:19 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 17/29] xfs: dynamically allocate the xfs-buf shrinker Date: Thu, 22 Jun 2023 16:53:23 +0800 Message-Id: <20230622085335.77010-18-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B1DD100019 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ox9rmruhdb1z5s8ndf8d7mnaoz3enys8 X-HE-Tag: 1687424180-994749 X-HE-Meta: U2FsdGVkX18NxKpHLRidrgN7757q/qFfGeW3uSk+6DwXBYDY0Q5Ooho9UabafTDj2ViGbn7bBePsaSyu4vL5xrTI7dZm/o1aDGwPDPk8Khnoe2Btf4xcFChimFkoGpFFWtF0mtJyBRCAhGU/PBhP5RtTVbbP3PBIyCenysajsGuVwbZDJOgobaI04gqs+RjYpSSnHoRPaCtFuKsd/ToR3g/Fb6DV1q8g7lg5PV/DnynFwKBxHkVVptX63/7Fk4I10AUJYexP1U5zS2A+RBRH1RVakCuTNlcHfwjGDUtj2oA4oiLKhMiVOM6QWM3eSTGwckGAT3QEv8tdDdPtTOrFPdUP6UHNDp6U+4wA0YWL7zzKHAuNn1OIzb+m3RAhyhDmpaWMF7D9Dya4hRLvpodeFs7IQO+mmduNAzTYDaTse0N5/I775AjbBqeZMTIOfWgSkm13Rr2qElDU1kDzsVeWe+TsjS7P3RUfqQLYkIKOULvOi5UAxKtJKKXcwRnOMq6tbrMB4f7ljHjqy2gkc22a9EHdNjJDzdYUthLfN619HqsEiRLaFCh3bYODF3n/XoVnb3OB7eA6YNT4KD+lOP2UN7g24ReQDEXJFk2b+4pEHfCCpK0DCJ2tuDGIjnI/mkKdKRAIwE3JxPFjCsW3EEpeZGrm0RRaD4AePrg3htuwkckPYJF4OxzCnwpstf0GJGt8cj7V7aQqyzaBsTqPgQySw5m0As3J00RhE8ojnADEuuiPTRn9tjCZQZjm7QxOtn2+A8TvK7BSkYOHunjsSn5NWJQQehZmUPeNy87dR4f5r6PDeBUfDbgUO20CydUk2C1PLtniRhKAXQvr7J3SJBj14MLpwNOf3bBKQK9pEOXucQ6Gl2Fr+Vtj8s2OA1Abj65Vo5xsPQNJZqa7Uf/nR5mnYQk8P85S9K+cbxM2R+L9t04sEKEDldBOo7McJ+5JyP6Ox5adkOrLICoEs67zZbA GMXFlEF0 Oth3WTjHqLJzhh2hhNJAydWt/M4q/C1bb8v8q5q1X6ICPbbdJTrXAkgifS3LLYFN8lWRdr2wUbggS1Fy3UBbCAqVtY8EbpCcnjrxf1/K3A6KhL/1QV9O1zkNJKbW0ZdAxC+9EEl81FN8xX3psGRbaW9lJGylwBD458gaSlM7Q9nLmg9XlHSBCv7JfpGlTfXgtBR5btrkW8DyNUJ+P23QSGRVvpmJDfIXD3gETsMq3TSXnWW9Y6W/gIudQO/F4/mju1eoaa9PM9Zl2X0vzrC5GqIcVhOEuCIGeWIyPn8AUUiqxJZIqu5yFweRzQPz8EanA3Fgt+rLf0JkeIVqMSLdjyy3Pttj68L0B/aKV/TF/JazxjN8jbQQKEB7ZpMWD38hrOYr3W/EwHiLfDcN9Joomvd/Ka7fe2opyaJIn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the xfs-buf shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct xfs_buftarg. Signed-off-by: Qi Zheng --- fs/xfs/xfs_buf.c | 25 ++++++++++++++----------- fs/xfs/xfs_buf.h | 2 +- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 15d1e5a7c2d3..6657d285d26f 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1906,8 +1906,7 @@ xfs_buftarg_shrink_scan( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_buftarg *btp = container_of(shrink, - struct xfs_buftarg, bt_shrinker); + struct xfs_buftarg *btp = shrink->private_data; LIST_HEAD(dispose); unsigned long freed; @@ -1929,8 +1928,7 @@ xfs_buftarg_shrink_count( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_buftarg *btp = container_of(shrink, - struct xfs_buftarg, bt_shrinker); + struct xfs_buftarg *btp = shrink->private_data; return list_lru_shrink_count(&btp->bt_lru, sc); } @@ -1938,7 +1936,7 @@ void xfs_free_buftarg( struct xfs_buftarg *btp) { - unregister_shrinker(&btp->bt_shrinker); + unregister_and_free_shrinker(btp->bt_shrinker); ASSERT(percpu_counter_sum(&btp->bt_io_count) == 0); percpu_counter_destroy(&btp->bt_io_count); list_lru_destroy(&btp->bt_lru); @@ -2021,15 +2019,20 @@ xfs_alloc_buftarg( if (percpu_counter_init(&btp->bt_io_count, 0, GFP_KERNEL)) goto error_lru; - btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count; - btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan; - btp->bt_shrinker.seeks = DEFAULT_SEEKS; - btp->bt_shrinker.flags = SHRINKER_NUMA_AWARE; - if (register_shrinker(&btp->bt_shrinker, "xfs-buf:%s", - mp->m_super->s_id)) + btp->bt_shrinker = shrinker_alloc_and_init(xfs_buftarg_shrink_count, + xfs_buftarg_shrink_scan, + 0, DEFAULT_SEEKS, + SHRINKER_NUMA_AWARE, btp); + if (!btp->bt_shrinker) goto error_pcpu; + + if (register_shrinker(btp->bt_shrinker, "xfs-buf:%s", + mp->m_super->s_id)) + goto error_shrinker; return btp; +error_shrinker: + shrinker_free(btp->bt_shrinker); error_pcpu: percpu_counter_destroy(&btp->bt_io_count); error_lru: diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h index 549c60942208..4e6969a675f7 100644 --- a/fs/xfs/xfs_buf.h +++ b/fs/xfs/xfs_buf.h @@ -102,7 +102,7 @@ typedef struct xfs_buftarg { size_t bt_logical_sectormask; /* LRU control structures */ - struct shrinker bt_shrinker; + struct shrinker *bt_shrinker; struct list_lru bt_lru; struct percpu_counter bt_io_count; From patchwork Thu Jun 22 08:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C858EB64D8 for ; Thu, 22 Jun 2023 08:56:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 105DF8D000C; Thu, 22 Jun 2023 04:56:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B6F78D0001; Thu, 22 Jun 2023 04:56:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC0508D000C; Thu, 22 Jun 2023 04:56:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DE5898D0001 for ; Thu, 22 Jun 2023 04:56:30 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B24F51A0B25 for ; Thu, 22 Jun 2023 08:56:30 +0000 (UTC) X-FDA: 80929777740.02.6E4D2D6 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf02.hostedemail.com (Postfix) with ESMTP id E64A48000E for ; Thu, 22 Jun 2023 08:56:28 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RR2ZBd6H; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tTZEGaIAEFBjvotA1NXyBA/23sGgDmW1pgUANJLmL6g=; b=Z2aIDzF5CxmdRuNrdQzqtFdRa3JREhb1cvE8kDQXf6ZGoj/hFxlWteSruQJ8OhgB0W4HZ/ SMHmdd6iy3SgivQekMMpJR+QkcRJKAMwyTuaX+6Vb/IuZSlPM26CfSL3BnBy1jj1Dqagk6 83AZkBdfHMH+RelzmBGSskbFv9sMQ6M= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RR2ZBd6H; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424189; a=rsa-sha256; cv=none; b=zrwAH1tVXu1inBRDAC0m4jZ9Swsu9kFycl9nF0wdkNH7dWwbetYpgsnTUmUwoeRqlV2Awu LzLrozqI8bJem+Vhqr+Vq95/7QfXXBRsSOM2xBtn0ZdUE9PyL5hwQtaTw2rAy2bINGiNMz YIcFo9CKKs8XM1H1ImmTUNTd8DIzBdk= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1b515ec39feso12621895ad.0 for ; Thu, 22 Jun 2023 01:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424188; x=1690016188; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tTZEGaIAEFBjvotA1NXyBA/23sGgDmW1pgUANJLmL6g=; b=RR2ZBd6H05awxDqDqVthUnK8c2aiJ3/40MwwmbAPlBGQiSswQkBwcxNT/M8cNWLnOA 82Up+za+KnH3T12j6kLa8MWMs1OLNz27XFTdAlPbZ38sAlCSEZeBeei4jIEF3DOGcfFf sqLx7qpbJbMhGZOLWmDQDC3qBDYjKQCUQ+pHZmp4+zwVYsBG9SEPO8LfQRHMRrJmIPZH mEo73QmcjA4JR+vhCDbl2LS55wzlx82Sg5Hmni0lGgyYmJGvSzc9k9c1k7gD3cDGUkxo aNJG0MDIsuR+BfQGSCBSsk1/g4GTm7BXhgoQS8izvkkK6lhK2DzzoNDI+34/82OB/BW3 znuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424188; x=1690016188; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tTZEGaIAEFBjvotA1NXyBA/23sGgDmW1pgUANJLmL6g=; b=avPqUzpFt3TQR5Wk9yoIxGw/zFehIgkfLr1e96+Kclp7qusME8okWqpLZOTGiPF1BT Wd7t1cLa37jeT8pVOPtgBnBzTq+F8tLcPx5M13wd4r7VsUUW7KBEIA4Z+uQ7dv8CTeei 2r13p0DFN5Yccv380xbzqBlCAG8oK0DpjYOLCX2NtsdnDb3PR/6AwrOs9dlFpzQbZky8 SZ0yUuBYzBVxa37Oj7K9n/uECErqHrPf0b9fE4Oqhg6vJ5XzcGZ9hSFL8BNsY518TPkH tumxIA8DSxiEG3jkqEnOiPuMKCJwTnXJxWa8RBiYkUiaELe6otnAoW3Y4efX5aL7ZlaM YzSg== X-Gm-Message-State: AC+VfDyeoad5ROseMMshlyucsAfejOJwz5BXmkn7AJxpB2YaWA9N7xKQ tsZ3daziO1domXRAmwtrNFw+nw== X-Google-Smtp-Source: ACHHUZ6OWalxJOPaEyVlek6DUPm2brDmU3Y4XuKo9voozhXt0KV98CJSzaGxqKjpiLWREy1Rn2GjcA== X-Received: by 2002:a17:902:dac6:b0:1ac:656f:a68d with SMTP id q6-20020a170902dac600b001ac656fa68dmr21469547plx.4.1687424187780; Thu, 22 Jun 2023 01:56:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:27 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 18/29] xfs: dynamically allocate the xfs-inodegc shrinker Date: Thu, 22 Jun 2023 16:53:24 +0800 Message-Id: <20230622085335.77010-19-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E64A48000E X-Stat-Signature: rsn934zn8ze4tfs5wo4poogiwnozfkf6 X-Rspam-User: X-HE-Tag: 1687424188-705860 X-HE-Meta: U2FsdGVkX19knil8zyjxcQEekH0LVR9n6X8G618xtOfBBD8SMe/i4weDIEDiS8+KwdBe/0Wi0GVrRG4yG/LHYne5K+4ayI4A385n8r2FtViepJWY65n/XPYU2rwwCqDOUVmdz02WcTzqyV3JF80g/TDzK1S6Xyd3dbAQm8DNsRgd8SeSIRegS9TKkcTiWIRdRUv38MinDqR5hKeG8NtuxRobzwDD/wYEam7vxd6EyGO21Q7l8H1VOOZ/QbrDyHNQL1lFDPUs9Idr6v9LQdDLZEKFdxWEHumjvNCRJ1/kCN5T8soGAgNBEYeXVBBGvMcw77oI2KmtqJ4HtXuunJ33/UpFUr9Q1B28pKBiQ4DdFNG/O4NT2rIGcQ8uYw8lW3Nyr1cyHsPmbM02XK3EZLnjuUdIySozLdsGguUoMDgJGY1jPcet3xjcTW1Y956OM244l0eiQcui4Pq34FRA9RCTZfdd3Zkh2OuYO1NYcgduPAzAaVc5se6pCxiB9IOGz8tnON8+9x5Dj1IQDjBkhcTeJH7WLKUgujYpeCVf5Mbji4mAn172MkUPDxsAspZtiLz5HadvJSIJN/MsZMRr02lGWRiM2I124Y/OCsoNXQvHaiREncAI/4/bcgMnugSQqdIxXZ+mzMqafuW8mBM9FOhJqjYxNRjeUIg8A/AnH6RjT1490DG08ZyMQiA+0E2UFO257KI+iyv7mC/KN8KjUMCH6eeeRbXUVem3aJDHBTIOfEfiI7ZTRHqsDeIO66cJXFL1Sdko7daezfzoCQSjHiijD7hrpMFpj4A7OUjrErQmGhZN6EafggpwhlhJCQLfrSkoyhA/H2jJ0jSbNec/c/JWkcandYVFUQ8pzg6tM/MJvg6UzASxpJ4aajk+ukrNDb/RGo1cq7NpIqkyqiklIQjuDFViiKCftWBPLxgzyiAKEHKuhoePxPr87ATMRbHXNuO5THrPUHTo+gtug4DgQoa x1DKflTO tsZx+92MPTxM4bQdlsjauczGZqZu3oTijVB9omnSRVBYZxhhel8uOISErYuKzc02kL9G5ud6BGbMPpMx+nzXTwBR2CFieUSGpuRmWTJalfKLYRn7ZLrEZpnuut8qJUqlfyaWH6ygzFWvbUN6QAIMLWHG8SFIqpjcgCnOdEQGMfPS+77PIWa4JWMxZjKmoGnxcKP4PFeaNUCCP1oLLCrRYQNwP4MyU9ipa6zv8RozXEcqwV0cBcyvzHlwPbyqNYeji25f9ssvAC3YaK+arxT28TEb9lUeny5FYDG06atJyJY6UBn02wGwqa6zM7zxdzeOvJashQuXP70RvtvZOyK2jlKXyEabACinpJCpopsFm9twQu3Jqc+K8ZZDNG6DOq6IR6RpD/9dztxWFN6TAwxSGVyrDs+T0HMq/0vud X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the xfs-inodegc shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct xfs_mount. Signed-off-by: Qi Zheng --- fs/xfs/xfs_icache.c | 27 ++++++++++++++++----------- fs/xfs/xfs_mount.c | 4 ++-- fs/xfs/xfs_mount.h | 2 +- 3 files changed, 19 insertions(+), 14 deletions(-) diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 453890942d9f..1ef0c9fa57de 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -2225,8 +2225,7 @@ xfs_inodegc_shrinker_count( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_mount *mp = container_of(shrink, struct xfs_mount, - m_inodegc_shrinker); + struct xfs_mount *mp = shrink->private_data; struct xfs_inodegc *gc; int cpu; @@ -2247,8 +2246,7 @@ xfs_inodegc_shrinker_scan( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_mount *mp = container_of(shrink, struct xfs_mount, - m_inodegc_shrinker); + struct xfs_mount *mp = shrink->private_data; struct xfs_inodegc *gc; int cpu; bool no_items = true; @@ -2284,13 +2282,20 @@ int xfs_inodegc_register_shrinker( struct xfs_mount *mp) { - struct shrinker *shrink = &mp->m_inodegc_shrinker; + int ret; - shrink->count_objects = xfs_inodegc_shrinker_count; - shrink->scan_objects = xfs_inodegc_shrinker_scan; - shrink->seeks = 0; - shrink->flags = SHRINKER_NONSLAB; - shrink->batch = XFS_INODEGC_SHRINKER_BATCH; + mp->m_inodegc_shrinker = + shrinker_alloc_and_init(xfs_inodegc_shrinker_count, + xfs_inodegc_shrinker_scan, + XFS_INODEGC_SHRINKER_BATCH, + 0, SHRINKER_NONSLAB, mp); + if (!mp->m_inodegc_shrinker) + return -ENOMEM; + + ret = register_shrinker(mp->m_inodegc_shrinker, "xfs-inodegc:%s", + mp->m_super->s_id); + if (ret) + shrinker_free(mp->m_inodegc_shrinker); - return register_shrinker(shrink, "xfs-inodegc:%s", mp->m_super->s_id); + return ret; } diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c index fb87ffb48f7f..67286859ad34 100644 --- a/fs/xfs/xfs_mount.c +++ b/fs/xfs/xfs_mount.c @@ -1018,7 +1018,7 @@ xfs_mountfs( out_log_dealloc: xfs_log_mount_cancel(mp); out_inodegc_shrinker: - unregister_shrinker(&mp->m_inodegc_shrinker); + unregister_and_free_shrinker(mp->m_inodegc_shrinker); out_fail_wait: if (mp->m_logdev_targp && mp->m_logdev_targp != mp->m_ddev_targp) xfs_buftarg_drain(mp->m_logdev_targp); @@ -1100,7 +1100,7 @@ xfs_unmountfs( #if defined(DEBUG) xfs_errortag_clearall(mp); #endif - unregister_shrinker(&mp->m_inodegc_shrinker); + unregister_and_free_shrinker(mp->m_inodegc_shrinker); xfs_free_perag(mp); xfs_errortag_del(mp); diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h index e2866e7fa60c..562c294ca08e 100644 --- a/fs/xfs/xfs_mount.h +++ b/fs/xfs/xfs_mount.h @@ -217,7 +217,7 @@ typedef struct xfs_mount { atomic_t m_agirotor; /* last ag dir inode alloced */ /* Memory shrinker to throttle and reprioritize inodegc */ - struct shrinker m_inodegc_shrinker; + struct shrinker *m_inodegc_shrinker; /* * Workqueue item so that we can coalesce multiple inode flush attempts * into a single flush. From patchwork Thu Jun 22 08:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B1EBC001DF for ; Thu, 22 Jun 2023 08:56:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C8EF8D0002; Thu, 22 Jun 2023 04:56:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 179148D0001; Thu, 22 Jun 2023 04:56:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 041368D0002; Thu, 22 Jun 2023 04:56:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E6AFE8D0001 for ; Thu, 22 Jun 2023 04:56:38 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C09401A0B20 for ; Thu, 22 Jun 2023 08:56:38 +0000 (UTC) X-FDA: 80929778076.06.290B910 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf24.hostedemail.com (Postfix) with ESMTP id D1BDB180013 for ; Thu, 22 Jun 2023 08:56:36 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="Ekn/zV8B"; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424196; a=rsa-sha256; cv=none; b=X9EIKwURb1+gKjhun7lWJvDEFk7fQy4F7UXTph+sgTKOcfCvEsFSYzk3oFT6NymqoFYkA7 dPQjtgv6qziZ6Ue7Oqjxa4dkI2MBy5hDNltN+R65y8Z/Hh+c82TJqYAvluYaV/QxVripAa JPe/Ufq1Bv8gzL6QJUEDQKif9FEo/4o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="Ekn/zV8B"; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424196; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tuTRU39Llc4f1pG+4j4EE0WoT0ftJSPXyoG6lJxyplk=; b=VkfN0shzVMjHoAD9vGcNYXFFYzVy3EgwIdGbkc7D7dYIRCuvpGKeCd9Lu/jZ1ib7/vgjCu i6LL6DeGrzt3FCI3jg7PSA1DqjkwqK6KxE+D+AwDOJoymiNjyvklj65dEfOUkUEKksbC09 Fg3ifOsnLuTNl487exZ5xhJ1beJtk+I= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-54f73ef19e5so678963a12.0 for ; Thu, 22 Jun 2023 01:56:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424196; x=1690016196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tuTRU39Llc4f1pG+4j4EE0WoT0ftJSPXyoG6lJxyplk=; b=Ekn/zV8B3Pykd2GrPoTdVnpJwk5evd2tyUP0vKsA0sao3e0fC5CJiY4BTrhaxMXIHV uTPkEPgehKIEn13/c91Rf6hAUh1pL4moe2PR0AHRF6Kdv1EoYZyskenz8RKeyckC9Itb y+HACd1zVUdu9ru2Mkg+skG1k3PTY0I37XiwGKuN4fZctDPgzdbV+AVCjywsN0HsaZ69 Cuv8lvpq3tEmZe+xhsg7ur6ivxEYcTfKMKXUZmg3vRSFd562H9J1GbGm5bH6oFZpkxLe LAGfy+lFWFcKu0xxXlVXi22gjm4qtT4oNXfdGZLZ8KWmVvCWym8e4Apiabz+y3j9a60o NyUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424196; x=1690016196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tuTRU39Llc4f1pG+4j4EE0WoT0ftJSPXyoG6lJxyplk=; b=VjMRDOG/fLld5aHFPumAMJgWCfZCqLupWEU4JKQOgu67okcBMqa4i+LK8sbFlqA6K1 T3Q0moEBqFyPZQUmmQBk24M5xR+o1JAhHXcDQWxB4T6ioJX/21GV/pwpORIlWRQ2fiLo EMPi0pPOAopKlR9YyK9lPoGtXrSY6KBXE6esc1BlzaHYLYdcoaJMJUR9HsIjnvRY2hp5 vKciwZKl6toePTeO1frkpCRzqt2Ba3KZpfudWeMT8CQHD3gVutKc4ixZvygsrZ8U5A7T IP36keGoIN6K2kAV26jmTIXaHksAp+Fc3chQnnl29em0Z9ZS3JiFTnWjQ8xmSP6pQurF lmIA== X-Gm-Message-State: AC+VfDyIsgJ5nBT0luLwO2VGN4jn5yxQCMvLfZ1BRZvC4rTZ4JnJxgKk zJ3XnJthFqq2ieo1y/inXnyu7Q== X-Google-Smtp-Source: ACHHUZ67Tf+k1CheKb86ycYHhx5x/FfiuFJZEOT7coQiqeyoaYPxA8zsjGzghk4F71F8kxwZIRTJAQ== X-Received: by 2002:a17:902:ecc6:b0:1ae:1364:6086 with SMTP id a6-20020a170902ecc600b001ae13646086mr21603647plh.2.1687424195747; Thu, 22 Jun 2023 01:56:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:35 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 19/29] xfs: dynamically allocate the xfs-qm shrinker Date: Thu, 22 Jun 2023 16:53:25 +0800 Message-Id: <20230622085335.77010-20-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D1BDB180013 X-Stat-Signature: cytcm5s5f5trkcckso7nz4h8apwfbfkd X-Rspam-User: X-HE-Tag: 1687424196-949327 X-HE-Meta: U2FsdGVkX19mv7KxeYHy+ELC3A9giFvbaNOHRKOLIIe+f4wm1ha12ecae9VasXXOuv72AL6JFqJMxkRw34JFB9MetW5FylWoCHWlpSIHQcyoLsj0wFVu0KABo/JedjpJmWBtuOUdVdXZE8MbhZh6BwcD6F5GEN/Uog2pHZ9/GU7tai0FTXveKoJMQdyv4zeoMSWjPlPz46o4bR6WhJ8tYUUR7XjsZr59if5POhN5mW80Piwn2uPh6Dgfc3EdGCjacExIPYjPKhLlzsQgQFnnJUc86ZLN/C+M2P9RHwSbZgXhrF3+Y6XJ8Rq6tCR+4nrDDGPvUsNuIFuqQ/H4H5Tv7L+9UgWsLsZFXp9GuibSQmo4texHWc11G2t6eeF7pKGEDvDssWeDyDTebZVbxEUYWsFWGNv1EWWBH7Bdpa6OTuDSRaXcGYwWXEi4pUSPbgQuQMM3DvosS08005AWJ5+UnCeprlTtUmgKFBAgwSESQgTljeX6cumSPeUNTKTKhT2X1jeHyjX+CNx/IVUQSicOPLVEIsgLnxvatZpVseuuGOwhvhKO6dh27dJRv3bXxbwRucj0QCokytacHUC1OXIIe8+fXqrfgJRqSKwpMrvtbjJ68Q34ce0yd2aZSoGStgtk+IgNb2diaUhb9/2UTS/4U/WfoU63BWnmqWWhFVqYtF842WXL0Jn4hhbATY45/3OR3WU0YUijvRMNXcFqT9QnKUB8Ci4gVEMYbWabjqnCddps0ApRv6li1sQM1tJJKxoqi29tDIqmOFbHD8FgzsW6KmuNrM28q1nhNQfE77lFM2CRI8KB08dSHbtxhUOw6FbMB1P0ghZhi4v6l8iCclHjvA3+r1wpiREAmGg5H2zUzjmgFME5vF4t+sBZqItnBv1cMEFfUJgzyhHl0tEmVd894HRbOJVoGfBn1iCpnisy5B5Rcr0Fm69tcnRnGU2oTIWu4G0BhLHWZZfbx5ffan+ mUhoWRiM PGc4VKDhT2QuDNZb70uUovxa0ndfNokUtt8BaXagcLfpNQkKHYa7hV+Yhu6YztDDEp2zIrh48+4J84RvQSK8m8qCOYYcTX0zPgacYMyJ4Sz5C9GiPJd9GHCLifN9gZisKquSkoQTDFpHsOkQohBYVjepXV9ph6n9EYk9dWcRH5/lUPZNXtWnK1o83pRDeEJMRwGZQp/zYTkkohfdm8g5cCVTWc8JHjixPdQ4PJmxn/AefpOywp9ejul7MBo/tGhJAvOrFmgqqJ11+CqpBQobPf6zEELvApnpPszfWwLRHXztAGYZfCqPv2dg4FVLlQySyziD73iTIU/jq2PHr2fLBSKPYJQEkDLkIaKpKla6/Mm9Vc4NtCv6CEPz42Zr5CRGHBBn9b5ajFgGgPIPscaZxz6bmx3wQ49Q0ukvQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the xfs-qm shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct xfs_quotainfo. Signed-off-by: Qi Zheng --- fs/xfs/xfs_qm.c | 24 +++++++++++++----------- fs/xfs/xfs_qm.h | 2 +- 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 6abcc34fafd8..b15320d469cc 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -504,8 +504,7 @@ xfs_qm_shrink_scan( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_quotainfo *qi = container_of(shrink, - struct xfs_quotainfo, qi_shrinker); + struct xfs_quotainfo *qi = shrink->private_data; struct xfs_qm_isolate isol; unsigned long freed; int error; @@ -539,8 +538,7 @@ xfs_qm_shrink_count( struct shrinker *shrink, struct shrink_control *sc) { - struct xfs_quotainfo *qi = container_of(shrink, - struct xfs_quotainfo, qi_shrinker); + struct xfs_quotainfo *qi = shrink->private_data; return list_lru_shrink_count(&qi->qi_lru, sc); } @@ -680,18 +678,22 @@ xfs_qm_init_quotainfo( if (XFS_IS_PQUOTA_ON(mp)) xfs_qm_set_defquota(mp, XFS_DQTYPE_PROJ, qinf); - qinf->qi_shrinker.count_objects = xfs_qm_shrink_count; - qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan; - qinf->qi_shrinker.seeks = DEFAULT_SEEKS; - qinf->qi_shrinker.flags = SHRINKER_NUMA_AWARE; + qinf->qi_shrinker = shrinker_alloc_and_init(xfs_qm_shrink_count, + xfs_qm_shrink_scan, + 0, DEFAULT_SEEKS, + SHRINKER_NUMA_AWARE, qinf); + if (!qinf->qi_shrinker) + goto out_free_inos; - error = register_shrinker(&qinf->qi_shrinker, "xfs-qm:%s", + error = register_shrinker(qinf->qi_shrinker, "xfs-qm:%s", mp->m_super->s_id); if (error) - goto out_free_inos; + goto out_shrinker; return 0; +out_shrinker: + shrinker_free(qinf->qi_shrinker); out_free_inos: mutex_destroy(&qinf->qi_quotaofflock); mutex_destroy(&qinf->qi_tree_lock); @@ -718,7 +720,7 @@ xfs_qm_destroy_quotainfo( qi = mp->m_quotainfo; ASSERT(qi != NULL); - unregister_shrinker(&qi->qi_shrinker); + unregister_and_free_shrinker(qi->qi_shrinker); list_lru_destroy(&qi->qi_lru); xfs_qm_destroy_quotainos(qi); mutex_destroy(&qi->qi_tree_lock); diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h index 9683f0457d19..d5c9fc4ba591 100644 --- a/fs/xfs/xfs_qm.h +++ b/fs/xfs/xfs_qm.h @@ -63,7 +63,7 @@ struct xfs_quotainfo { struct xfs_def_quota qi_usr_default; struct xfs_def_quota qi_grp_default; struct xfs_def_quota qi_prj_default; - struct shrinker qi_shrinker; + struct shrinker *qi_shrinker; /* Minimum and maximum quota expiration timestamp values. */ time64_t qi_expiry_min; From patchwork Thu Jun 22 08:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44E6AEB64D8 for ; Thu, 22 Jun 2023 08:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D89548D000D; Thu, 22 Jun 2023 04:56:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D38DC8D0001; Thu, 22 Jun 2023 04:56:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C022E8D000D; Thu, 22 Jun 2023 04:56:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ADE238D0001 for ; Thu, 22 Jun 2023 04:56:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8F2C512010E for ; Thu, 22 Jun 2023 08:56:46 +0000 (UTC) X-FDA: 80929778412.06.B9BB00B Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf19.hostedemail.com (Postfix) with ESMTP id AC2C21A0007 for ; Thu, 22 Jun 2023 08:56:44 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=E8DR3f2H; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uYJ0Isr1XZATX6JGPvTVuhl3oDZVpp/hY4/4gwVGW9E=; b=qzWs3RE+d52HXKZUY0bqwsx35FJ7b6NamigfiJH6THp90KFPMgGhD1mRVZJUgtGsm+wjBa FqSj5dS1o5muifOPTtJIflAPxCHzh7Be5WACxmx8yiHgrl8FnaSgHDzmChry/QmGNAW08x SotPz9THcu90EAIl9Sjfw8cH81+VxQI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424204; a=rsa-sha256; cv=none; b=1pqIGAp2SeqhYapYqgJMI68vt4MVuQ8o97vlZYl3rUdXEs2ibGnd4BcR+s+NyAp7k4a2Jk HlfM1w0t6w7onYpVmt05Nb8DmT5dv0RD6PvnhyU6Q5n1JRX1xQ2G8AbgoQeNwOGiejK6bv gfxDRkyWQCNr85qdOsnN4PtHZeYqSbo= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=E8DR3f2H; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1b693afe799so2466145ad.1 for ; Thu, 22 Jun 2023 01:56:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424203; x=1690016203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uYJ0Isr1XZATX6JGPvTVuhl3oDZVpp/hY4/4gwVGW9E=; b=E8DR3f2Hoat1dr+eWUDbHZgFwAog/Y14Fyr901nadnjVcnG4wi/dzsYXue5lsR1fMf J4lGKXcngL6tZ98mDRA+meQDFEQqI8M8C2U7Jzsh+P5pb02lqX9x9oPIGG3MqGtcEQBd gtcaY6d5Wvcszm2wAYU1/GbNiI4/W6jf6+osdVtp14uPtbwdSj3xBB3cuLupxVd/0Bva 3aNVjboq1g3R8QF56ponf3Z0k9HzBLbK6msqxIkke8ZOuuk+1QHFYDcPM2BRj2r4DT2+ yfbsCvnlQ4hLgF8msig/7Mg+peyLnaUmJHbCBGncbsN2mdvtqIXKdXSMs3btfn7dPCo7 J6Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424203; x=1690016203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uYJ0Isr1XZATX6JGPvTVuhl3oDZVpp/hY4/4gwVGW9E=; b=K6rnnGK6eJw7GYG/jg42D76bMQKnZ+3jQFKMGO9BKqx7RugzMSzr5/HM3WEUmIlLpE 5CmeHP1BaeuQ5CGCnyD/qxG/6GWUwnig8pPCKVjoXeS1uqitVpF0gdNER1BUvm6EYtgd +bocJZHrqPpd2fdc1Rl255ceYHPMXjxJr/4zG2JCcyQis34XV4BSZgBWnPlGbQE3e5TI rOn8emnnIz/pEfNVctpfQW+k2VLJs/ic/YkdkuoXLd/EvlpaaQ+cL+GcwqVaevARe1F0 1S0V+7O8shp3uK0rxq62KWWUGMXz6cTmJqp30TF2slxOzGEJ+LreGRmm1pp0yv26Cfrq OXyQ== X-Gm-Message-State: AC+VfDxHeMxC0mznaqu83Y4V5reKOYNkTpQM2W/lyXBhEmyXI8JJfDJ6 FgJMFT2CTwXwU9FIympKnie1Lw== X-Google-Smtp-Source: ACHHUZ4t5WHqyo4ZYahXOm5IhQBts9kOj8de1dBmPyGeGIdQ3YL7Www59M51QQRN7AwqoJV6XrQ1Ww== X-Received: by 2002:a17:902:ea01:b0:1a9:6467:aa8d with SMTP id s1-20020a170902ea0100b001a96467aa8dmr21674286plg.1.1687424203648; Thu, 22 Jun 2023 01:56:43 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:43 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 20/29] zsmalloc: dynamically allocate the mm-zspool shrinker Date: Thu, 22 Jun 2023 16:53:26 +0800 Message-Id: <20230622085335.77010-21-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AC2C21A0007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kka3hrcegmd7qg7fiw5xch3twybb7rb6 X-HE-Tag: 1687424204-610768 X-HE-Meta: U2FsdGVkX186CTqUwPbLyRKasEslYSWnTBeyMXhf3+qr1urEjNBaSyIUfYtCWADPp4y1Z3B4hpcdfssbtU9b6GKB27ih4IlRMchAjAFeMZU+183vFXx3nSXV7RYsLT3oRRTWZPYBSh26XDvLBgi+iJfZNfygeZ2bYQ1T1qNsNXawU24LokoiYgzpleGnXJOjszvlFkapazNSJImL3Y0nhxF2sl/BdNC2f0KG5QBYnJ14HKKbsOiNDKAOsiO0l2CU2dOAf8TEelbWOCMeXAoTT4aRdgCnMGwAM5jYXvXbrWyA+rIuH5WDGEBmJJRv3brQuA76hDFuSPiRPssnbBgcQidZx+8WUQQeP7rZFTtuo6hdGLaBC5cQIzLtSYAzIChRphF1bDJ2bcrh80N4kgoZ+4/wQmSrzVE7MAoeEs3tfk4aqTTaXU8JyoeietwmAw8yP7EaOrDxbfuZ2MU2doGT6C4I03j9Q5PV1f30w0YRxWKoAe88P8/CMrJr+CnAFatHbf086yLbt4ciNMHTXEv3d+lrMz6LY1P3TSR/4trKxvOAL6d8tPRX7a0wlwy5dGcSYwS9epQHSffbEIh39kLiDvOBTWKP5n82RlWWHFHiAlyBdLNG2Ek1eyihYvLIB9A7wF2pLPSOubvsEI7k7vio1SwhRLHSc26l1QOGZhbwBBA0NLU72b9MC1VOvm4oeY0CybS+vvFlWIrCGpyZVPXhdiG1kv/qIvDOHUrlEWsV9bhCyGn1V54qCnJbFIn/7yF3tmhz+yiWRIinaACBcAtfSekdN39c/d3J2k3Dt/9V93r49Ab8OxioXuasCICYrYaz6dNtCAywhc9rqn+tyNQNAz0+io2y2acYS7Go47o6/+PQ5rQz0kV/8eF56IGfjbdOhbZvxs7MZxQxOCHjEYBj1OCav1d5xsztd9qmPukb8pzf0a/WBwhDB5Q/xX3dKrQ5X8PjN2TnM8LFdKYZj38 LdIcNScn 35JxyIpRfGZn/bv/kiOQdaFvqYkK9B9YGui5kd3oPKStke0TprBdnAO8UD3WcTXuUN768ApRaN7OwqWlj8Xx3aoxvaxSH2rQSbrsny3vyDD6rQ7C8dldj/EMlOlpbW1FTjCIb8Ih8ibxsD2GCGxHmyteNpDo1Hhxpg8d4CgL2o5CJ3fsnHervqtJpcN0v3ANDkQU/PYED4so1lt0pxXE/vCN/MtmI5K4PCjwMOBzBz5uHw1/90xScFqtMtkj6XVXsjMh8B7IIHI3HrZM4h0V0ew1t6V9BrXZPSsvkT/PNO2Y86g8t/c2pyQUZVmPcI9QTFM3xd0itAHn9xiQeJE8/nWJmeabGy84Z+eJgp468BSyTR0xNPvn9DryTCESLKPmDsiJZRJSZ3n1rWBlcZ/oT6B6mAJfRgpxlF454/a9p69/KCsRTHEzaHmGKi5ewIUqYulkOhyTX2ggwbnM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the mm-zspool shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct zs_pool. Signed-off-by: Qi Zheng --- mm/zsmalloc.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3f057970504e..c03b34ae637e 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -229,7 +229,7 @@ struct zs_pool { struct zs_pool_stats stats; /* Compact classes */ - struct shrinker shrinker; + struct shrinker *shrinker; #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; @@ -2107,8 +2107,7 @@ static unsigned long zs_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { unsigned long pages_freed; - struct zs_pool *pool = container_of(shrinker, struct zs_pool, - shrinker); + struct zs_pool *pool = shrinker->private_data; /* * Compact classes and calculate compaction delta. @@ -2126,8 +2125,7 @@ static unsigned long zs_shrinker_count(struct shrinker *shrinker, int i; struct size_class *class; unsigned long pages_to_free = 0; - struct zs_pool *pool = container_of(shrinker, struct zs_pool, - shrinker); + struct zs_pool *pool = shrinker->private_data; for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { class = pool->size_class[i]; @@ -2142,18 +2140,24 @@ static unsigned long zs_shrinker_count(struct shrinker *shrinker, static void zs_unregister_shrinker(struct zs_pool *pool) { - unregister_shrinker(&pool->shrinker); + unregister_and_free_shrinker(pool->shrinker); } static int zs_register_shrinker(struct zs_pool *pool) { - pool->shrinker.scan_objects = zs_shrinker_scan; - pool->shrinker.count_objects = zs_shrinker_count; - pool->shrinker.batch = 0; - pool->shrinker.seeks = DEFAULT_SEEKS; + int ret; + + pool->shrinker = shrinker_alloc_and_init(zs_shrinker_count, + zs_shrinker_scan, 0, + DEFAULT_SEEKS, 0, pool); + if (!pool->shrinker) + return -ENOMEM; - return register_shrinker(&pool->shrinker, "mm-zspool:%s", - pool->name); + ret = register_shrinker(pool->shrinker, "mm-zspool:%s", pool->name); + if (ret) + shrinker_free(pool->shrinker); + + return ret; } static int calculate_zspage_chain_size(int class_size) From patchwork Thu Jun 22 08:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 407D9EB64D8 for ; Thu, 22 Jun 2023 08:56:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D618E8D0006; Thu, 22 Jun 2023 04:56:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D118E8D0001; Thu, 22 Jun 2023 04:56:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C00E38D0006; Thu, 22 Jun 2023 04:56:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B28BA8D0001 for ; Thu, 22 Jun 2023 04:56:54 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8B209120727 for ; Thu, 22 Jun 2023 08:56:54 +0000 (UTC) X-FDA: 80929778748.08.CD8C431 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf27.hostedemail.com (Postfix) with ESMTP id B1C0140009 for ; Thu, 22 Jun 2023 08:56:52 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=H+1EjwPI; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=02DUTqACM/aSl9866C36A3I60VPYUs4OjMZ7Yfrc1/o=; b=rgfK5KPt/csdCXNbA9NLrjz+9NyaPJXpU6gQpb8/eIc+hsCokOC0i+C3nua8Yml6exAgZm hICBkacxOZ29qzhhJCZlxCYsKSt2YNKTt1HmkVf7/A3h4Z6ulEORm4SZz4SPsX6bYZqHFl 6EJI62IE1G7INSlSjLCySxA0gtEOL2w= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=H+1EjwPI; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424212; a=rsa-sha256; cv=none; b=ewhgXBqcsy3s1Cp4iBt2xyVCbUDAiyxLimNZnshf4Rq9cvs9NLVg/VDK2ctTuVIA9A7bzI rLQv85L4s3PQa9uqsVUa0e4iOnGP4OGN/WgkjZZqirr6g8sN7HRCD5btv49a7sETQTzzDM EF4MNbaKPzNpRrqxmC5e3FPpHEw3GtM= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1b5079b8cb3so12591725ad.1 for ; Thu, 22 Jun 2023 01:56:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424211; x=1690016211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=02DUTqACM/aSl9866C36A3I60VPYUs4OjMZ7Yfrc1/o=; b=H+1EjwPIJiQm5S0OenD/xh0iPse4xpKj+qmkmBU/srKV3WPi3ULtA4pLp++dC5Bqh9 7BlRLloRVIKfQ42/hs1T7Du0cHLmrcEPoZEBW7JCQkTGLYx1UQ3fbHqq8+O3YFu52yg3 0+mCV5UtaM+1WMi4gyiYw8XnFCdNZ/oJcDwzQAYJvzkblrrUfBpPXXo27BHoXCIIli5o hjfG1EZpMvdsdRL66NPCU/nz333dm9KHMwYWw99s85mydOWvyhkPwMLzowDcLeXpMtBs G9dmlhU3PRkH82ZAs6EjGsULbQMB1pyMMS03ZQNG2aou+qooblLYB2n9++fqbuAHUsk3 kqMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424211; x=1690016211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=02DUTqACM/aSl9866C36A3I60VPYUs4OjMZ7Yfrc1/o=; b=h6v5A8QpmkrA1nIpqD/P4/eZXrJ9jo6qtcg7zWvedPGb4NyB8alEJbN7QJnI6+uyk0 gwx78wsKlBcbv9sefn5bPKX6cwcVM7uVbpGVzihFucjS2Rjy1Tm6INyG00KdMpr+EAdp IYiwf8Xm/hrKIJMhKfw1J2OTzDZuyNfgaqEkgkDn+tpzPn96yhb4GVkbAw5sB1kCOiU6 chkJhoa/y/amlv6fNkh3CTNWcSlkQZui+yRlYhWWpU3D2DRAXGGm/RSgjReEzFB/79Ho lKl/wz3VUpThTkeyL1NvZCHSLqxwaUAOnd5qUlRkh0sL2Ad+X5EDiykHcJFuydqMWYvN 4mhQ== X-Gm-Message-State: AC+VfDyZhuDCh4c2mRitU6D2jXBFSK/Qcqm+h37fRmskkBGN0+snMm3A p+7x3T5DYATYeJK1yyC9XU7buGBJphLrQQTN9+Y= X-Google-Smtp-Source: ACHHUZ4bv27VnKnfH9aSIwqm+pi4rG014ro45egLBBsO6iOayIjodJqPa/Ct3mSpE0n2zy/iA6cg+A== X-Received: by 2002:a17:902:ecc6:b0:1b1:9272:55e2 with SMTP id a6-20020a170902ecc600b001b1927255e2mr21755968plh.3.1687424211650; Thu, 22 Jun 2023 01:56:51 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:51 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 21/29] fs: super: dynamically allocate the s_shrink Date: Thu, 22 Jun 2023 16:53:27 +0800 Message-Id: <20230622085335.77010-22-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B1C0140009 X-Stat-Signature: feph1f1uy9fskkdcw4qk55ufei5m7or4 X-Rspam-User: X-HE-Tag: 1687424212-168595 X-HE-Meta: U2FsdGVkX19nEA9RrBktYoDl+uvaKjyIdWmKd9IVzx1k6Pi7pwu0WsEmUjs8/jBgjONEbLil98oVf8SAXQeReu/Bz2T2EDaGHWXRNRLmuibflmR/4CnL8GbOtUhoNZCSSBtpm3Yo4UYdCjWXJi1htiPDQaIQD+entl7KyRUY8+HFTicr/3cAqTS6keenxLBNH6XV16UmwMQYFc5QAW/sBIWqcuSe4VAbbIJf+WlpCsES7XH+9KSrfjndb6lzFH1NnW1aSkdUaYRUgnopCm9qbjFgP6B1rTPltp9TlAxxiP5+vPu04BkilKqV3O0QeS76uWLhD1+/yFWfkBlhWxnb5zIAFlZx5zcErcMKHsKAm0/kBQoHhkdFv/MTUK58sNEEJz+b8Y/H/7yf1929En5j6yBBAoKIjTUvpXSK5vHfoVVjLP89joDGgISxHGxsofiygdS7qv8zhjQvGXbeeQJcRllPav3WdzB/8A6xrWTkUst2hnwZQQ3811b3FDjgiV2pdTjZEHD+mQ9mET+yHFj+7Ie5Q8akKQgx1FD0HwF/28Rjm+5/hEcSxO9RpwKnzDcZnykncwqdn9ZYDg4K6J9ScBo0wqidQWa5nNlU9BV/JHAbd4vcZdNKiCH8Tq7goyv9tetlRwsIIwXVrRqvlf25cX2jNknv9+x38P12s23Qg53nCA9Md3HgMi/QEf5qaNBflITglC0HByoQnO6LoSOsByaqW6xWPSntbHWB16dZ78u1N0jGEO5yP4+ftLGvTWDebccjjeaBxuZXtfNyb+J+HG72EOaMq0rU0M6R53Pp5XxCiRkSbRms4JHHQbfv5Y3R73L4SD412hhwCgRoADlCqroy71QY/Z3pAGXvS4sI8ymCDR/JFCla5PN4SPOyYIFhlbgGd08o/DDAqQgNRVU5nvwghKft4w/pbadwv5kNGJGypiUEDIb/pTd8y2kGJ0kvg1Cd6AvNMwYe7CBMbfH AFrTkbn3 emYy1j+yGEJz7IeBdx7aATb3aUg0SXkLsflab5lIdGxJjjkJBllb/1X4eZKmXiCi9mc2g2AEmLvENW7qLEuzwPrpi5vAYBYqIaWlPjNX+bL9xw8Md7gmPdjKVr5Ka2xzQI3DEyGF4ArRSZcS9LF52xPtWg/1+vMiGJSkk+V1b6p9jBILi7u67/CVkqdaPW6fF70DwctJBX9Ca4G3caQIbmxl9D58cgzCPGGF9yYFIa/s+N6htL4vYpXwRwjk5SXXtpOagfA5xHl9TcR/j/adZYFMW9y252vDWNxTyTbhmOVjr6cdTRvSKjPOwnlVOBe69XHtAMVP/DeGVwh0lqjEa+Y1b+f8KwHmjl6KGc4QpfRAX7I5wWOSv7kgTu9MBaBmILGERxl5V0NR2QwOJyWefW+9z/MFvEjH+u3RIUtUbGgzPCwCoKCcjcjxsWTVw+KmzqUlbcyFqqbRhuTtm9fcIZnXqOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for implementing lockless slab shrink, we need to dynamically allocate the s_shrink, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct super_block. Signed-off-by: Qi Zheng --- fs/btrfs/super.c | 2 +- fs/kernfs/mount.c | 2 +- fs/proc/root.c | 2 +- fs/super.c | 38 ++++++++++++++++++++++---------------- include/linux/fs.h | 2 +- 5 files changed, 26 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index f1dd172d8d5b..fad4ded26c80 100644 --- a/fs/btrfs/super.c +++ b/fs/btrfs/super.c @@ -1513,7 +1513,7 @@ static struct dentry *btrfs_mount_root(struct file_system_type *fs_type, error = -EBUSY; } else { snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev); - shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", fs_type->name, + shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fs_type->name, s->s_id); btrfs_sb(s)->bdev_holder = fs_type; error = btrfs_fill_super(s, fs_devices, data); diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c index d49606accb07..2657ff1181f1 100644 --- a/fs/kernfs/mount.c +++ b/fs/kernfs/mount.c @@ -256,7 +256,7 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k sb->s_time_gran = 1; /* sysfs dentries and inodes don't require IO to create */ - sb->s_shrink.seeks = 0; + sb->s_shrink->seeks = 0; /* get root inode, initialize and unlock it */ down_read(&kf_root->kernfs_rwsem); diff --git a/fs/proc/root.c b/fs/proc/root.c index a86e65a608da..22b78b28b477 100644 --- a/fs/proc/root.c +++ b/fs/proc/root.c @@ -188,7 +188,7 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc) s->s_stack_depth = FILESYSTEM_MAX_STACK_DEPTH; /* procfs dentries and inodes don't require IO to create */ - s->s_shrink.seeks = 0; + s->s_shrink->seeks = 0; pde_get(&proc_root); root_inode = proc_get_inode(s, &proc_root); diff --git a/fs/super.c b/fs/super.c index 2e83c8cd435b..791342bb8ac9 100644 --- a/fs/super.c +++ b/fs/super.c @@ -67,7 +67,7 @@ static unsigned long super_cache_scan(struct shrinker *shrink, long dentries; long inodes; - sb = container_of(shrink, struct super_block, s_shrink); + sb = shrink->private_data; /* * Deadlock avoidance. We may hold various FS locks, and we don't want @@ -120,7 +120,7 @@ static unsigned long super_cache_count(struct shrinker *shrink, struct super_block *sb; long total_objects = 0; - sb = container_of(shrink, struct super_block, s_shrink); + sb = shrink->private_data; /* * We don't call trylock_super() here as it is a scalability bottleneck, @@ -182,7 +182,10 @@ static void destroy_unused_super(struct super_block *s) security_sb_free(s); put_user_ns(s->s_user_ns); kfree(s->s_subtype); - free_prealloced_shrinker(&s->s_shrink); + if (s->s_shrink) { + free_prealloced_shrinker(s->s_shrink); + shrinker_free(s->s_shrink); + } /* no delays needed */ destroy_super_work(&s->destroy_work); } @@ -259,16 +262,19 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags, s->s_time_min = TIME64_MIN; s->s_time_max = TIME64_MAX; - s->s_shrink.seeks = DEFAULT_SEEKS; - s->s_shrink.scan_objects = super_cache_scan; - s->s_shrink.count_objects = super_cache_count; - s->s_shrink.batch = 1024; - s->s_shrink.flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE; - if (prealloc_shrinker(&s->s_shrink, "sb-%s", type->name)) + s->s_shrink = shrinker_alloc_and_init(super_cache_count, + super_cache_scan, 1024, + DEFAULT_SEEKS, + SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE, + s); + if (!s->s_shrink) + goto fail; + + if (prealloc_shrinker(s->s_shrink, "sb-%s", type->name)) goto fail; - if (list_lru_init_memcg(&s->s_dentry_lru, &s->s_shrink)) + if (list_lru_init_memcg(&s->s_dentry_lru, s->s_shrink)) goto fail; - if (list_lru_init_memcg(&s->s_inode_lru, &s->s_shrink)) + if (list_lru_init_memcg(&s->s_inode_lru, s->s_shrink)) goto fail; return s; @@ -326,7 +332,7 @@ void deactivate_locked_super(struct super_block *s) { struct file_system_type *fs = s->s_type; if (atomic_dec_and_test(&s->s_active)) { - unregister_shrinker(&s->s_shrink); + unregister_and_free_shrinker(s->s_shrink); fs->kill_sb(s); /* @@ -599,7 +605,7 @@ struct super_block *sget_fc(struct fs_context *fc, hlist_add_head(&s->s_instances, &s->s_type->fs_supers); spin_unlock(&sb_lock); get_filesystem(s->s_type); - register_shrinker_prepared(&s->s_shrink); + register_shrinker_prepared(s->s_shrink); return s; share_extant_sb: @@ -678,7 +684,7 @@ struct super_block *sget(struct file_system_type *type, hlist_add_head(&s->s_instances, &type->fs_supers); spin_unlock(&sb_lock); get_filesystem(type); - register_shrinker_prepared(&s->s_shrink); + register_shrinker_prepared(s->s_shrink); return s; } EXPORT_SYMBOL(sget); @@ -1308,7 +1314,7 @@ int get_tree_bdev(struct fs_context *fc, down_write(&s->s_umount); } else { snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev); - shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", + shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fc->fs_type->name, s->s_id); sb_set_blocksize(s, block_size(bdev)); error = fill_super(s, fc); @@ -1381,7 +1387,7 @@ struct dentry *mount_bdev(struct file_system_type *fs_type, down_write(&s->s_umount); } else { snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev); - shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", + shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fs_type->name, s->s_id); sb_set_blocksize(s, block_size(bdev)); error = fill_super(s, data, flags & SB_SILENT ? 1 : 0); diff --git a/include/linux/fs.h b/include/linux/fs.h index 53e0b5e98046..dd6f8ce28385 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1228,7 +1228,7 @@ struct super_block { const struct dentry_operations *s_d_op; /* default d_op for dentries */ - struct shrinker s_shrink; /* per-sb shrinker handle */ + struct shrinker *s_shrink; /* per-sb shrinker handle */ /* Number of inodes with nlink == 0 but still referenced */ atomic_long_t s_remove_count; From patchwork Thu Jun 22 08:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54D1EEB64D8 for ; Thu, 22 Jun 2023 08:57:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E99E98D0002; Thu, 22 Jun 2023 04:57:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E49EF8D0001; Thu, 22 Jun 2023 04:57:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D121A8D0002; Thu, 22 Jun 2023 04:57:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C223B8D0001 for ; Thu, 22 Jun 2023 04:57:02 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 95E7612017F for ; Thu, 22 Jun 2023 08:57:02 +0000 (UTC) X-FDA: 80929779084.15.D04039E Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf28.hostedemail.com (Postfix) with ESMTP id B2707C000D for ; Thu, 22 Jun 2023 08:57:00 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ARIMz0kP; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kOpMuT1tSay47KCV2B0lCCsOSa7MUah7TRoPRzKvzWA=; b=dSwEH04v/NQVjHqk/I4FG87foSTB5EZcdltB0SDQvtrYnG174AFj3ipy1CMSNW4p9KnjDa KCVQzXcu5YoyB3hCjSKKsU1IPlo28KeJiI9HyvffJ/r5T+WcbSWBtBoKiEM4BTHhaXW/lP JhSCG2l9xBTU2h5LxOD1z7MsQXdyEPo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ARIMz0kP; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424220; a=rsa-sha256; cv=none; b=gVeT6z8H1P2rd5N/2gfZ2P+9rPRgyJHZLWW/i7z0Miggv+30J8nKFTS5Uxa8cDSnbxCtGl zzQJAf3QSG+c7P3I1uEzaogqs60/g5yrIqZx9OAiGMsmVWyLe4+AUJxYzqcvPsBwUff+KL vMmucfxDDVukwSrwh6Hz6sJzplMz3SQ= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9656645ad.1 for ; Thu, 22 Jun 2023 01:57:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424219; x=1690016219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kOpMuT1tSay47KCV2B0lCCsOSa7MUah7TRoPRzKvzWA=; b=ARIMz0kPY37zDpA2KUfwIMWCUvAyZHIYiFTXtRMngKg5guyDvAvZNDKsvB2ZgT8UTW A25yLHuKDIGVoaFMSsnctFsAVv7yr9hBv5sSwiDDhv/mHcXUMpPqBe+IawTWyFdB0del fl+Pxlo1XCZ20awrxgcMp8gUHyp7xw9pGxosYaiH18lXNbdNEUFeaQtGj4iDzF4J11pb jyEKbhwN4kpW66g5W4dvNRCNuWMTsByWhUSRp4GIzQ739D16xyqzHvKa0SrFnLXwxenz WdarMgZ86g8pGHWvsC0B3J0QiIU/iI2cSgeTldT8LPJGZSfxzUyvtAnV3WYD0uKX4hXF pNRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424219; x=1690016219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kOpMuT1tSay47KCV2B0lCCsOSa7MUah7TRoPRzKvzWA=; b=Br9GeWxY46st9VvyDqSaoR1HhX7D3iyHOKPQuCXqOG0lS81tLu1ErEGbbXklN8pRFV Ag93Qh135qVSXjakARYFfgfEBslOY0EAoOozUSEzvls3VL16a21Po1PzejNkj8UTvzk9 QH49Ag0OrVnBkjz3d0RZlDIWq/7FGostIOb7OABJXRODOn3Hd68AVe7eAZHiodYM+0X4 3m5H59wrAYCkp56MJ+fpGo0WRkwdGwgUEXBtI2oEC3HUZkFoFuZQzukXSgyb2ZzqGnS8 KortGaroNQ55/SrFi+WTpLUbK7nn8wdu2I88ZlKytvbMwqiGkgeP3rSPvMP3J4lWKXaw abPw== X-Gm-Message-State: AC+VfDyJBsUtMluLp9XDX0b+lKFum1DdcVVDLnKdfIdZiXfLXUQeO+b5 t0CD6nzQJrg40zJm5gmjeVHWPA== X-Google-Smtp-Source: ACHHUZ7DqrxYhESUqlF1zio/fEoWG5nPFDtKmAeSV45runPF4798dCp+1eyZAVtYptcjl0nDKDsZDg== X-Received: by 2002:a17:903:230c:b0:1b3:ec39:f42c with SMTP id d12-20020a170903230c00b001b3ec39f42cmr21844690plh.5.1687424219648; Thu, 22 Jun 2023 01:56:59 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.56.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:56:59 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 22/29] drm/ttm: introduce pool_shrink_rwsem Date: Thu, 22 Jun 2023 16:53:28 +0800 Message-Id: <20230622085335.77010-23-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B2707C000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: paik8u3yfdpk8xduqq6g1hcg7bepb5su X-HE-Tag: 1687424220-671015 X-HE-Meta: U2FsdGVkX1/fqynWvBwULEubUndsHbcoBBduc+2fFsbsjGlZmdN+szcMZsgOS24jo/p0pbA1L+/KTUe0fe1qdRc+xb7msp+IYDCOzReuAJCpiS0ZPPh7U9nGdPvFF6aS6gOUq11z3EY6NJhhz4I4slpQRy+ve40k8iJQLxEsMASxvp/t6V3JklAJrlf6sHJmMjXxNgqRGrMF9xx/oxipRyRXLCg7tsUT+Gl8dU6ySo6O5AeBI/ZQnyC//f53pOcb2b/vh2eBB5ZLVpDPgccNIASUVTqAWrkWeqYiT6zC7Y0hH50uTQYIJsikedRnvzgZMXJUoAjJaFfR85jvyX0hdsRUgkuMKStI43YMwLjUqSlRGqYgXPDhl6q7SHX7g8uhK8umelsB9dVD2x2w2RjGY8WHysV/hbs6SfHAsQYN6iyWppDH872+zoStaL5EtsP5XSxOJyFWxa1r0sYhVvrsKZOVEYelmyp9fPFMid5D0Gz1QXKlgZikw2px+Ltxw8YPWJXqquNLZTMfZXDiNXC6UKaHC9BXJjljteawogWaNoAIFfDUqAgYoxHZb272gfdw8xLdxU6vqffUuS+ocgWpQ72iwcachSuvhflxUtHDiZtRyuyEq/5PaaSjaDAhiqAp8rzkylUDIDIop77kXxfj7gEMWRfewu1fnqRapgpiJ8AWnk0MKAMmOuFk7wcLlPiJy+OW1xvahkzASIEPq+2QWjvWvztXb+AO7GrWnZXvA8+nBR8awCjt2PEKnOm3ARmEoQlqFHUtsFj4r1Wu0ZuF21/kAlu/xghddRD0aJIleLNsPdXISGWjdTmLlgmKm7lXT5T7YI2ZOb5FiY7Z70xKDgONbSI1B0+sqGUkQlpeO95WDtMTYnrz2n/Oap2m8uxwoPjSpnZKBfoE4pnwcEvn28vcROXKO9iYCSO8WE8pF1BNyQEbq4Qqai0R7oca5RgvbLlc5veoKqjxekbYVV0 RWHOqZKw YA1EFjocWq71xJrl/ru5fFvBOImAuQdQp1O75PfKEOjK4wYklkrMjh7Tts7YPgjc1HCocTstVy57rd5XTOIrlsotG0mcgjA1imJ28pKkrWM7r+hIw9pAztYjYTsrKdBLtGSIK0gxTGHrOmVeYQ/m6zFWw92f3g1XWwO2F7VLY0DU4i59lkEpjA1YrYoTS8LBzDsUdAmbng4fpMYxF+KQFjjRNPAfYiZvwspw08LcvUNrcXzGbNC5V9L5hWRBa995Lm8XcryUiu7/ZalYCEP+QE2hnVbUzZytacmu8xYcH2i/pIyNNaE4ih0BeHt5eBdD2RTYHf/46TRRQCKwUsBKCcXWfEKNVAT6yVKIaeiGNAbv16GpZYmzrQe1kFJ1vZiaug5H3F70+8TYPyORlNY/yPn9BhcBDUDr2HleD0N1OiZXe3DQ4P5AfrnYmYenMkz8ys6Cqw2qjEbw+85xfF3T0LEKvYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, the synchronize_shrinkers() is only used by TTM pool. It only requires that no shrinkers run in parallel. After we use RCU+refcount method to implement the lockless slab shrink, we can not use shrinker_rwsem or synchronize_rcu() to guarantee that all shrinker invocations have seen an update before freeing memory. So we introduce a new pool_shrink_rwsem to implement a private synchronize_shrinkers(), so as to achieve the same purpose. Signed-off-by: Qi Zheng --- drivers/gpu/drm/ttm/ttm_pool.c | 15 +++++++++++++++ include/linux/shrinker.h | 1 - mm/vmscan.c | 15 --------------- 3 files changed, 15 insertions(+), 16 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c index cddb9151d20f..713b1c0a70e1 100644 --- a/drivers/gpu/drm/ttm/ttm_pool.c +++ b/drivers/gpu/drm/ttm/ttm_pool.c @@ -74,6 +74,7 @@ static struct ttm_pool_type global_dma32_uncached[MAX_ORDER + 1]; static spinlock_t shrinker_lock; static struct list_head shrinker_list; static struct shrinker mm_shrinker; +static DECLARE_RWSEM(pool_shrink_rwsem); /* Allocate pages of size 1 << order with the given gfp_flags */ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, @@ -317,6 +318,7 @@ static unsigned int ttm_pool_shrink(void) unsigned int num_pages; struct page *p; + down_read(&pool_shrink_rwsem); spin_lock(&shrinker_lock); pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list); list_move_tail(&pt->shrinker_list, &shrinker_list); @@ -329,6 +331,7 @@ static unsigned int ttm_pool_shrink(void) } else { num_pages = 0; } + up_read(&pool_shrink_rwsem); return num_pages; } @@ -572,6 +575,18 @@ void ttm_pool_init(struct ttm_pool *pool, struct device *dev, } EXPORT_SYMBOL(ttm_pool_init); +/** + * synchronize_shrinkers - Wait for all running shrinkers to complete. + * + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory, similar to rcu. + */ +static void synchronize_shrinkers(void) +{ + down_write(&pool_shrink_rwsem); + up_write(&pool_shrink_rwsem); +} + /** * ttm_pool_fini - Cleanup a pool * diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 8e9ba6fa3fcc..4094e4c44e80 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -105,7 +105,6 @@ extern int __printf(2, 3) register_shrinker(struct shrinker *shrinker, const char *fmt, ...); extern void unregister_shrinker(struct shrinker *shrinker); extern void free_prealloced_shrinker(struct shrinker *shrinker); -extern void synchronize_shrinkers(void); typedef unsigned long (*count_objects_cb)(struct shrinker *s, struct shrink_control *sc); diff --git a/mm/vmscan.c b/mm/vmscan.c index 64ff598fbad9..3a8d50ad6ff6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -844,21 +844,6 @@ void unregister_and_free_shrinker(struct shrinker *shrinker) } EXPORT_SYMBOL(unregister_and_free_shrinker); -/** - * synchronize_shrinkers - Wait for all running shrinkers to complete. - * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. - */ -void synchronize_shrinkers(void) -{ - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); -} -EXPORT_SYMBOL(synchronize_shrinkers); - #define SHRINK_BATCH 128 static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, From patchwork Thu Jun 22 08:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D623EB64DA for ; Thu, 22 Jun 2023 08:57:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F36378D000D; Thu, 22 Jun 2023 04:57:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE70F8D0001; Thu, 22 Jun 2023 04:57:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D87C68D000D; Thu, 22 Jun 2023 04:57:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C8E138D0001 for ; Thu, 22 Jun 2023 04:57:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A1109B004D for ; Thu, 22 Jun 2023 08:57:10 +0000 (UTC) X-FDA: 80929779420.23.429CBEF Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by imf11.hostedemail.com (Postfix) with ESMTP id B938640003 for ; Thu, 22 Jun 2023 08:57:08 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=btQFnTi1; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZsvVWwvjCNbIpzLWR+JMbRdqcPMoA6IncSRNtrMJjtk=; b=w+GMNDQsm04fn2+li3xtX0lx0MexefxrsDwo38b7s9z+m/rogff/ckQhFe26nW/FgliWqm OEiIqrJLX3TNFTEObwJ3m3jN+AHOM5+jSJXQhTAbRSPZU9do1cNwDzbYNWQ5tAvhDd8qK+ tGfcAdmBBJGZ/Nt3wqU5pTl0Ju/OLDc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=btQFnTi1; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424228; a=rsa-sha256; cv=none; b=Th97LH00+WryebxR33oBjGW4gNDJZ2ymjUxTSLItcsz6eVZ1zE00FKylTDcEI+6sOPTdrj e+9o1cSSLAipCyEpFRWmGqcO8lWGNcB84X7n4MrdElpmBC6KRvf/XbDbmT6PO8OgzgZJgr XjHIgpI3Dj6NeVgLz1Z3ynYt4rhRaXs= Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-543a37c5c03so1091416a12.1 for ; Thu, 22 Jun 2023 01:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424227; x=1690016227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZsvVWwvjCNbIpzLWR+JMbRdqcPMoA6IncSRNtrMJjtk=; b=btQFnTi1+Jxgpt9ZEv6SV1CxbtEwEnpYQ1WSSwhfGLNNoz+3LZU80wQIeDEMGJinhi 8+hFjjrmI0Ti2weoui+1nba3fuqZZ6jN3eQN4e+CjSeo0Ggyrqg3j3GGKdGsxTGtEKKB 6n4ntOH8ClqO9PsI9RmJeFYa/h0xYT0vefdwMfx1z83TBiR/mTSXCJDkn82jEtfe964Z +W5pu2WQm5/BmUewzteVLri/ASv79HUROzRO9GvLtgUyAJkwQyVDE/hnKFhVeWh3zZR/ QE/dybVozXXKkm+aMId69JeSrmobjvQPBxgZrNjMENKTPWKUy1hHiQ4+Qa8p1f7rRS3j Hymg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424227; x=1690016227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZsvVWwvjCNbIpzLWR+JMbRdqcPMoA6IncSRNtrMJjtk=; b=jZUxB6+LSfYAFnMz7fDGJdg44DMU6bTO4hadMTmJ6nppHD5P15pQfu9awpG+4NHkUK G4NGIzB5lO1klidz02bJeNg9LzAivDftbTPOLHEjdtvqaN2P6Y+xW8lV1k/XjQ334ot7 p7a4QrpmnpBXZEGTWx13U20gsX4vGGX28jmBXCHZTQECKBuu5beGdfUxMBYOpwFAhtnE WSjD6rgIHTXBN5t3j0wimmlfKnpirlDLokL0/4EezydmqIO+FG74UrCFKAz/LluDNpjF XAoyNxhpmS2paHlsjCaPB52Ttpr2nkreii+lSSygsAvCPIvCXNOMAtiIRQq7Vpm3XJW8 qhNA== X-Gm-Message-State: AC+VfDxswiF9sRb01Mblzwu6/DSj1hiVuRW8PVV7ota63yFs5Jl9HR5a fZalDCbrwCSLQiXpUDB/H9p0Yg== X-Google-Smtp-Source: ACHHUZ6IzlQYehvkbwm6qTB7qvbww6jKgEwXRnqs2X+ZAvHSdOqqO5hMcbKXrRtXSYnzqJ1v0JsE2A== X-Received: by 2002:a17:903:1105:b0:1b3:ebda:654e with SMTP id n5-20020a170903110500b001b3ebda654emr20780380plh.5.1687424227653; Thu, 22 Jun 2023 01:57:07 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:07 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 23/29] mm: shrinker: add refcount and completion_wait fields Date: Thu, 22 Jun 2023 16:53:29 +0800 Message-Id: <20230622085335.77010-24-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B938640003 X-Rspam-User: X-Stat-Signature: r663i5fgypizx4etzp88ssdxhh5otwge X-Rspamd-Server: rspam01 X-HE-Tag: 1687424228-1929 X-HE-Meta: U2FsdGVkX196jgKCZwfRMSeAmCGXkWbe0exB+q2WDi9hPDLkc3c/U7Xc2NMAcaBNSSH5gWd62lnaRNMczunCdlriMtT2rv76IReICw/wOsIh0lUZG9LGB3zYPqKpaYTol/PYOrYixlCtrinc74Tk9edPyyKHuGfluau3Jl5kryvPVivGN5kjqjpkcaaLkz4Sf15SCi20y17OwoWCB5pc2fcIjX2tEEArRL5z9WGBRlW2daHr3qw6iXxmS24PplsFgVLJKpgy3mE8EfPd+k4d0NjU9NU+6B06pv7I77SjuEskU6sWrTHNjeDqkpw8foL1xErBRbTK4VE80APqzVF4se/aAR9JXIQz2VqdSQoB6afFrWrX8iWVCG9Z+ydrwR9lI225mrMxFmZdU1vEi4Qg1gVEp131pzpQ14IGJJ04BDGOKNSCxwGIVaIQOT3ek/+LS3LfZhJhnUyzAY8TCIpBmw35W67bHMEkVAcbF5bwzwlC+tUxm4CXy/8uF/2nZ30zBwSwFednZWvx3uBVpFXAK8ydKf2rbHHBy5SaWeEcVMpyQ6w7O/Ycnzk9AOUXUkc9ciZFJ6oCt7E6s0LUp3wXVTNdYT7S1LpXeUNfiMxscgaB2fkyyIRLSg2F3HlBwka8a8od9+xwfXybZld1PFuBYApZ6mvk79VveCz1ti2/Lm5H+c3rwl4q4fQ/KqF8/bCZcufbTYTYa1ks1UJ6DisohPUvuBzvSs5YRfnh10s09SO1nlCEVQVLvzVBRUuZSOWLrtuEE8ipG/xwmhGoBU38BKHRyv3eSr1uem3+vvJ4vmtmeXGtAUUfnxF4pQrAfQx9i+jaUVaLz0NNVu1XlS1EOYoh+htv9nVHi+cCg890j5aALFskEBav21rMBwzjK4DgzLwcw2tdPBzemIG06PSAc4cOeGMAGVcUAQ0ODxjsCPlM+Fa/iTYh3uhAcTo54doZm9yx0x7cgXniSvhzaos YwlSvAES 7yfepWfSShdqBv+OdXj1+bMkb3G89hosbnOXJs14aBhPbKIpJIYnn2paBeYgk/NJW7j4pJAmudgVC6rQQg8z/2ToV0zlz4A+Khetr1wCyfn7/v5EpxTgeAM/PSpYkSd52TUlk3QNqxswD6ebQ6/z+llKTdpAs/+RwTGEkEO8e/eQWcI3ovQO2cMFtobs7JqtD+9BgALmIehntz1My9x9BxkXcYtjXSS2L0OBW/wJf6AKcLMpumZ0wnDynpxKgh1Lh4M3FKlFx8Ic6iRvS9IPSuxz7gMG5jH+E8UFSwJKS/4R/8J6HTkAq2EMiBYgwIEeN4Div0zL5Fzd5DGhXu1hn/yhSzQxiH/m1+ogp7bbMX2twmX42hp5jTqIgZSL3Nx4XN4r2cxMsLlqnaLOn1WikCAta8ff26bLln5Ue2EHzCIBd/NqlAUtvcqi91Bk6f/ktV3IGPA/LRxSG40wTJXTOYeLFv+J0vxQWlP1NRrXlW9FvHfogZP5y+vgWug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit introduces refcount and completion_wait fields to struct shrinker to manage the life cycle of shrinker instance. Just a preparation work for implementing the lockless slab shrink, no functional changes. Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 11 +++++++++++ mm/vmscan.c | 5 +++++ 2 files changed, 16 insertions(+) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 4094e4c44e80..7bfeb2f25246 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -4,6 +4,8 @@ #include #include +#include +#include /* * This struct is used to pass information from page reclaim to the shrinkers. @@ -70,6 +72,9 @@ struct shrinker { int seeks; /* seeks to recreate an obj */ unsigned flags; + refcount_t refcount; + struct completion completion_wait; + void *private_data; /* These are for internal use */ @@ -118,6 +123,12 @@ struct shrinker *shrinker_alloc_and_init(count_objects_cb count, void shrinker_free(struct shrinker *shrinker); void unregister_and_free_shrinker(struct shrinker *shrinker); +static inline void shrinker_put(struct shrinker *shrinker) +{ + if (refcount_dec_and_test(&shrinker->refcount)) + complete(&shrinker->completion_wait); +} + #ifdef CONFIG_SHRINKER_DEBUG extern int shrinker_debugfs_add(struct shrinker *shrinker); extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, diff --git a/mm/vmscan.c b/mm/vmscan.c index 3a8d50ad6ff6..6f9c4750effa 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -740,6 +740,8 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); + refcount_set(&shrinker->refcount, 1); + init_completion(&shrinker->completion_wait); list_add_tail(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); @@ -794,6 +796,9 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; + shrinker_put(shrinker); + wait_for_completion(&shrinker->completion_wait); + down_write(&shrinker_rwsem); list_del(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; From patchwork Thu Jun 22 08:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 772B4EB64D8 for ; Thu, 22 Jun 2023 08:57:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15F268D0006; Thu, 22 Jun 2023 04:57:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 104688D0001; Thu, 22 Jun 2023 04:57:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE6F18D0006; Thu, 22 Jun 2023 04:57:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE6368D0001 for ; Thu, 22 Jun 2023 04:57:18 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AD033809D5 for ; Thu, 22 Jun 2023 08:57:18 +0000 (UTC) X-FDA: 80929779756.23.2B06B64 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf22.hostedemail.com (Postfix) with ESMTP id BFCE3C0005 for ; Thu, 22 Jun 2023 08:57:16 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZL5KbVwm; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WJL0OVu56Nd2o0mxjMzDLbJ1YgOSFE+1vh1fraMJRQc=; b=Z/oLLXJzj6UEK8U/Wcnk6w35UgQBhV7Km9uhFa8iGpB4jzghklWSYFoxZZCKXchFGP5BM1 laHne12KzEH34sBgI0cMQEkdSmI04+/LS9S/Yf80SZDto0slyJ8dy8TtJLcvDkFK74vQ95 5zlnRJxZnBpJtB5X7wHUaKNqBnxWV2U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424236; a=rsa-sha256; cv=none; b=xIXi9hybT2mkNNdBOAgwp+cGFGbUqyd6/qiM7jtx4X7yql/J884Av9kP1Pcno5CVGlG9vO 5HYK34/ygF9om3ndwjU6/AY3L5fo3U+lxYNzFgYMX8CQ2RRSsN1hUORNaCMHes0VMqCUV3 y+XtcLX8O7kE7VGQXXt7E48vBF/Ecw0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZL5KbVwm; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1b5466bc5f8so9657395ad.1 for ; Thu, 22 Jun 2023 01:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424235; x=1690016235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WJL0OVu56Nd2o0mxjMzDLbJ1YgOSFE+1vh1fraMJRQc=; b=ZL5KbVwmDDj8Gy97lbIJB7YMY8SjOouZdkJA02qk3b6DF+dyuydzFaZQylnxfC9S+S TSPs6tccznQsZMFsLcEJsyUCVw2T8A7tWiogG+8JrU9eTxXc/9BRSgGLnZUgk22umk+B SSngiAqdElEoARuVlyhvyt60KkFHlCbx7fUzwDo8HLyuhKs4MGrMo82qC5g6M3adQ/kW 5sTm4LvGQVeaef4Ot0YEMEqtMtR7HuPlighv8JjZTv9ZOAYC2jp2PBoT7dm2kyLjCYpo 2319G2aelwdOOipT4nDLl60x2b/otqGmO31pzkQB3PluAdANWpdqDdwlhTA0kEDJVKL2 ha0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424235; x=1690016235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WJL0OVu56Nd2o0mxjMzDLbJ1YgOSFE+1vh1fraMJRQc=; b=TMTT2WVmrgVoaJI1KWIeENm5ZDv2m0VKrCGLtXjspHmM/MK01fswSCLAuKVEABELEL hxwOTmAWTnkFwg7JsU3qWO6d0IWeMVOS8S0L1JIn/AWbCDx+d8s2ixtPkyXHdgiDwbVS YOaHFlTNhfZHMhlH1Ffs6EJP2biAKJa6MqRYWrvVUo9FUWRfRJIzOUxLT+m4yit6MLh1 RxF2qdPim+2H++vpoFAlt+83l+Uf6v++3ZOWp7O05rDwq15drW2JCbA6zykPhoMpT+j+ wucTyO/QMOhNA29sgU6yh6VcwUr5JE8RKJyIdD8Oq6qkypqTXCQV3NF2gI7Ux1GY6KC6 6TxQ== X-Gm-Message-State: AC+VfDw9AzldHzJgVXIzD8owrr0BEp9hCpTnLHCVbTz57URhIAD0/xwf gnJ9XiyDyg0EJcEm25KOLKywvA== X-Google-Smtp-Source: ACHHUZ7SQ8rS1EvkM8z3rgy8xjUWZCKvjOIofntYWhPm+3TRoZDLL1AdCBUgXDGJA0nPgYbkXii2Dw== X-Received: by 2002:a17:902:c945:b0:1ae:3ff8:7fa7 with SMTP id i5-20020a170902c94500b001ae3ff87fa7mr21702029pla.4.1687424235565; Thu, 22 Jun 2023 01:57:15 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:15 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 24/29] mm: vmscan: make global slab shrink lockless Date: Thu, 22 Jun 2023 16:53:30 +0800 Message-Id: <20230622085335.77010-25-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BFCE3C0005 X-Rspam-User: X-Stat-Signature: cn844cz9u9wkdd1153nyjjh1gdyni7up X-Rspamd-Server: rspam03 X-HE-Tag: 1687424236-997987 X-HE-Meta: U2FsdGVkX19sgIXrD7rz1vrBl51VkUTmRgwXohptnCj9e5i6ZrmRGtJXa9C/qRcbYGMBdXPr4eWJRKuTe1G07zH2pKw+bE99QITPiDsk+nDnbn98oRX3kiF3v5NdbHUleihb7rOfIvsYSgEwjmHFmzEhKmnGKjPr1v9JISf3MBk/Bxa1jsgOrg4sgrWA6s6PHsrTl1XeNBLsBZN67a4GzIGEvCdN7wkMqO/gkicde+eBdtNj+KW/rZclTvPwfFvKGew6c6Wg5jtPnMJjM1ej/t1HhTlo+0qNqpGdsmIpnYXMm7kypBYOEbUqJhVN3k7XSQEXH8uGX4Fh7Fd1f9/bhV1ToGTepN7tp6bjpm7vt+G9mWh22BVk/XERNuWOpjH4tHRyQPFEL3/S2dasSkFYRDQNWvfn9s3zaD2K7Wr7MsIICfwYg0tmaF96UUXkBvzNYGptd0/sumvwnVV6J0VSx5WguZ8AqI3yXXrQn86l+98RsUxEXBKqKCtvasfSusPCg9D0e2UcNpgWThvTCLaW3/sSOMsEJ2JAg7H9VxJ47NahiHUCYYWWLRR66N/PDxIsl0/0zq3Oh0o2DZ4pPoC8T4ch6PJ77U//v5J9OZahVYYaiN/DT++YjuaebrNzJwVgMkML4WlMhX7MPfko4Iz7+En3H318NbGMI50LutLdAMIcfHxrX+7puGrbecbHh1uRB6ehuHhduIx3Nxw81NjraEsmR0jsySmd8P4HtfUphYdVETiNo3Cf7Iof8isgjaif0pn/nVXyoPvxwhkPNHQZQVKseBIVAROjwrfSlkSj6+Ytds14zsHz+pcJqRYFoEvYntF7R8HOlHUfX6Xwh6U1mKNRb6g9tGDb/chS8UlgBjrHkFcxAqaJoRRX+rvN9d8eppvlZ6dMMve2blnQMAbc/txRfFDF1qI7zMMcwCXrZYUBGY5ODcwNQtaWCKhoRATEW6KpKRGnXdnCj7hZZsF 488FpjRb 8XaTFZiVRtNBEdmhyT//FIvxq7E5QHd2sqlsGkANC8kK9Z5NTwPPp26m+KmpoJ2hMCKZ9CfxFXh3Oe2PkjuGY84G1BPbOQY2M+Fm0ifC35RINI1ULjp+8ncXWeWFqhVd4OeazmIkWZbnz5dAaM7L31L8wJzlVlmRRImqqDGZXiJp+iDOQCD8/9GjrksRzx3/t/DNm4Ey6IcaO1vfEJsfFKgleJfWIGBDuoM3k9LdUeTyCbaXXUzwow4uSZ8fydgmJoIavD5Y8lWLONtCd50WHx+OAbejNWfs5yaUdZGvv/bP+C8W/3rOoZ7aQo7L7ovoFvyxt/7Pqg+qdx85tbHs3GKH4EjvPbqSphxalaOEvoIaxZfgFBdDFg3v6ZMD4VL26OXqHljOTbcldZUAbgr3YogmngxKqRPs6cldCfymVch7KJElCViynRZlHkzWnUmbT1sSoNU7VZJy02QpcYzI/UoypWwVRrZ8t6wlLfj1WEmde66uzya4j8c0pCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global read-write lock in shrinkers subsystem, which protects most operations such as slab shrink, registration and unregistration of shrinkers, etc. This can easily cause problems in the following cases. 1) When the memory pressure is high and there are many filesystems mounted or unmounted at the same time, slab shrink will be affected (down_read_trylock() failed). Such as the real workload mentioned by Kirill Tkhai: ``` One of the real workloads from my experience is start of an overcommitted node containing many starting containers after node crash (or many resuming containers after reboot for kernel update). In these cases memory pressure is huge, and the node goes round in long reclaim. ``` 2) If a shrinker is blocked (such as the case mentioned in [1]) and a writer comes in (such as mount a fs), then this writer will be blocked and cause all subsequent shrinker-related operations to be blocked. Even if there is no competitor when shrinking slab, there may still be a problem. If we have a long shrinker list and we do not reclaim enough memory with each shrinker, then the down_read_trylock() may be called with high frequency. Because of the poor multicore scalability of atomic operations, this can lead to a significant drop in IPC (instructions per cycle). We used to implement the lockless slab shrink with SRCU [1], but then kernel test robot reported -88.8% regression in stress-ng.ramfs.ops_per_sec test case [2], so we reverted it [3]. This commit uses the refcount+RCU method [4] proposed by by Dave Chinner to re-implement the lockless global slab shrink. The memcg slab shrink is handled in the subsequent patch. Currently, the shrinker instances can be divided into the following three types: a) global shrinker instance statically defined in the kernel, such as workingset_shadow_shrinker. b) global shrinker instance statically defined in the kernel modules, such as mmu_shrinker in x86. c) shrinker instance embedded in other structures. For case a, the memory of shrinker instance is never freed. For case b, the memory of shrinker instance will be freed after the module is unloaded. But we will call synchronize_rcu() in free_module() to wait for RCU read-side critical section to exit. For case c, the memory of shrinker instance will be dynamically freed by calling kfree_rcu(). So we can use rcu_read_{lock,unlock}() to ensure that the shrinker instance is valid. The shrinker::refcount mechanism ensures that the shrinker instance will not be run again after unregistration. So the structure that records the pointer of shrinker instance can be safely freed without waiting for the RCU read-side critical section. In this way, while we implement the lockless slab shrink, we don't need to be blocked in unregister_shrinker() to wait RCU read-side critical section. The following are the test results: stress-ng --timeout 60 --times --verify --metrics-brief --ramfs 9 & 1) Before applying this patchset: setting to a 60 second run per stressor dispatching hogs: 9 ramfs stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s (secs) (secs) (secs) (real time) (usr+sys time) ramfs 880623 60.02 7.71 226.93 14671.45 3753.09 ramfs: 1 System Management Interrupt for a 60.03s run time: 5762.40s available CPU time 7.71s user time ( 0.13%) 226.93s system time ( 3.94%) 234.64s total time ( 4.07%) load average: 8.54 3.06 2.11 passed: 9: ramfs (9) failed: 0 skipped: 0 successful run completed in 60.03s (1 min, 0.03 secs) 2) After applying this patchset: setting to a 60 second run per stressor dispatching hogs: 9 ramfs stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s (secs) (secs) (secs) (real time) (usr+sys time) ramfs 847562 60.02 7.44 230.22 14120.66 3566.23 ramfs: 4 System Management Interrupts for a 60.12s run time: 5771.95s available CPU time 7.44s user time ( 0.13%) 230.22s system time ( 3.99%) 237.66s total time ( 4.12%) load average: 8.18 2.43 0.84 passed: 9: ramfs (9) failed: 0 skipped: 0 successful run completed in 60.12s (1 min, 0.12 secs) We can see that the ops/s has hardly changed. [1]. https://lore.kernel.org/lkml/20230313112819.38938-1-zhengqi.arch@bytedance.com/ [2]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [3]. https://lore.kernel.org/all/20230609081518.3039120-1-qi.zheng@linux.dev/ [4]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 6 ++++++ mm/vmscan.c | 33 ++++++++++++++------------------- 2 files changed, 20 insertions(+), 19 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 7bfeb2f25246..b0c6c2df9db8 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -74,6 +74,7 @@ struct shrinker { refcount_t refcount; struct completion completion_wait; + struct rcu_head rcu; void *private_data; @@ -123,6 +124,11 @@ struct shrinker *shrinker_alloc_and_init(count_objects_cb count, void shrinker_free(struct shrinker *shrinker); void unregister_and_free_shrinker(struct shrinker *shrinker); +static inline bool shrinker_try_get(struct shrinker *shrinker) +{ + return refcount_inc_not_zero(&shrinker->refcount); +} + static inline void shrinker_put(struct shrinker *shrinker) { if (refcount_dec_and_test(&shrinker->refcount)) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6f9c4750effa..767569698946 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -742,7 +743,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) down_write(&shrinker_rwsem); refcount_set(&shrinker->refcount, 1); init_completion(&shrinker->completion_wait); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -800,7 +801,7 @@ void unregister_shrinker(struct shrinker *shrinker) wait_for_completion(&shrinker->completion_wait); down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); @@ -845,7 +846,7 @@ EXPORT_SYMBOL(shrinker_free); void unregister_and_free_shrinker(struct shrinker *shrinker) { unregister_shrinker(shrinker); - kfree(shrinker); + kfree_rcu(shrinker, rcu); } EXPORT_SYMBOL(unregister_and_free_shrinker); @@ -1067,33 +1068,27 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; - - list_for_each_entry(shrinker, &shrinker_list, list) { + rcu_read_lock(); + list_for_each_entry_rcu(shrinker, &shrinker_list, list) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, .memcg = memcg, }; + if (!shrinker_try_get(shrinker)) + continue; + rcu_read_unlock(); + ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } - } - up_read(&shrinker_rwsem); -out: + rcu_read_lock(); + shrinker_put(shrinker); + } + rcu_read_unlock(); cond_resched(); return freed; } From patchwork Thu Jun 22 08:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54A54EB64DA for ; Thu, 22 Jun 2023 08:57:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC0828D0002; Thu, 22 Jun 2023 04:57:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E71348D0001; Thu, 22 Jun 2023 04:57:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D11E18D0002; Thu, 22 Jun 2023 04:57:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C21B48D0001 for ; Thu, 22 Jun 2023 04:57:26 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 94B10A0292 for ; Thu, 22 Jun 2023 08:57:26 +0000 (UTC) X-FDA: 80929780092.02.D956851 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf21.hostedemail.com (Postfix) with ESMTP id AE8CC1C0017 for ; Thu, 22 Jun 2023 08:57:24 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VXm59oxe; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424244; a=rsa-sha256; cv=none; b=hr5qCHsJO5f+EQK5vZMPnmeGyS1lIQ95+9FlWFdi1FdFsYEh0q/p5Z3ToWvMf1+ryzODlP wD8QZw8mDHFBLnIz1wXTEPTmBL8NtY/sl6gkGbcH1BH9DL0MiZiLA3VKbHj2plD+VNZmc2 VNpN1QKNDPOBiXG+sV5O2XLF9g2YCXc= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VXm59oxe; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424244; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A8Fm0YGxp8yCkyBdSSDbuW3Cqk4JmtEEZRGj6EeWqug=; b=b+S/zPOJEH+S6u0WElTJes90XzCNf/Vpssu7NipNYrXS21JW2frJd4p9h9iSLUcASB++cl NKA7KGFmuX+CJaaFYo2NbtXYjRbs5NQwqJrV/Zm71tJZF6/MMxJ7FlXXq54Bb0Wuq6sIcx LhJZCpOUPwmjTMcmxLAaTtfZF88Mnng= Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-54fd4a7ce25so1096509a12.0 for ; Thu, 22 Jun 2023 01:57:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424243; x=1690016243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A8Fm0YGxp8yCkyBdSSDbuW3Cqk4JmtEEZRGj6EeWqug=; b=VXm59oxeB5FguY23iZ6wxsHZUa/n0ixVM5+RJovFhBsSqk7jUN18WD75GpJCgArR0W JBsEkA9d39jtf7NG1SF+QvFe/XJdGOqobYaTEX0aq6YpeCNoIMMKV4Gdkmm7QKEFMCrH QNswdcRq+FKKLLJSUCefQDEwUuyAf6EXdUAA8SWzr8d5PlDWqAvD+NLb3ziu4Q1ZbddG 3AHOgUEiuKr4IMyzSNLNbfYcakjlWAem1FVIRkWrvCWnM54TEnQwCZtRjpDkJUyTk1gU FK042RTEuCbaP60yQQ0pKyvy7feFAoM0ianmMZaWx78B21cSO7WyFn4vwESs5Li+xEdq ozzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424243; x=1690016243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A8Fm0YGxp8yCkyBdSSDbuW3Cqk4JmtEEZRGj6EeWqug=; b=O9BM38h5GZaV9Mt/KjG2cJnq7ekI0GfkfouZGkzUTXuMnc17P5YBLHx85OfHMEpbmC WFTV+m5qvjBO8ck/8CPiZyAruthOooSQa1KzpckMIdbpt+/kUf5VdF8CvRt24VTo/JRh tudSnGSNG649Qk2Oz4zzKH6D4eHD7s8hW4xTgi1ujj2r4gSMQ8kRgku9HqSj/OfRURCx P/cXz6S47MSDsG/FiZ+1nMOnuoLAecRS8StCskzn2ZzKOAmVNh9W50YTwcM2fhK7gB7L BGSBgPHlryoMnDOdHpRRK9/Tfjq9yxMTwpkpvyMXcp34qVYNUL5S/Eh/p1DpR3DsxtTK T6pA== X-Gm-Message-State: AC+VfDz3sf+jIs0JZ4bjy818xVtxxGq2Hb2sKzP7D2Bj9bbZxJDV9haR fcTT4sjxL0KOdznB1Y+1iAMj7A== X-Google-Smtp-Source: ACHHUZ6zp07NG2bkXXxglRYaIEoFlfKXA7hhsSA7stt9xCgrqpdelrec/tU9od+DRY9y6jYGAzcstA== X-Received: by 2002:a17:902:da91:b0:1b0:3d54:358f with SMTP id j17-20020a170902da9100b001b03d54358fmr20812342plx.0.1687424243512; Thu, 22 Jun 2023 01:57:23 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:23 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 25/29] mm: vmscan: make memcg slab shrink lockless Date: Thu, 22 Jun 2023 16:53:31 +0800 Message-Id: <20230622085335.77010-26-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AE8CC1C0017 X-Stat-Signature: 7jbj8b36c6iei4fwebur7c4ac8da3se7 X-HE-Tag: 1687424244-449013 X-HE-Meta: U2FsdGVkX1+2tabyPXA8DN/m/P+SDG0KcVUTR36QTxMSKgt02KG0K4L239Ah6i+IqPhycPwyOkBt8Bkk4YYdrifWdELBPlezCtGzp6R50qGMQ6jo0iH/uAd+4n+PbbUElgaZkArf2/mJqXA6BUHKBdWH/LuydnEwomwS77MwEE6ODbxxGKbHQE3LZKb1pHS7AXe1nwNz5F6IMO371CI3Trt1CxqMqWjE2QDLD3gRHlamKDCx0oCHBSexfDYDJ9OfwkSrKrbKV62qrCfItj137eXGqRPUkcK6XNfYEg92vHPLNgBE9ITVXNyefimGUSmabMklgsuIEn4QYOX4ogxn8yC/XRawHvK3Y25bpRQybVLKp/Ihgw/RliqYrKicTSmew41mwv1XCjTXpIBll/EJd7KTBoeDlI57a6bMJlaKtaHSSxlTSNBW7AmDBVdxkbFrhP2n6E4OoPh41tJblqjKh98v+n+quZ2ao/VTC1wxTOMX3PsJZxjjt90tyyeJ91o4uXe6L7g3dqR4bLoI0Z3ucKUEd5WMnrwK1IT2OriAQrdofh6ScpfbJ3atcO1bDs6ljO4j5rfQLcdPyA/youa6a6Ilfyznj93Zbx+rDxGBwjNZ0czgtm3v3qZPgPMBHwW4XyRA53QNn4sAWprRRWg0KeiCy3pHlDrTPXPN2yNAkr9cMu5SLaFbZWPlF6gqPwrlfS8tPOLwUxbJY3Nh9AdLYGuNmxN/R6XOo3/YsS3ksERkyfr+VUFxBIbafRQMJlln52z1VmLx68wfZj//TBSB8RirXR7rYMf7EwercVUv2q6a4Fh0Pd/mrMqCINTern4BVwH48ZzR+DhC5DoZb+MtouTb3TxUnhf1Uoi0zO6gARwupcEBwBuFahJd6ut19H9fWL5p0v6Q9n4A6CRgEzUTmHDcIVw5UNjYbnunGoY8+ZwPmdCl16EwRs3xogIYEGxK+9VZ1xBkUrL0ft4pYHL lsXWn9Lh elqZ/EXjgK1Ydnwl1lyJh94E+Egg3CeCdDjDAjzzfGCxZ7j36/rvCgDNj/QAtGDnxZl5yHoRGgjhBd+yZ47A11DH6r/2O+nG7+uni/YLj0pTyTGfkFp811bGol82DH7GVso3y8kOjkQlYdhthr+grAz0nT6JG1zz6jsIhPf8AQuCv6CXKNjLYi85wZwb4qKsAUZR+JrX/du6eZs234i+xsJkX/6W0MNBXfSfczRlFd1rwOgFA/N6oOvnhLf/2sJIOM8EfY4/ON3rYtE+6TfEojFUdFTi5f74K70Jhkxk97/bujtEadY9Mcyllu8zP5A6FzvBy5oDNU7pXylT0R8B2Q1pIqCGVXl5bNIxxcLzs/CrLvBQVAXEft79/b6vk3tqwhDXaNhpI3n85fP9AfJZLCzzvdzAFnhw2biBn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, this commit also uses refcount+RCU method to make memcg slab shrink lockless. We can reproduce the down_read_trylock() hotspot through the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test mkdir -p /sys/fs/cgroup/perf_event/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 [-g] 1) Before applying this patchset: 35.34% [kernel] [k] down_read_trylock 18.44% [kernel] [k] shrink_slab 15.98% [kernel] [k] pv_native_safe_halt 15.08% [kernel] [k] up_read 5.33% [kernel] [k] idr_find 2.71% [kernel] [k] _find_next_bit 2.21% [kernel] [k] shrink_node 1.29% [kernel] [k] shrink_lruvec 0.66% [kernel] [k] do_shrink_slab 0.33% [kernel] [k] list_lru_count_one 0.33% [kernel] [k] __radix_tree_lookup 0.25% [kernel] [k] mem_cgroup_iter - 82.19% 19.49% [kernel] [k] shrink_slab - 62.00% shrink_slab 36.37% down_read_trylock 15.52% up_read 5.48% idr_find 3.38% _find_next_bit + 0.98% do_shrink_slab 2) After applying this patchset: 46.83% [kernel] [k] shrink_slab 20.52% [kernel] [k] pv_native_safe_halt 8.85% [kernel] [k] do_shrink_slab 7.71% [kernel] [k] _find_next_bit 1.72% [kernel] [k] xas_descend 1.70% [kernel] [k] shrink_node 1.44% [kernel] [k] shrink_lruvec 1.43% [kernel] [k] mem_cgroup_iter 1.28% [kernel] [k] xas_load 0.89% [kernel] [k] super_cache_count 0.84% [kernel] [k] xas_start 0.66% [kernel] [k] list_lru_count_one - 65.50% 40.44% [kernel] [k] shrink_slab - 22.96% shrink_slab 13.11% _find_next_bit - 9.91% do_shrink_slab - 1.59% super_cache_count 0.92% list_lru_count_one We can see that the first perf hotspot becomes shrink_slab, which is what we expect. Signed-off-by: Qi Zheng --- mm/vmscan.c | 58 +++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 767569698946..357a1f2ad690 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -213,6 +213,12 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, int old_map_size, int old_defer_size, @@ -339,7 +345,7 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) struct shrinker_info *info; rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + info = shrinker_info_rcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); @@ -359,7 +365,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -392,18 +397,28 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, struct mem_cgroup *memcg) { struct shrinker_info *info; + long nr_deferred; - info = shrinker_info_protected(memcg, nid); - return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + nr_deferred = atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); + rcu_read_unlock(); + + return nr_deferred; } static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, struct mem_cgroup *memcg) { struct shrinker_info *info; + long nr_deferred; + + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + nr_deferred = atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); + rcu_read_unlock(); - info = shrinker_info_protected(memcg, nid); - return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); + return nr_deferred; } void reparent_shrinker_deferred(struct mem_cgroup *memcg) @@ -955,19 +970,18 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int i; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); +again: + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -982,6 +996,10 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, continue; } + if (!shrinker_try_get(shrinker)) + continue; + rcu_read_unlock(); + /* Call non-slab shrinkers even though kmem is disabled */ if (!memcg_kmem_online() && !(shrinker->flags & SHRINKER_NONSLAB)) @@ -1014,13 +1032,19 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, } freed += ret; - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } + shrinker_put(shrinker); + + /* + * We have already exited the read-side of rcu critical section + * before calling do_shrink_slab(), the shrinker_info may be + * released in expand_one_shrinker_info(), so restart the + * iteration. + */ + i++; + goto again; } unlock: - up_read(&shrinker_rwsem); + rcu_read_unlock(); return freed; } #else /* CONFIG_MEMCG */ From patchwork Thu Jun 22 08:53:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D9FDEB64DA for ; Thu, 22 Jun 2023 08:57:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE8FC8D000E; Thu, 22 Jun 2023 04:57:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E98188D0001; Thu, 22 Jun 2023 04:57:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D600B8D000E; Thu, 22 Jun 2023 04:57:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C973C8D0001 for ; Thu, 22 Jun 2023 04:57:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 97032B017D for ; Thu, 22 Jun 2023 08:57:34 +0000 (UTC) X-FDA: 80929780428.06.F3C6706 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf14.hostedemail.com (Postfix) with ESMTP id C63A7100006 for ; Thu, 22 Jun 2023 08:57:32 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=I9WP6mv1; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424252; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YpqAnPCFjvBxCUJyQNRij7IdVYJH6Jib7EICYqtSs4Q=; b=eeYLYEdpJ7mAnK6H/1VHg+46MtzfGne+0X52lL7Q9bh9esBCVn+C4zV2iYrxkqgszYTQPk zCr+LXUT7RhgRsOmB1/nGuhXi8R3jFaTM5Osl9205zfn/E6t1lh1+MJT9QgFtWTyUKApxO 8/bY33DSTo32dq5SV5pLzqBv+7qpncs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424252; a=rsa-sha256; cv=none; b=OQdiJK3I1nyID4ZBfjH0P5Mv8eCyRsStP9JnXCsVdcqN0SKhfOJMWqZezikdmSCNrfObif 6eOIIW4ERWhAwOgiJe0/8gI32bdiLrQG1CXgYE/rmeKXXHh/nM4KEwvXHgF0SUIGPJTfc5 rg83BoNLp6ZEP+Av5O9+ULpOVRiKxj0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=I9WP6mv1; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1b5585e84b4so6829415ad.0 for ; Thu, 22 Jun 2023 01:57:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424252; x=1690016252; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YpqAnPCFjvBxCUJyQNRij7IdVYJH6Jib7EICYqtSs4Q=; b=I9WP6mv1rDRCGtgaLzA9lhBXeFPAHaj8y0iyY2DOjVEWCsT2F7nfNJVwyIikm+kMao zO6QJGdxbeP+Xd2yLUuD3B5OfZQRm3B4q5mm2BLvCFNnuFgYbDreBXTpY911jw5PjLjw 5qBcX7//daeu9fw2bM5v9uWJQ3Yj963JAAdMROcF9Ug6KJW+BwPKjtr4qnEx+svXyNj4 WQtBJXjBe1rX51UFB2Mb+BbnprNNOvKWLYHk9IXF69UQEETABYtrhl0ZHyku4x/+376s ovmIBUSgYXrVJUG8L2WcJE6QBBiCJCq44tzgQbbUxJYU3EpXiLR3t2TzR3wvfdUTA7wG r7SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424252; x=1690016252; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YpqAnPCFjvBxCUJyQNRij7IdVYJH6Jib7EICYqtSs4Q=; b=NzqwFXaVfUWiyNfxTCsWcxHHY7Km4lyDD+r8iJuFOB3+QyiURPYJLjX0I3/vdCCIXj ai/IISSMMAZ9GoH6EfqdF+9QMMdTH14kXY/3S2TUm6bXUho4xZ+wK9CVCe/3AF/O6LQe bk0pf1t5V3JUweKxn1qUYHdxmhcu/aQ0Ygy8wYmQra2eY521YZNyiMZFov8QS1aGTca4 MFGq2y9x9Zw+ZQeYXNpjtcqBo8xULS484l5VGlihBjMorAUlDDQBZj5nW04n0uYlvgGt Kh9T4WWBc9XyomqJSgzyfP2dya7FNLpWyisjZU9DZ8fo9iYtzqUUCT6H+4mQBYM8MIFr bRUg== X-Gm-Message-State: AC+VfDxKnxU2YmJYpDimUkdtXcmEwlDK/1yJv2cq/i+Gb+IFVj0MLIjr 8K66wJp0ZpQnE4Qz9XY0DKUmKQ== X-Google-Smtp-Source: ACHHUZ6FoF5W+f0N2ZUpFBUVBh1vs3Mj6G9dcM80ve0PhcTP+4niqZQRBrFmYJGvwbhs9MGB3E6MuA== X-Received: by 2002:a17:903:32c4:b0:1b3:e352:6d88 with SMTP id i4-20020a17090332c400b001b3e3526d88mr21673124plr.6.1687424251794; Thu, 22 Jun 2023 01:57:31 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:31 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 26/29] mm: shrinker: make count and scan in shrinker debugfs lockless Date: Thu, 22 Jun 2023 16:53:32 +0800 Message-Id: <20230622085335.77010-27-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: h7jyx53ejrww6yest4uwf5no4m6mywiq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C63A7100006 X-Rspam-User: X-HE-Tag: 1687424252-913621 X-HE-Meta: U2FsdGVkX1+qP9Doz+2u6PNsHgrq41J2FW6koa3iqj+Xt8CApdj1dwqtpwSZ4cxItl/7lEszUgUBuqdMLcaQrdGYnIwD0EisV5/lyRagsOWSJajELgME0WUUv/5uIdZ18ZjcozLy0onmawlJtIHWL4jJ5XkPZJptt22Li2Ozcq8mHT2/Rad7piEzzJVagWN1b6wnjDALZJYc6bu2LA59p82IiHqBJRnTqq8NNFwzGLbRNOWAsjbrPcfhNbUyGVc+otkg/Bou0fKocKGNz4X6Jx7S2m0iottpUM/22fxdRjBnyeowLGw868D48l6pShCEc82ZxcW16moaSdDiJO45apVXgCYNfZR4bQ3PZs5EBTbH0v5/I+lZNXF1rPqxCx/dLLlulcLSnt2JUpmWq/Zee5cu6xX7Ehtkwo33IiXPjgWCynP8UhB0VpFzaHat03qgMx2UHLY1ZEHfvdPWda4uOoCN4wfBkPU1uWD85mAqt8ZzDZWlEdU9dpVhIlC5WD7st4DHxYOgbnl67KgQ+6B4rofT7wC6QbhN+M4Sdm03IAQfD6blFC2H3HQVhsdz0b+uchnvCvRgzXz2w0vvr9hmjDYkM7jDeN7O9IpES+i3kG+hBxujmrDqzR+HH4+kRqGLOOJNNhM8WFM4Br7cD/xakVdXrp6m9p6PX8OLuRYoc7LBCKwf3HJIG3XpQHQr2KWi5WTGN/jwgbWTYUGy24Ab0ON1NcdelzV6dFbDYQlpnxnZPhL28Z156fuTE9kwVigFaOWSvpTNOv3uhx1AuMp/MZMost96auA4AjyJNVNE5upbYjCHQTPKhAQ0CfrFV49uvVDrv7RXdEegWioTlOrjOYOslJk41KPCBfNoFdzHetaZQ5NIbvgrcF3JGyDU6YMYqYn6hOLJk0Uou/fFXpdJP5qNo1wFjzsYHMnkRBKCeHS7HXy6z2BUSIKISRKclYFeAkK04kzc97YTynyeYdB 2L8se6N7 JutFgmu3fWaj0/m1odl4ap6KBM9Fs6VkeuUd44Sc2TsufXrvcJFRNvJVhlk2qmpxeP7TZoSe+4GatbTqVRnXQV4/KmJ08YV2acjfb9+4Pm5HahS1RgPWSHl/tu8sHEIycfXl79UCROKrdyUz36PjqcmnwIgfarSc1jh3K364rCRhbR0WMXGJZmyCC28CM9EWNAU6ASu6cTvX4v0iujHBiV+bPMCUQfU6z1DuS3uLQN52DayL83gX3z0aGhEffya2ZaO5mRKHb20vYTpbN7RrRuWzJpkD89+tddORT7AUk/1wqS5qR6Zq8is295/X4FTcfHR7ZOsPk3Ui4FXKHTY7gFCob9+7XUmCQye7Dm9VyzmYWXySbEizs2l5cHVroxHt/cprIbk0dci8lakSNme0j5W8sFdpz7HEESpyO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also make count and scan operations in memory shrinker debugfs lockless. The debugfs_remove_recursive() will wait for debugfs_file_put() to return, so there is no need to call rcu_read_lock() before calling shrinker_try_get(). Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 3ab53fad8876..c18fa9b6b7f0 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -55,8 +55,8 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { + ret = shrinker_try_get(shrinker); + if (!ret) { kfree(count_per_node); return ret; } @@ -92,7 +92,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); rcu_read_unlock(); - up_read(&shrinker_rwsem); + shrinker_put(shrinker); kfree(count_per_node); return ret; @@ -146,8 +146,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { + ret = shrinker_try_get(shrinker); + if (!ret) { mem_cgroup_put(memcg); return ret; } @@ -159,7 +159,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + shrinker_put(shrinker); mem_cgroup_put(memcg); return size; From patchwork Thu Jun 22 08:53:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD120EB64DD for ; Thu, 22 Jun 2023 08:57:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C0228D000F; Thu, 22 Jun 2023 04:57:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 570638D0001; Thu, 22 Jun 2023 04:57:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4604D8D000F; Thu, 22 Jun 2023 04:57:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 377FA8D0001 for ; Thu, 22 Jun 2023 04:57:43 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0CB8F1C810D for ; Thu, 22 Jun 2023 08:57:43 +0000 (UTC) X-FDA: 80929780806.16.0458B99 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf13.hostedemail.com (Postfix) with ESMTP id 3C1532000C for ; Thu, 22 Jun 2023 08:57:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Kv6dWh2I; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aRQUwtIbFrjRbKaWDWITc6Upds+hXxxlwcQqvFPvJow=; b=zfHRe1Fhk8MAL2GcrqeUovO7fRWk9/pgL/Y/8AQvs9quLA2danVoBUDbTArF8JGC8xVF9a mUoUcZCpaYOzrNoT0XGbvGyauGiW/TdWCi6AoOrwR+VhTt5tuYcZzNQ7IO8ylO1IIZeZoF H7T5S/tRoyMAxVVh5MMjxg3qB7kNaxY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Kv6dWh2I; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424261; a=rsa-sha256; cv=none; b=2iH1Zx64m1kOvC0k6nM1tyG72X/HneShnwwddpNcFoQQyOACEJTpPTItUnI5g39Q5p+prD PdW58oBQoGBYwZQfGElNBAEoaRlstFggazHaWyj5rEvWXlWf8fum+12pPJpGHqbvT7aR75 1TBNWd/HTz2BMNNUQdk3PcEOVNiZC/M= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1b515ec39feso12624225ad.0 for ; Thu, 22 Jun 2023 01:57:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424260; x=1690016260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aRQUwtIbFrjRbKaWDWITc6Upds+hXxxlwcQqvFPvJow=; b=Kv6dWh2IIchnBu9RbCVLsx8HZLugRgYdolMOnFdVv/Mze68pcgFvadMFaompRsV/Ii Vr4ahAwXirWdfwADAjPFAtSTivwacduawKzyeKPVnp7cESQMTGYfaqAI7425tQvAdQ85 8rLDDpZCwcAbH67+6k+hMmW8Fi1etqH6LYFHkuQfbD+ESFiwPmeSocm5kXqI50kDiSjf PCd9vDumWTECSSKSpOEWhL8aCyPGbcSxO0SaB85LOV7uPD9a5LVIfvAATYV0qgfKCSQQ J+7inf8lDO5dvyUSQTFKKOcNdRaAbUMX2WbzVCHQayCH8tZA09PfVWiof+vKchWmVnWN 0e0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424260; x=1690016260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aRQUwtIbFrjRbKaWDWITc6Upds+hXxxlwcQqvFPvJow=; b=VBytiE382eWliQZ3RxTgAUj/XTJ4/tIQtLHauTRdj9nVbyWMk7imHnBPOZm8HtkQHK 3/fSDZd/R7URXypbejX7naXqjelrs1I0YwMtLXCI0p4L5nu75phPmEYXpifmDxmKZTq1 TmZWG15svXtNg0rHyo1ZipgendaySmnxIWVJoF++JIkoCMebWR1GLw8Fj8oZIQUL20kN e/1iBWx0KUc33FUbu+stdF2Obx8CrU/FIk0TGoTl+2Z9Lrn6E6HAG2h3S6Mba0oIXJ00 XJ/iILzRdBn1MiE+3I0CwmF/+Vj23xuHb1L3HoXyGdSUjFOewJ9DQCxHk7tpofKqgjQo wvTw== X-Gm-Message-State: AC+VfDyFE+ltHDKkmC/qnsJ2x4037liuuaC0v0hulYlJmvWG6GVfjniw 8oRgdLkBoL+H4opbvIs9I1lo+A== X-Google-Smtp-Source: ACHHUZ4KKZ8AXrupsFYw18EsJWvGfL6qm7cyYV/rSPTKOdYWRQ8IhOkLOim7I9s7W96jDohmbJS9Lg== X-Received: by 2002:a17:903:41d2:b0:1a6:cf4b:4d7d with SMTP id u18-20020a17090341d200b001a6cf4b4d7dmr21650808ple.2.1687424260203; Thu, 22 Jun 2023 01:57:40 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:39 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 27/29] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Thu, 22 Jun 2023 16:53:33 +0800 Message-Id: <20230622085335.77010-28-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3C1532000C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: b5hwup9nrwn6hjubhrpp7um4pi5tu7g3 X-HE-Tag: 1687424260-38298 X-HE-Meta: U2FsdGVkX1+fEBHUsXeOZ0w7AvuJjJZk/ZCN0yE3f3xx/MH9PS0ymKopRWdHIHkqNYhscKYDnt+yf5ttM3DM9E32wp2AwotfcD8pbq5Q8x17JoBp+jBep0KzFBlTGlxUkDiekzPUw4XHxwL4JeWaalbEHZ2T1DnEyA9gQHbi/frUqIHp1MrNc5ZEaB1O+B9+6WyHkN60dOo/WEX16Qjeiq6XTAR9xhMX7JJYs2MjEN2dSXVKCFPm9/Q13okhEEsiXLDoXaRd+XP58i/8xPUNkS0a9zzoDuhOD7dUpCeynDuz6nPhs8BkAUJO/Xiaw4vYhTYPvo5aE+Jsa1oKKNbjKJ/LgTaQkScYg7wTL0QPyqbFcN2d+wCSGDyKnmeZdTakgASxCVNMyIEJaNXzJqFiJnBPyE/buStrsJhKD0tcddIauo3x+fed4G4uJIo6evpwWE52J8NBk5qqYGxS5rDJjUc7FbuMib6wrWDUiA0JrMQ0EzXmjXjztbQtRtfqlLzXtaVrlsK+9h1n2nFZKwfb1tPmScbfuq3CGZFEUBxB/TiRdUO8sdmedcpnvLSdKOsLKUz/uXQkXyXsBUtEudFwhkLWwOFCzg3vKLb0CyqTj2itOpby3AjtlHAwhiE4xRyKFmvS2YuM/3fNjPezS3UyUd51QRsz4aDvE0nOCYcg/c6VAVq0PKoGblB5Bi9tY8NxvuTX+HQnGCF7JgWvuyPEpHVQfgjTGEFf8aJpke+hmnVcGw9zidzyYJ4CNno6wSJJSnsQxvS/yzwNFj5x2MWQutW4zv3E4f87cxqT28UNCgZPpJqprHVAOWkwfxvtxSVXkRZeyILukcKAf4af2cfs9Q74/haRL6hzxLwOjydS4rvf1h5i4sRXy01QXiXmmybAFNTA5qI+GFWc17b1l8yThhnGzA4vSNFEbBXJbqoh6R+IJNQKkdHgdod7dXv07sS87mdsh6hhTQ0Et6BlEk2 7fd8uFG/ wAl1JZi8NPgDQuuIlweRgEt+6K1toM7ntKc92pmGHOvxycUUT8c21BtTuKJzw7HJXltvNT9cUuJK7Gc7O7mIVbPNgmj7AxrPXPV/62qx74FvIfEELs/Q/Nbx4Es0QVDLFUPK45cw7AetJhKVbc1Z+ngDAjyoNiREXuB7LbVWCbx53WJVmKU08guovmXps66r1H9Q3vsRt06HWgX1fmecZCUbwJ2JlMFgs9/SdIzHN12/EYjbgwyrE70HWAKkSuAfWaKkiet+qKWa7F432yATKtrAwaYIJCL9wB2GLL7wj0X3xsH67Ka1ks4wx6FK36w5PMe8has5zmWdJtTRmZBwLgcvuwdSAlRBmoqE1st5U3HDDH0IMxuWW9lI7TfRZ6JPoimBLu2nSl75b9YmYvA3oFnfJsh2aNscJUlh8iTVz+xMBe7B/rbI2yGeQiIp8Vru54Z8aYp9dtFHxXUkGtFVr1Z7YCA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 357a1f2ad690..0711b63e68d9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Thu Jun 22 08:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6024EB64DB for ; Thu, 22 Jun 2023 08:57:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 746D88D0010; Thu, 22 Jun 2023 04:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F6CE8D0001; Thu, 22 Jun 2023 04:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BEFA8D0010; Thu, 22 Jun 2023 04:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4C9E98D0001 for ; Thu, 22 Jun 2023 04:57:51 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 28E91120AE8 for ; Thu, 22 Jun 2023 08:57:51 +0000 (UTC) X-FDA: 80929781142.21.FCE18F6 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf22.hostedemail.com (Postfix) with ESMTP id 42C36C000D for ; Thu, 22 Jun 2023 08:57:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jvhIGmYE; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+P5gj8bno5Yz104PoCsSeC7OSXGzrjC+L9aN78tCOOI=; b=afRw/mFxj2QXxpKRkwVVqyxDFVOWulhj21PUPHgy3JZOaRZbHjUrnUlp8rc9YlifHf/S+h ARm9xkMfjcQdjyYDguHPCF9iq744RbKcfxOHDiRbbYvtbLGz2oR8HUttXqz4yrqhPLpKH/ Hj8jsLhL9aBdFzbsnfHnQsK83Wzs+vY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jvhIGmYE; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424269; a=rsa-sha256; cv=none; b=j1SYZRt/ulX0UhIscgouI2sMl2k8jYXAEfG6cHz5VG1OmTOn4AFveKR+eiCZupU44DHfJS gVF8K6ikD/sjkb0CImmIj+aYXIFPBoJWbFcUsast7g5XAi7eD3ELfFQTPOP4+dOyinafpg 4vm/zenKbIPYOQ6YK7TVEBEuYFd0gXI= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1b693afe799so2467045ad.1 for ; Thu, 22 Jun 2023 01:57:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424268; x=1690016268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+P5gj8bno5Yz104PoCsSeC7OSXGzrjC+L9aN78tCOOI=; b=jvhIGmYE039/syWqcfLyIdo6t9Qplyt6OsSxUBVvxtsPAkw48Ft91VW4Wz2O26wmhx R0vxBU2Gya4gcRkfUKW9xk7qtBQ/FB/itjiP7zdM6kYiVPyrZbA9p3LfIx3f71WW1xO7 LLQ+0DljNwUv19wViW0KnJsUJ8+fuKrQN9IBdM3/tIdgoFSPmAiPoFKu75bet98dvkR/ 1RKnODQtDxONo/sndY6eLEsUvIJFr3WP9l67MNEVgdfQojcwBUWJp7hhTFie6SRhHC3h g1EMHwWUjxMnky7BCMGJqXeVS7+DBkpxiGCDfvbg7pDqaD88DLgjwvpVG1NnvXM5z1NX rt6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424268; x=1690016268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+P5gj8bno5Yz104PoCsSeC7OSXGzrjC+L9aN78tCOOI=; b=WW5dW+36ymRazcZAZjjxs42988msjYYgygpPIzwy2iM5InBBydAeji0CCnE7nyc4sH noup/vF9ZbEEAyunBjSgeQ2s6R4Z+utrLLZADuMB90J2bkKMetfrRa30Lsn4+IFWVjw6 m3F+curaEfsaXk5el5b9uvlj2R60Xe8mPqdVGOffRfbdp1sfiWBaG6DCbnYcM3js1WSg lMhSgt4Pqcrf5ZoeO18vcu+590MLBfNASYTyyuHC9qhsn+qEXQV2NfYDtt0nCl+WRdIQ Yh9F6MMcw9v5OLqmvdXRJu3VPUgIxV3tGKKgITIV3IZo3e9owr7QazS/jdNpxM9gDTKq QBEw== X-Gm-Message-State: AC+VfDzis5Lqj8a0wEP+P4yIf0EwlGxy9Yw6BQzq/EkODV9oQHUWCPEX qRMBQcdud0iITdXqn/fpLCCEIA== X-Google-Smtp-Source: ACHHUZ57rUw3vY36SztXPXN7lhzSsB9eXMClAkEXukaFLir2ldlvIHkHUZcbPwVIoKn5tOhLnnwBWg== X-Received: by 2002:a17:902:ecc6:b0:1ae:1364:6086 with SMTP id a6-20020a170902ecc600b001ae13646086mr21606033plh.2.1687424268179; Thu, 22 Jun 2023 01:57:48 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:47 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 28/29] mm: shrinkers: convert shrinker_rwsem to mutex Date: Thu, 22 Jun 2023 16:53:34 +0800 Message-Id: <20230622085335.77010-29-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 42C36C000D X-Rspam-User: X-Stat-Signature: 93dq5ppxb57nr3kuoowcch1dxknqehdk X-Rspamd-Server: rspam01 X-HE-Tag: 1687424269-788024 X-HE-Meta: U2FsdGVkX1+I7US7RDEMjsYmfs/3W+u+Q7H9mDu6a7P3EZAabYTWMOI6YG5T6Py4SA9q+0g0GQRbCpmHhE2jfcoxWuOiSP9VXxAdp8dBWPD52Rg9mud/J2evD9WSHfO8a6A89gxyDcvdBEXgdJBJwYcFEaRYl18uJp9ut3w1E0/hfegydhw8/zaVAfBN55wZIlyLdiTfxryafFdIF5nHCc5S99DlG4Z+U0JKRWfvWQVqbEB+ulLJXqQXBC7Ln61J96QLuSJSYLdWGO+swf/0nyRIjFwdHkRHM3uzzdaS2imsDDL59P41o68A1eHZste8eAtSuAQ9lC1wk4p02tARsUhZLkVZ5Zek8Yt7r5R9qDftRrhfCZ/DPZ8g14JFLCQqKQrN1Iv1cs/na28kSPXTgOhqy95SY5MJW3O5uhUviyHCYxoNx7cLjvYoC7sYDXkOZEfhuBto1PI0qh6YaQC4vPIidJbv6zkM3W6Wy7ed3mO3pnIr7T08WRwT7du77L/PSiNYmMhmaG2LVfOujBnMv9sEtGip695CvK3ZsRwaA4SvO44DzLpCNJhxBqceVKCTuNLpFEsU4UAbHgYdEM4YyiSCvC2EolGgnnqldi6MaLJlketi+dS2veyAZ6Ly+jJnUEckTRZ46YZXRH86PDTd2H9CKeVAnF5t7FcD3WuE07ZBuWerwx4axtEzQCweg7YYNLi4g3fFeDkEfDl2NfZtsveX6RtIsVOwhWr9DaFzO1B8FMlSQ/66tYiUfNuLRNurdKx8xpedOJNWCMlE4L+X5dD0ZJ+eXrp6o0KtDucFk90yF9PaaOpJgolw/vQNWgRwRqotK8AvS0wJ/XwMjsSJWHatgyOykmxAm8ANHHLp2f9H3rONLdoK5XLlTl4iqc/reyn6ozGdkgtCc6UjiPTDwDvNzXEuosaINmBEfqdpySdJr4yX9/Vw3qu9YNYbKhYMlmc9wgVU1Hq7sCNQAXk vJDLHFWp 8Zyq9WiN5XnsRCgXbFtGtPzv4dX42ceyDOPe+2OnrD34mdl6HEwB7drvOkQ+C1OTe7Br9XEQjXuV9jh8xFFDrQsyPWENk13vUmrp+lZEefCHP5ltmgSTe77j9CdWp3EOiSBtmqzHtU41DGmgzuWXkA1CHD0RJZmWy/m+HeDFExDVJZVpmrOAaqQOlnBo6nic46FfccpNafWVHC/xq6nfTZvc7ttTslAuWAIf84V6/0KrbRGBBpIxZKGjB4AhvhiQvKqX5EkbFs6uw1ViZ7D7nZotDH6MCPXWrRQW9bAlu/lz+t4hX6jlQpwVsTOQu73K0lU5fMqLFzlahyDgDS0teF4OFv8ATx4sGShg2TwIVtafEi8o72a1U3gfsLqVJOOe4Q2HW81NSMVsXaERmIV+rdM3QsaA5HERseVYShg8N2Jp4r655PnQZBOl9hMyeA8raFzRIX7rnXrhZvP5BW/NIx2/BNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 791342bb8ac9..471800ff793a 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index c18fa9b6b7f0..7ad903f84463 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; static DEFINE_IDA(shrinker_debugfs_ida); @@ -177,7 +177,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -220,7 +220,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -238,7 +238,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -251,7 +251,7 @@ struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -280,14 +280,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 0711b63e68d9..bcdd97caa403 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -190,7 +190,7 @@ struct scan_control { int vm_swappiness = 60; LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -210,7 +210,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg, @@ -283,7 +283,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -299,7 +299,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -315,7 +315,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -364,7 +364,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -378,7 +378,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -388,7 +388,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -743,9 +743,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -755,13 +755,13 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); refcount_set(&shrinker->refcount, 1); init_completion(&shrinker->completion_wait); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -815,13 +815,13 @@ void unregister_shrinker(struct shrinker *shrinker) shrinker_put(shrinker); wait_for_completion(&shrinker->completion_wait); - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); shrinker_debugfs_remove(debugfs_entry, debugfs_id); From patchwork Thu Jun 22 08:53:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13288661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1548EB64D8 for ; Thu, 22 Jun 2023 08:58:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 524438D000E; Thu, 22 Jun 2023 04:58:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D36F8D0001; Thu, 22 Jun 2023 04:58:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 300DA8D000E; Thu, 22 Jun 2023 04:58:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1BFC28D0001 for ; Thu, 22 Jun 2023 04:58:00 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 92EA440467 for ; Thu, 22 Jun 2023 08:57:59 +0000 (UTC) X-FDA: 80929781478.24.F2B01B7 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf19.hostedemail.com (Postfix) with ESMTP id 7F12B1A000A for ; Thu, 22 Jun 2023 08:57:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IAFXPAQd; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687424277; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PAKGQW5AKOb0i6yhTIQHZdj2Ew7zYpKm64DNJQ5dWlo=; b=VjxA8bN2nrU59DyKSSaMBZVSyv+tGec8Q4zSSb16J9KXIzaH9zHFNiDrpx06uBihMc56rt 0PND/KThjmK70xWU/m/sinrpkOUsefI84NkC10fDdy8QekOJ4BznvQ2F2fvontbCUWPPn3 G/ZOMtsMD++eeuxRTguObZWswaGgM94= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IAFXPAQd; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687424277; a=rsa-sha256; cv=none; b=zA8stfFiaSmnvNx+BKY88q6oJYMo7VhZaZNidiyDwik/CuijxP3W6VShmCbDFXkxicGAa7 XZSk9H131/+Sr2nAY2iYfflEhI6qgVtbNzxcKM69/0FZ/UYy8H0hurBdjBszhVqTx3SNR1 T8xfhZQquynCbKYz/w6UXjTbx7wS9oo= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1b5585e84b4so6829885ad.0 for ; Thu, 22 Jun 2023 01:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1687424276; x=1690016276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PAKGQW5AKOb0i6yhTIQHZdj2Ew7zYpKm64DNJQ5dWlo=; b=IAFXPAQdKRfFFdX8R+urqMUX8XywDjZlIZgvkiyvxKq3OuDxGDvyj53uf/GwQxOSgj UCrH4ulAqIMmQ7X056WmoGhyeCnVWVEc9MkUT54moxveVQUVm/n1V13P8OOuQyRngFVd lkgqQbjTLhW3PPA79SQnKo1nifBNALOotV0hDjlRRH90/2Gp8IdTAPqu+2ReapWlGMbq zlnEUk3uncMYp9LT0mLOQRJqCv6RJvoFO7mY6AD+tEO3DKYoCqm/BoBWKqYPC/hMbbbK /L8GxpwKn26Y52nmXelpvPk6gmsdpmYPKF8i9Fu/0MG+e/S0nlhty3rfnlUBtkssiN/B zjDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687424276; x=1690016276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PAKGQW5AKOb0i6yhTIQHZdj2Ew7zYpKm64DNJQ5dWlo=; b=aRsoveUnTN0tNBJxdN3C28CycFYMCjfCClEbOfDAcKgeSsx9IKP5Ir5dV+HFbCA3zq wdncEn5pYkxCUw8G09a2O8vCZ3gnwVSzF5M39gjEhhuqCt0DmxyL00AxH2wQS2q8BdLo dPCDaPoPZAX0d0Me6X3eOy8ZBhFQKDLkXX9DCMpsJyzZ5IJI6vt6U61AOtQvoR4t8nv/ 42pfVDkf/6dt+W+nBL0WJT2qhxerpObRsnj9XS9mQ/vGAIAfPnPVJaNsZGxOgOMklFvm j/UhecLpTBQqf3kfy2A5pZLW1lbNxyCK3r3rHRgUEzjhcuazNPZyIvmYATHOhz2ONABS AziA== X-Gm-Message-State: AC+VfDyA3Y5q8LkH/UxNu5aRzYUoUb4jStrr7uZA6jqsWkwM7qSMx8p4 30CL5H7C6630Wfb0V62kSTUttw== X-Google-Smtp-Source: ACHHUZ7H5apRKh4VQCyo6UXxNAhEyp0xI76DSHJvkAU1Y/8mfRtLEhaUfJSBdNPbX02WDiX9mNvMOA== X-Received: by 2002:a17:902:ea01:b0:1a9:6467:aa8d with SMTP id s1-20020a170902ea0100b001a96467aa8dmr21676629plg.1.1687424276223; Thu, 22 Jun 2023 01:57:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id h2-20020a170902f7c200b001b549fce345sm4806971plw.230.2023.06.22.01.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 01:57:55 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH 29/29] mm: shrinker: move shrinker-related code into a separate file Date: Thu, 22 Jun 2023 16:53:35 +0800 Message-Id: <20230622085335.77010-30-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230622085335.77010-1-zhengqi.arch@bytedance.com> References: <20230622085335.77010-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7F12B1A000A X-Stat-Signature: rzhehf6ibxg6c7e11qiofrrpd51qrw75 X-HE-Tag: 1687424277-488146 X-HE-Meta: U2FsdGVkX18Uv22ViJ+T5Qv1A6jgNT7T8lGUPo2VKXfdilyLnGJ4UfE8VYyYYAuKZ7PsyCL+gZtgnRr1GFXqja6xB6lPDzOSnrqxs3VT3hSsz4/CbxeNInr9yFjkYqjkg1j50lymibVTzf0sWJbUTYAh9b7wFHUqfYM53LTPpsTypb7shR1oiiylXACmuHOEXjPerk7EJ663VcxTBAtaRSJ+3kTz0pCFt9/VSs9e2QVubSImFdghNesKJRG2mDXEeCjwE6llj0OzzdJvZx34DgnKE3Xv5Ech3UAgzFKo6KFedbTwmqY2FuwOoAwZxVa1NWWedmnDrAybftlbLn4MliuxuuzGB8f2/dJO2aR6mNHs8E4S227o0yntuAmHaDyw2xmMxSuvgQYyfCz0GEd9Flgi3Vi+zUkdueOE18d7DM9VoH5I4asgSJDGvYwWZcI3zW86bgvBDe4eqkRhbGBSbsC7qZPgnZx7Ec5Dndj8GrD+B++PCMu4yib8NhayXfCgBlGWCnJ0smByGz2Yu2j1kcADMo84RP9vtKMACO487a0XY3LcIJVr0vGGJ65WliqWd8DZmn0yu50WZWGrqfKV0bGaEmdLpL/KjfO8ffQwcFutjEAij6CFyKY3JJIoevhyznPDpgCZDriEvfZhNFw2zB9H5wF0sstfTlgpN2AtgFNU/Wr3chPo6W1eyBobZzSazyU95aIxT2obetDo0KHWfIWSm9AA+3pezx/29CFYY/zj9Mu/Hb5QnUibt/zkdxh82ROtVWd5eugqWMdKqniPyeRAKK0GeLw3BSFRy4uQKLmhNo5UGLvzFrLDfVhbWLsFr8rJNEBKHaRNHYyHSy/3rsMo4UKJhRmlQecTQ2iIpMoYPmO6o8g8a3aDpDSMtuPZ2NxC/tBUx5bf884XtsvgpOclIETY5zj8o+Oa+fXZwpQZM40tuLP1zuv2RllF33Q4gNxE4MMiY14j6KpwHpi GHY8U8ZR WP0Nv5xWRO/BwRIZDQGmGgffzFMyqHXkkJ0KgyrfUEWK68XXn0vkKEcNsCfXDq3coaMAyyYkwJy0t4npJZdo8ReuAQ6ktim/3EqMxNOEU2eVzmhEyocgmd8/B1UUnDH/SsbM+dwEmMKBdAZfRcO2xBFd3/4EVgtbGddBBlEEkFdFW4ZofnMPVOmmMvkflQlWtDrWQrzc9n/k4eJlTodajg28vP9PkJLtPPecUSHWNxbI6Mbfb6ygK8+RRXPyMOIQd6j3kl9jKxSYSk4JF04LpB7dJfL2NbW8kXU7BMnaVPI73djxHm1Qb6t8UYCDw6sNZ5qOokHtAWq5tObU1JNQLQLurjWboNfdLjD4d6o0rAc6SXboGAisuZDaJZeya3Asar+swvMRLOEtRck5Ukc1SOveamqtWWIGsXmrL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The mm/vmscan.c file is too large, so separate the shrinker-related code from it into a separate file. No functional changes. Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 3 + mm/Makefile | 4 +- mm/shrinker.c | 750 +++++++++++++++++++++++++++++++++++++++ mm/vmscan.c | 746 -------------------------------------- 4 files changed, 755 insertions(+), 748 deletions(-) create mode 100644 mm/shrinker.c diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index b0c6c2df9db8..b837b4c0864a 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -104,6 +104,9 @@ struct shrinker { */ #define SHRINKER_NONSLAB (1 << 3) +unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, + int priority); + extern int __printf(2, 3) prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/Makefile b/mm/Makefile index 678530a07326..891899186608 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -48,8 +48,8 @@ endif obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ maccess.o page-writeback.o folio-compat.o \ - readahead.o swap.o truncate.o vmscan.o shmem.o \ - util.o mmzone.o vmstat.o backing-dev.o \ + readahead.o swap.o truncate.o vmscan.o shrinker.o \ + shmem.o util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ compaction.o show_mem.o\ interval_tree.o list_lru.o workingset.o \ diff --git a/mm/shrinker.c b/mm/shrinker.c new file mode 100644 index 000000000000..f88966f3d6be --- /dev/null +++ b/mm/shrinker.c @@ -0,0 +1,750 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +LIST_HEAD(shrinker_list); +DEFINE_MUTEX(shrinker_mutex); + +#ifdef CONFIG_MEMCG +static int shrinker_nr_max; + +/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ +static inline int shrinker_map_size(int nr_items) +{ + return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); +} + +static inline int shrinker_defer_size(int nr_items) +{ + return (round_up(nr_items, BITS_PER_LONG) * sizeof(atomic_long_t)); +} + +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_mutex)); +} + +static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); +} + +static int expand_one_shrinker_info(struct mem_cgroup *memcg, + int map_size, int defer_size, + int old_map_size, int old_defer_size, + int new_nr_max) +{ + struct shrinker_info *new, *old; + struct mem_cgroup_per_node *pn; + int nid; + int size = map_size + defer_size; + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + old = shrinker_info_protected(memcg, nid); + /* Not yet online memcg */ + if (!old) + return 0; + + /* Already expanded this shrinker_info */ + if (new_nr_max <= old->map_nr_max) + continue; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_map_size); + memset((void *)new->map + old_map_size, 0, map_size - old_map_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_defer_size); + memset((void *)new->nr_deferred + old_defer_size, 0, + defer_size - old_defer_size); + + rcu_assign_pointer(pn->shrinker_info, new); + kvfree_rcu(old, rcu); + } + + return 0; +} + +void free_shrinker_info(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct shrinker_info *info; + int nid; + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); + } +} + +int alloc_shrinker_info(struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + int nid, size, ret = 0; + int map_size, defer_size = 0; + + mutex_lock(&shrinker_mutex); + map_size = shrinker_map_size(shrinker_nr_max); + defer_size = shrinker_defer_size(shrinker_nr_max); + size = map_size + defer_size; + for_each_node(nid) { + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); + ret = -ENOMEM; + break; + } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); + } + mutex_unlock(&shrinker_mutex); + + return ret; +} + +static int expand_shrinker_info(int new_id) +{ + int ret = 0; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); + int map_size, defer_size = 0; + int old_map_size, old_defer_size = 0; + struct mem_cgroup *memcg; + + if (!root_mem_cgroup) + goto out; + + lockdep_assert_held(&shrinker_mutex); + + map_size = shrinker_map_size(new_nr_max); + defer_size = shrinker_defer_size(new_nr_max); + old_map_size = shrinker_map_size(shrinker_nr_max); + old_defer_size = shrinker_defer_size(shrinker_nr_max); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + ret = expand_one_shrinker_info(memcg, map_size, defer_size, + old_map_size, old_defer_size, + new_nr_max); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto out; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +out: + if (!ret) + shrinker_nr_max = new_nr_max; + + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct shrinker_info *info; + + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } + rcu_read_unlock(); + } +} + +static DEFINE_IDR(shrinker_idr); + +static int prealloc_memcg_shrinker(struct shrinker *shrinker) +{ + int id, ret = -ENOMEM; + + if (mem_cgroup_disabled()) + return -ENOSYS; + + mutex_lock(&shrinker_mutex); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); + if (id < 0) + goto unlock; + + if (id >= shrinker_nr_max) { + if (expand_shrinker_info(id)) { + idr_remove(&shrinker_idr, id); + goto unlock; + } + } + shrinker->id = id; + ret = 0; +unlock: + mutex_unlock(&shrinker_mutex); + return ret; +} + +static void unregister_memcg_shrinker(struct shrinker *shrinker) +{ + int id = shrinker->id; + + BUG_ON(id < 0); + + lockdep_assert_held(&shrinker_mutex); + + idr_remove(&shrinker_idr, id); +} + +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + long nr_deferred; + + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + nr_deferred = atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); + rcu_read_unlock(); + + return nr_deferred; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + long nr_deferred; + + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + nr_deferred = atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); + rcu_read_unlock(); + + return nr_deferred; +} + +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + mutex_lock(&shrinker_mutex); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < child_info->map_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + mutex_unlock(&shrinker_mutex); +} +#else +static int prealloc_memcg_shrinker(struct shrinker *shrinker) +{ + return -ENOSYS; +} + +static void unregister_memcg_shrinker(struct shrinker *shrinker) +{ +} + +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} +#endif /* CONFIG_MEMCG */ + +static long xchg_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return xchg_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + +static long add_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return add_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + +#define SHRINK_BATCH 128 + +static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, + struct shrinker *shrinker, int priority) +{ + unsigned long freed = 0; + unsigned long long delta; + long total_scan; + long freeable; + long nr; + long new_nr; + long batch_size = shrinker->batch ? shrinker->batch + : SHRINK_BATCH; + long scanned = 0, next_deferred; + + freeable = shrinker->count_objects(shrinker, shrinkctl); + if (freeable == 0 || freeable == SHRINK_EMPTY) + return freeable; + + /* + * copy the current shrinker scan count into a local variable + * and zero it so that other concurrent shrinker invocations + * don't also do this scanning work. + */ + nr = xchg_nr_deferred(shrinker, shrinkctl); + + if (shrinker->seeks) { + delta = freeable >> priority; + delta *= 4; + do_div(delta, shrinker->seeks); + } else { + /* + * These objects don't require any IO to create. Trim + * them aggressively under memory pressure to keep + * them from causing refetches in the IO caches. + */ + delta = freeable / 2; + } + + total_scan = nr >> priority; + total_scan += delta; + total_scan = min(total_scan, (2 * freeable)); + + trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, + freeable, delta, total_scan, priority); + + /* + * Normally, we should not scan less than batch_size objects in one + * pass to avoid too frequent shrinker calls, but if the slab has less + * than batch_size objects in total and we are really tight on memory, + * we will try to reclaim all available objects, otherwise we can end + * up failing allocations although there are plenty of reclaimable + * objects spread over several slabs with usage less than the + * batch_size. + * + * We detect the "tight on memory" situations by looking at the total + * number of objects we want to scan (total_scan). If it is greater + * than the total number of objects on slab (freeable), we must be + * scanning at high prio and therefore should try to reclaim as much as + * possible. + */ + while (total_scan >= batch_size || + total_scan >= freeable) { + unsigned long ret; + unsigned long nr_to_scan = min(batch_size, total_scan); + + shrinkctl->nr_to_scan = nr_to_scan; + shrinkctl->nr_scanned = nr_to_scan; + ret = shrinker->scan_objects(shrinker, shrinkctl); + if (ret == SHRINK_STOP) + break; + freed += ret; + + count_vm_events(SLABS_SCANNED, shrinkctl->nr_scanned); + total_scan -= shrinkctl->nr_scanned; + scanned += shrinkctl->nr_scanned; + + cond_resched(); + } + + /* + * The deferred work is increased by any new work (delta) that wasn't + * done, decreased by old deferred work that was done now. + * + * And it is capped to two times of the freeable items. + */ + next_deferred = max_t(long, (nr + delta - scanned), 0); + next_deferred = min(next_deferred, (2 * freeable)); + + /* + * move the unused scan count back into the shrinker in a + * manner that handles concurrent updates. + */ + new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); + + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); + return freed; +} + +#ifdef CONFIG_MEMCG +static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, + struct mem_cgroup *memcg, int priority) +{ + struct shrinker_info *info; + unsigned long ret, freed = 0; + int i = 0; + + if (!mem_cgroup_online(memcg)) + return 0; + +again: + rcu_read_lock(); + info = shrinker_info_rcu(memcg, nid); + if (unlikely(!info)) + goto unlock; + + for_each_set_bit_from(i, info->map, info->map_nr_max) { + struct shrink_control sc = { + .gfp_mask = gfp_mask, + .nid = nid, + .memcg = memcg, + }; + struct shrinker *shrinker; + + shrinker = idr_find(&shrinker_idr, i); + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { + if (!shrinker) + clear_bit(i, info->map); + continue; + } + + if (!shrinker_try_get(shrinker)) + continue; + rcu_read_unlock(); + + /* Call non-slab shrinkers even though kmem is disabled */ + if (!memcg_kmem_online() && + !(shrinker->flags & SHRINKER_NONSLAB)) + continue; + + ret = do_shrink_slab(&sc, shrinker, priority); + if (ret == SHRINK_EMPTY) { + clear_bit(i, info->map); + /* + * After the shrinker reported that it had no objects to + * free, but before we cleared the corresponding bit in + * the memcg shrinker map, a new object might have been + * added. To make sure, we have the bit set in this + * case, we invoke the shrinker one more time and reset + * the bit if it reports that it is not empty anymore. + * The memory barrier here pairs with the barrier in + * set_shrinker_bit(): + * + * list_lru_add() shrink_slab_memcg() + * list_add_tail() clear_bit() + * + * set_bit() do_shrink_slab() + */ + smp_mb__after_atomic(); + ret = do_shrink_slab(&sc, shrinker, priority); + if (ret == SHRINK_EMPTY) + ret = 0; + else + set_shrinker_bit(memcg, nid, i); + } + freed += ret; + + shrinker_put(shrinker); + + /* + * We have already exited the read-side of rcu critical section + * before calling do_shrink_slab(), the shrinker_info may be + * released in expand_one_shrinker_info(), so restart the + * iteration. + */ + i++; + goto again; + } +unlock: + rcu_read_unlock(); + return freed; +} +#else /* CONFIG_MEMCG */ +static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, + struct mem_cgroup *memcg, int priority) +{ + return 0; +} +#endif /* CONFIG_MEMCG */ + +/** + * shrink_slab - shrink slab caches + * @gfp_mask: allocation context + * @nid: node whose slab caches to target + * @memcg: memory cgroup whose slab caches to target + * @priority: the reclaim priority + * + * Call the shrink functions to age shrinkable caches. + * + * @nid is passed along to shrinkers with SHRINKER_NUMA_AWARE set, + * unaware shrinkers will receive a node id of 0 instead. + * + * @memcg specifies the memory cgroup to target. Unaware shrinkers + * are called only if it is the root cgroup. + * + * @priority is sc->priority, we take the number of objects and >> by priority + * in order to get the scan target. + * + * Returns the number of reclaimed slab objects. + */ +unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, + int priority) +{ + unsigned long ret, freed = 0; + struct shrinker *shrinker; + + /* + * The root memcg might be allocated even though memcg is disabled + * via "cgroup_disable=memory" boot parameter. This could make + * mem_cgroup_is_root() return false, then just run memcg slab + * shrink, but skip global shrink. This may result in premature + * oom. + */ + if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) + return shrink_slab_memcg(gfp_mask, nid, memcg, priority); + + rcu_read_lock(); + list_for_each_entry_rcu(shrinker, &shrinker_list, list) { + struct shrink_control sc = { + .gfp_mask = gfp_mask, + .nid = nid, + .memcg = memcg, + }; + + if (!shrinker_try_get(shrinker)) + continue; + rcu_read_unlock(); + + ret = do_shrink_slab(&sc, shrinker, priority); + if (ret == SHRINK_EMPTY) + ret = 0; + freed += ret; + + rcu_read_lock(); + shrinker_put(shrinker); + } + rcu_read_unlock(); + cond_resched(); + return freed; +} + +/* + * Add a shrinker callback to be called from the vm. + */ +static int __prealloc_shrinker(struct shrinker *shrinker) +{ + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); + if (shrinker->flags & SHRINKER_NUMA_AWARE) + size *= nr_node_ids; + + shrinker->nr_deferred = kzalloc(size, GFP_KERNEL); + if (!shrinker->nr_deferred) + return -ENOMEM; + + return 0; +} + +#ifdef CONFIG_SHRINKER_DEBUG +int prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...) +{ + va_list ap; + int err; + + va_start(ap, fmt); + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); + va_end(ap); + if (!shrinker->name) + return -ENOMEM; + + err = __prealloc_shrinker(shrinker); + if (err) { + kfree_const(shrinker->name); + shrinker->name = NULL; + } + + return err; +} +#else +int prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...) +{ + return __prealloc_shrinker(shrinker); +} +#endif + +void free_prealloced_shrinker(struct shrinker *shrinker) +{ +#ifdef CONFIG_SHRINKER_DEBUG + kfree_const(shrinker->name); + shrinker->name = NULL; +#endif + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + mutex_lock(&shrinker_mutex); + unregister_memcg_shrinker(shrinker); + mutex_unlock(&shrinker_mutex); + return; + } + + kfree(shrinker->nr_deferred); + shrinker->nr_deferred = NULL; +} + +void register_shrinker_prepared(struct shrinker *shrinker) +{ + mutex_lock(&shrinker_mutex); + refcount_set(&shrinker->refcount, 1); + init_completion(&shrinker->completion_wait); + list_add_tail_rcu(&shrinker->list, &shrinker_list); + shrinker->flags |= SHRINKER_REGISTERED; + shrinker_debugfs_add(shrinker); + mutex_unlock(&shrinker_mutex); +} + +static int __register_shrinker(struct shrinker *shrinker) +{ + int err = __prealloc_shrinker(shrinker); + + if (err) + return err; + register_shrinker_prepared(shrinker); + return 0; +} + +#ifdef CONFIG_SHRINKER_DEBUG +int register_shrinker(struct shrinker *shrinker, const char *fmt, ...) +{ + va_list ap; + int err; + + va_start(ap, fmt); + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); + va_end(ap); + if (!shrinker->name) + return -ENOMEM; + + err = __register_shrinker(shrinker); + if (err) { + kfree_const(shrinker->name); + shrinker->name = NULL; + } + return err; +} +#else +int register_shrinker(struct shrinker *shrinker, const char *fmt, ...) +{ + return __register_shrinker(shrinker); +} +#endif +EXPORT_SYMBOL(register_shrinker); + +/* + * Remove one + */ +void unregister_shrinker(struct shrinker *shrinker) +{ + struct dentry *debugfs_entry; + int debugfs_id; + + if (!(shrinker->flags & SHRINKER_REGISTERED)) + return; + + shrinker_put(shrinker); + wait_for_completion(&shrinker->completion_wait); + + mutex_lock(&shrinker_mutex); + list_del_rcu(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); + debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); + mutex_unlock(&shrinker_mutex); + + shrinker_debugfs_remove(debugfs_entry, debugfs_id); + + kfree(shrinker->nr_deferred); + shrinker->nr_deferred = NULL; +} +EXPORT_SYMBOL(unregister_shrinker); + +struct shrinker *shrinker_alloc_and_init(count_objects_cb count, + scan_objects_cb scan, long batch, + int seeks, unsigned flags, + void *priv_data) +{ + struct shrinker *shrinker; + + shrinker = kzalloc(sizeof(struct shrinker), GFP_KERNEL); + if (!shrinker) + return NULL; + + shrinker->count_objects = count; + shrinker->scan_objects = scan; + shrinker->batch = batch; + shrinker->seeks = seeks; + shrinker->flags = flags; + shrinker->private_data = priv_data; + + return shrinker; +} +EXPORT_SYMBOL(shrinker_alloc_and_init); + +void shrinker_free(struct shrinker *shrinker) +{ + kfree(shrinker); +} +EXPORT_SYMBOL(shrinker_free); + +void unregister_and_free_shrinker(struct shrinker *shrinker) +{ + unregister_shrinker(shrinker); + kfree_rcu(shrinker, rcu); +} +EXPORT_SYMBOL(unregister_and_free_shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index bcdd97caa403..0816c6496483 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,6 @@ #include #include #include -#include #include #include #include @@ -57,7 +56,6 @@ #include #include #include -#include #include #include @@ -189,262 +187,7 @@ struct scan_control { */ int vm_swappiness = 60; -LIST_HEAD(shrinker_list); -DEFINE_MUTEX(shrinker_mutex); - #ifdef CONFIG_MEMCG -static int shrinker_nr_max; - -/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ -static inline int shrinker_map_size(int nr_items) -{ - return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); -} - -static inline int shrinker_defer_size(int nr_items) -{ - return (round_up(nr_items, BITS_PER_LONG) * sizeof(atomic_long_t)); -} - -static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, - int nid) -{ - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_mutex)); -} - -static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg, - int nid) -{ - return rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); -} - -static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int map_size, int defer_size, - int old_map_size, int old_defer_size, - int new_nr_max) -{ - struct shrinker_info *new, *old; - struct mem_cgroup_per_node *pn; - int nid; - int size = map_size + defer_size; - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - old = shrinker_info_protected(memcg, nid); - /* Not yet online memcg */ - if (!old) - return 0; - - /* Already expanded this shrinker_info */ - if (new_nr_max <= old->map_nr_max) - continue; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - new->nr_deferred = (atomic_long_t *)(new + 1); - new->map = (void *)new->nr_deferred + defer_size; - new->map_nr_max = new_nr_max; - - /* map: set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_map_size); - memset((void *)new->map + old_map_size, 0, map_size - old_map_size); - /* nr_deferred: copy old values, clear all new values */ - memcpy(new->nr_deferred, old->nr_deferred, old_defer_size); - memset((void *)new->nr_deferred + old_defer_size, 0, - defer_size - old_defer_size); - - rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); - } - - return 0; -} - -void free_shrinker_info(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct shrinker_info *info; - int nid; - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - info = rcu_dereference_protected(pn->shrinker_info, true); - kvfree(info); - rcu_assign_pointer(pn->shrinker_info, NULL); - } -} - -int alloc_shrinker_info(struct mem_cgroup *memcg) -{ - struct shrinker_info *info; - int nid, size, ret = 0; - int map_size, defer_size = 0; - - mutex_lock(&shrinker_mutex); - map_size = shrinker_map_size(shrinker_nr_max); - defer_size = shrinker_defer_size(shrinker_nr_max); - size = map_size + defer_size; - for_each_node(nid) { - info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); - if (!info) { - free_shrinker_info(memcg); - ret = -ENOMEM; - break; - } - info->nr_deferred = (atomic_long_t *)(info + 1); - info->map = (void *)info->nr_deferred + defer_size; - info->map_nr_max = shrinker_nr_max; - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); - } - mutex_unlock(&shrinker_mutex); - - return ret; -} - -static int expand_shrinker_info(int new_id) -{ - int ret = 0; - int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); - int map_size, defer_size = 0; - int old_map_size, old_defer_size = 0; - struct mem_cgroup *memcg; - - if (!root_mem_cgroup) - goto out; - - lockdep_assert_held(&shrinker_mutex); - - map_size = shrinker_map_size(new_nr_max); - defer_size = shrinker_defer_size(new_nr_max); - old_map_size = shrinker_map_size(shrinker_nr_max); - old_defer_size = shrinker_defer_size(shrinker_nr_max); - - memcg = mem_cgroup_iter(NULL, NULL, NULL); - do { - ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size, - new_nr_max); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto out; - } - } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -out: - if (!ret) - shrinker_nr_max = new_nr_max; - - return ret; -} - -void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct shrinker_info *info; - - rcu_read_lock(); - info = shrinker_info_rcu(memcg, nid); - if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); - } - rcu_read_unlock(); - } -} - -static DEFINE_IDR(shrinker_idr); - -static int prealloc_memcg_shrinker(struct shrinker *shrinker) -{ - int id, ret = -ENOMEM; - - if (mem_cgroup_disabled()) - return -ENOSYS; - - mutex_lock(&shrinker_mutex); - id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); - if (id < 0) - goto unlock; - - if (id >= shrinker_nr_max) { - if (expand_shrinker_info(id)) { - idr_remove(&shrinker_idr, id); - goto unlock; - } - } - shrinker->id = id; - ret = 0; -unlock: - mutex_unlock(&shrinker_mutex); - return ret; -} - -static void unregister_memcg_shrinker(struct shrinker *shrinker) -{ - int id = shrinker->id; - - BUG_ON(id < 0); - - lockdep_assert_held(&shrinker_mutex); - - idr_remove(&shrinker_idr, id); -} - -static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, - struct mem_cgroup *memcg) -{ - struct shrinker_info *info; - long nr_deferred; - - rcu_read_lock(); - info = shrinker_info_rcu(memcg, nid); - nr_deferred = atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); - rcu_read_unlock(); - - return nr_deferred; -} - -static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, - struct mem_cgroup *memcg) -{ - struct shrinker_info *info; - long nr_deferred; - - rcu_read_lock(); - info = shrinker_info_rcu(memcg, nid); - nr_deferred = atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); - rcu_read_unlock(); - - return nr_deferred; -} - -void reparent_shrinker_deferred(struct mem_cgroup *memcg) -{ - int i, nid; - long nr; - struct mem_cgroup *parent; - struct shrinker_info *child_info, *parent_info; - - parent = parent_mem_cgroup(memcg); - if (!parent) - parent = root_mem_cgroup; - - /* Prevent from concurrent shrinker_info expand */ - mutex_lock(&shrinker_mutex); - for_each_node(nid) { - child_info = shrinker_info_protected(memcg, nid); - parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < child_info->map_nr_max; i++) { - nr = atomic_long_read(&child_info->nr_deferred[i]); - atomic_long_add(nr, &parent_info->nr_deferred[i]); - } - } - mutex_unlock(&shrinker_mutex); -} - static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -479,27 +222,6 @@ static bool writeback_throttling_sane(struct scan_control *sc) return false; } #else -static int prealloc_memcg_shrinker(struct shrinker *shrinker) -{ - return -ENOSYS; -} - -static void unregister_memcg_shrinker(struct shrinker *shrinker) -{ -} - -static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, - struct mem_cgroup *memcg) -{ - return 0; -} - -static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, - struct mem_cgroup *memcg) -{ - return 0; -} - static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -568,39 +290,6 @@ static void flush_reclaim_state(struct scan_control *sc) } } -static long xchg_nr_deferred(struct shrinker *shrinker, - struct shrink_control *sc) -{ - int nid = sc->nid; - - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - - if (sc->memcg && - (shrinker->flags & SHRINKER_MEMCG_AWARE)) - return xchg_nr_deferred_memcg(nid, shrinker, - sc->memcg); - - return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); -} - - -static long add_nr_deferred(long nr, struct shrinker *shrinker, - struct shrink_control *sc) -{ - int nid = sc->nid; - - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - - if (sc->memcg && - (shrinker->flags & SHRINKER_MEMCG_AWARE)) - return add_nr_deferred_memcg(nr, nid, shrinker, - sc->memcg); - - return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); -} - static bool can_demote(int nid, struct scan_control *sc) { if (!numa_demotion_enabled) @@ -682,441 +371,6 @@ static unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, return size; } -/* - * Add a shrinker callback to be called from the vm. - */ -static int __prealloc_shrinker(struct shrinker *shrinker) -{ - unsigned int size; - int err; - - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - err = prealloc_memcg_shrinker(shrinker); - if (err != -ENOSYS) - return err; - - shrinker->flags &= ~SHRINKER_MEMCG_AWARE; - } - - size = sizeof(*shrinker->nr_deferred); - if (shrinker->flags & SHRINKER_NUMA_AWARE) - size *= nr_node_ids; - - shrinker->nr_deferred = kzalloc(size, GFP_KERNEL); - if (!shrinker->nr_deferred) - return -ENOMEM; - - return 0; -} - -#ifdef CONFIG_SHRINKER_DEBUG -int prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...) -{ - va_list ap; - int err; - - va_start(ap, fmt); - shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); - va_end(ap); - if (!shrinker->name) - return -ENOMEM; - - err = __prealloc_shrinker(shrinker); - if (err) { - kfree_const(shrinker->name); - shrinker->name = NULL; - } - - return err; -} -#else -int prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...) -{ - return __prealloc_shrinker(shrinker); -} -#endif - -void free_prealloced_shrinker(struct shrinker *shrinker) -{ -#ifdef CONFIG_SHRINKER_DEBUG - kfree_const(shrinker->name); - shrinker->name = NULL; -#endif - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - mutex_lock(&shrinker_mutex); - unregister_memcg_shrinker(shrinker); - mutex_unlock(&shrinker_mutex); - return; - } - - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; -} - -void register_shrinker_prepared(struct shrinker *shrinker) -{ - mutex_lock(&shrinker_mutex); - refcount_set(&shrinker->refcount, 1); - init_completion(&shrinker->completion_wait); - list_add_tail_rcu(&shrinker->list, &shrinker_list); - shrinker->flags |= SHRINKER_REGISTERED; - shrinker_debugfs_add(shrinker); - mutex_unlock(&shrinker_mutex); -} - -static int __register_shrinker(struct shrinker *shrinker) -{ - int err = __prealloc_shrinker(shrinker); - - if (err) - return err; - register_shrinker_prepared(shrinker); - return 0; -} - -#ifdef CONFIG_SHRINKER_DEBUG -int register_shrinker(struct shrinker *shrinker, const char *fmt, ...) -{ - va_list ap; - int err; - - va_start(ap, fmt); - shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); - va_end(ap); - if (!shrinker->name) - return -ENOMEM; - - err = __register_shrinker(shrinker); - if (err) { - kfree_const(shrinker->name); - shrinker->name = NULL; - } - return err; -} -#else -int register_shrinker(struct shrinker *shrinker, const char *fmt, ...) -{ - return __register_shrinker(shrinker); -} -#endif -EXPORT_SYMBOL(register_shrinker); - -/* - * Remove one - */ -void unregister_shrinker(struct shrinker *shrinker) -{ - struct dentry *debugfs_entry; - int debugfs_id; - - if (!(shrinker->flags & SHRINKER_REGISTERED)) - return; - - shrinker_put(shrinker); - wait_for_completion(&shrinker->completion_wait); - - mutex_lock(&shrinker_mutex); - list_del_rcu(&shrinker->list); - shrinker->flags &= ~SHRINKER_REGISTERED; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); - debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); - mutex_unlock(&shrinker_mutex); - - shrinker_debugfs_remove(debugfs_entry, debugfs_id); - - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; -} -EXPORT_SYMBOL(unregister_shrinker); - -struct shrinker *shrinker_alloc_and_init(count_objects_cb count, - scan_objects_cb scan, long batch, - int seeks, unsigned flags, - void *priv_data) -{ - struct shrinker *shrinker; - - shrinker = kzalloc(sizeof(struct shrinker), GFP_KERNEL); - if (!shrinker) - return NULL; - - shrinker->count_objects = count; - shrinker->scan_objects = scan; - shrinker->batch = batch; - shrinker->seeks = seeks; - shrinker->flags = flags; - shrinker->private_data = priv_data; - - return shrinker; -} -EXPORT_SYMBOL(shrinker_alloc_and_init); - -void shrinker_free(struct shrinker *shrinker) -{ - kfree(shrinker); -} -EXPORT_SYMBOL(shrinker_free); - -void unregister_and_free_shrinker(struct shrinker *shrinker) -{ - unregister_shrinker(shrinker); - kfree_rcu(shrinker, rcu); -} -EXPORT_SYMBOL(unregister_and_free_shrinker); - -#define SHRINK_BATCH 128 - -static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, - struct shrinker *shrinker, int priority) -{ - unsigned long freed = 0; - unsigned long long delta; - long total_scan; - long freeable; - long nr; - long new_nr; - long batch_size = shrinker->batch ? shrinker->batch - : SHRINK_BATCH; - long scanned = 0, next_deferred; - - freeable = shrinker->count_objects(shrinker, shrinkctl); - if (freeable == 0 || freeable == SHRINK_EMPTY) - return freeable; - - /* - * copy the current shrinker scan count into a local variable - * and zero it so that other concurrent shrinker invocations - * don't also do this scanning work. - */ - nr = xchg_nr_deferred(shrinker, shrinkctl); - - if (shrinker->seeks) { - delta = freeable >> priority; - delta *= 4; - do_div(delta, shrinker->seeks); - } else { - /* - * These objects don't require any IO to create. Trim - * them aggressively under memory pressure to keep - * them from causing refetches in the IO caches. - */ - delta = freeable / 2; - } - - total_scan = nr >> priority; - total_scan += delta; - total_scan = min(total_scan, (2 * freeable)); - - trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, - freeable, delta, total_scan, priority); - - /* - * Normally, we should not scan less than batch_size objects in one - * pass to avoid too frequent shrinker calls, but if the slab has less - * than batch_size objects in total and we are really tight on memory, - * we will try to reclaim all available objects, otherwise we can end - * up failing allocations although there are plenty of reclaimable - * objects spread over several slabs with usage less than the - * batch_size. - * - * We detect the "tight on memory" situations by looking at the total - * number of objects we want to scan (total_scan). If it is greater - * than the total number of objects on slab (freeable), we must be - * scanning at high prio and therefore should try to reclaim as much as - * possible. - */ - while (total_scan >= batch_size || - total_scan >= freeable) { - unsigned long ret; - unsigned long nr_to_scan = min(batch_size, total_scan); - - shrinkctl->nr_to_scan = nr_to_scan; - shrinkctl->nr_scanned = nr_to_scan; - ret = shrinker->scan_objects(shrinker, shrinkctl); - if (ret == SHRINK_STOP) - break; - freed += ret; - - count_vm_events(SLABS_SCANNED, shrinkctl->nr_scanned); - total_scan -= shrinkctl->nr_scanned; - scanned += shrinkctl->nr_scanned; - - cond_resched(); - } - - /* - * The deferred work is increased by any new work (delta) that wasn't - * done, decreased by old deferred work that was done now. - * - * And it is capped to two times of the freeable items. - */ - next_deferred = max_t(long, (nr + delta - scanned), 0); - next_deferred = min(next_deferred, (2 * freeable)); - - /* - * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. - */ - new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); - - trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); - return freed; -} - -#ifdef CONFIG_MEMCG -static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, - struct mem_cgroup *memcg, int priority) -{ - struct shrinker_info *info; - unsigned long ret, freed = 0; - int i = 0; - - if (!mem_cgroup_online(memcg)) - return 0; - -again: - rcu_read_lock(); - info = shrinker_info_rcu(memcg, nid); - if (unlikely(!info)) - goto unlock; - - for_each_set_bit_from(i, info->map, info->map_nr_max) { - struct shrink_control sc = { - .gfp_mask = gfp_mask, - .nid = nid, - .memcg = memcg, - }; - struct shrinker *shrinker; - - shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { - if (!shrinker) - clear_bit(i, info->map); - continue; - } - - if (!shrinker_try_get(shrinker)) - continue; - rcu_read_unlock(); - - /* Call non-slab shrinkers even though kmem is disabled */ - if (!memcg_kmem_online() && - !(shrinker->flags & SHRINKER_NONSLAB)) - continue; - - ret = do_shrink_slab(&sc, shrinker, priority); - if (ret == SHRINK_EMPTY) { - clear_bit(i, info->map); - /* - * After the shrinker reported that it had no objects to - * free, but before we cleared the corresponding bit in - * the memcg shrinker map, a new object might have been - * added. To make sure, we have the bit set in this - * case, we invoke the shrinker one more time and reset - * the bit if it reports that it is not empty anymore. - * The memory barrier here pairs with the barrier in - * set_shrinker_bit(): - * - * list_lru_add() shrink_slab_memcg() - * list_add_tail() clear_bit() - * - * set_bit() do_shrink_slab() - */ - smp_mb__after_atomic(); - ret = do_shrink_slab(&sc, shrinker, priority); - if (ret == SHRINK_EMPTY) - ret = 0; - else - set_shrinker_bit(memcg, nid, i); - } - freed += ret; - - shrinker_put(shrinker); - - /* - * We have already exited the read-side of rcu critical section - * before calling do_shrink_slab(), the shrinker_info may be - * released in expand_one_shrinker_info(), so restart the - * iteration. - */ - i++; - goto again; - } -unlock: - rcu_read_unlock(); - return freed; -} -#else /* CONFIG_MEMCG */ -static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, - struct mem_cgroup *memcg, int priority) -{ - return 0; -} -#endif /* CONFIG_MEMCG */ - -/** - * shrink_slab - shrink slab caches - * @gfp_mask: allocation context - * @nid: node whose slab caches to target - * @memcg: memory cgroup whose slab caches to target - * @priority: the reclaim priority - * - * Call the shrink functions to age shrinkable caches. - * - * @nid is passed along to shrinkers with SHRINKER_NUMA_AWARE set, - * unaware shrinkers will receive a node id of 0 instead. - * - * @memcg specifies the memory cgroup to target. Unaware shrinkers - * are called only if it is the root cgroup. - * - * @priority is sc->priority, we take the number of objects and >> by priority - * in order to get the scan target. - * - * Returns the number of reclaimed slab objects. - */ -static unsigned long shrink_slab(gfp_t gfp_mask, int nid, - struct mem_cgroup *memcg, - int priority) -{ - unsigned long ret, freed = 0; - struct shrinker *shrinker; - - /* - * The root memcg might be allocated even though memcg is disabled - * via "cgroup_disable=memory" boot parameter. This could make - * mem_cgroup_is_root() return false, then just run memcg slab - * shrink, but skip global shrink. This may result in premature - * oom. - */ - if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) - return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - - rcu_read_lock(); - list_for_each_entry_rcu(shrinker, &shrinker_list, list) { - struct shrink_control sc = { - .gfp_mask = gfp_mask, - .nid = nid, - .memcg = memcg, - }; - - if (!shrinker_try_get(shrinker)) - continue; - rcu_read_unlock(); - - ret = do_shrink_slab(&sc, shrinker, priority); - if (ret == SHRINK_EMPTY) - ret = 0; - freed += ret; - - rcu_read_lock(); - shrinker_put(shrinker); - } - rcu_read_unlock(); - cond_resched(); - return freed; -} - static unsigned long drop_slab_node(int nid) { unsigned long freed = 0;