From patchwork Tue Jun 4 23:48:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13685987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EECE2C25B78 for ; Tue, 4 Jun 2024 23:49:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BA6F6B008C; Tue, 4 Jun 2024 19:49:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 768816B0093; Tue, 4 Jun 2024 19:49:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 609696B0095; Tue, 4 Jun 2024 19:49:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3DF2F6B008C for ; Tue, 4 Jun 2024 19:49:25 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BF75C1603B1 for ; Tue, 4 Jun 2024 23:49:24 +0000 (UTC) X-FDA: 82194850248.01.3E6EFF5 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2110.outbound.protection.outlook.com [40.107.212.110]) by imf24.hostedemail.com (Postfix) with ESMTP id E761D180014 for ; Tue, 4 Jun 2024 23:49:21 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b=Q1A1DPoC; spf=pass (imf24.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.107.212.110 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1717544962; a=rsa-sha256; cv=pass; b=vFBIr/tpmrdZf9+KtW17YiUE1tPLLC+myxO9PeTzr4SkbY52EiosfTrS2FMEW9H8ucLYLc ICu9G/RE2K+MNoP3l44dnEQ/128XFEXFBf229dfbVTkVQBIUnBjNv5jnGbmpG2mypjzS/D jIEOurOiXJF3/TVOwjAstuSCs39qPBA= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b=Q1A1DPoC; spf=pass (imf24.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.107.212.110 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717544962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=7O7/r5POaN5FbgrD4LDpQYGnVCFI76fgP4riHmKhUn8=; b=FIxs2ATsCNE15EvAfGrxdb34eTORNQY9ni8081Top6z1iazklkOqy8JsmhShqvccnsNKwl vd0PI4Ka3U/KHCGjSFORnwo+4W5PQvZJkoEqcX8jDchXMGLx7Yfns6vZp8RfZcTprSPc0M 5UC35xUTQ1ETizZtWnKiJ9NAHK/Fv8o= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F9xOTAdyLKc/07rlaz5jRzBcXUk8n0baDFx2uoOXcyWkjmf+EQ6LTdySt0VxnFfAvCocaytq544aW1DOEcQWSnViXkeoI0W20JXkJQwnqGKvpQZZwsEN4+rQLxc1b30TlFyfXYRItNmfTSiRZAGx2IKVrDJf7kCGpUJwvd5ezYn99aa9Jng09u1irp7jerw2e83uKrhlYMmNFqsThO4asCYiCkawrbb9hvBopZfgCbChfmV99ise8dZQX2D4qpIjQ6oBqzJX1QtdPoObFTwfnTPp4XSatlL5rWvQ1sVU7XWVcl5/9cNIFazY/nAU6DI/lkjxdM7ULrCmrWJW3c1nCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7O7/r5POaN5FbgrD4LDpQYGnVCFI76fgP4riHmKhUn8=; b=bq15ONj4KoEyxYWyp3EP/XsV0AKWb9NbF6Ls7DMgYU8a/RSm/kJS7/uGdXMRP6ICuCZFuWceWWQBf+HDmFEVa+0TpHi9nDiMrmQ4E5x4MF+pLJkPdoVXNNEh0ny+yRBfEiUzHkGcKHyQWakJ6hY1DrQ2d9I2zUcuwqzdjSuGfnkVIvrEeTN2KygnWZeNkqF9OG63L51pn2SexyUoYrMyC6kWYvIs9JMsxvx3cfHr+4XqbGxEdGSf5k7MXa89eJawzUoD6zKF2P5z67Rj/789e4j5SOPRm+xYvHQzSoIpoqNTSg03bze8/pFFj0hEbwZRwCGZzgxWUzwXYHGdcYls9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7O7/r5POaN5FbgrD4LDpQYGnVCFI76fgP4riHmKhUn8=; b=Q1A1DPoCvQTtmGOa8ZGPb0eubtQObmSQkrNigHeg0Sahqzp8W1iEwqzTs7lyiSFM0e10q2KIvRCwplibQ/r3R3GqVlyVktHjReqVVbTE/fS6XT9JH1HWWSzoFdl2N7ZAIp3NJwzpbU36fttCbMSGpvYC4zTjAJum+01Jn3ZyOdg= Received: from BYAPR01MB5463.prod.exchangelabs.com (2603:10b6:a03:11b::20) by CH3PR01MB8575.prod.exchangelabs.com (2603:10b6:610:168::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.30; Tue, 4 Jun 2024 23:49:15 +0000 Received: from BYAPR01MB5463.prod.exchangelabs.com ([fe80::4984:7039:100:6955]) by BYAPR01MB5463.prod.exchangelabs.com ([fe80::4984:7039:100:6955%6]) with mapi id 15.20.7633.021; Tue, 4 Jun 2024 23:49:15 +0000 From: Yang Shi To: peterx@redhat.com, oliver.sang@intel.com, paulmck@kernel.org, david@redhat.com, willy@infradead.org, riel@surriel.com, vivek.kasireddy@intel.com, cl@linux.com, akpm@linux-foundation.org Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Subject: [PATCH 1/2] mm: page_ref: remove folio_try_get_rcu() Date: Tue, 4 Jun 2024 16:48:57 -0700 Message-ID: <20240604234858.948986-1-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 X-ClientProxiedBy: CH0PR13CA0024.namprd13.prod.outlook.com (2603:10b6:610:b1::29) To BYAPR01MB5463.prod.exchangelabs.com (2603:10b6:a03:11b::20) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR01MB5463:EE_|CH3PR01MB8575:EE_ X-MS-Office365-Filtering-Correlation-Id: e1b54cd6-d675-4c62-a66c-08dc84f0f133 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|376005|7416005|52116005|1800799015|366007|38350700005; X-Microsoft-Antispam-Message-Info: mpPjGGP0w8tZTd9/ZDg0tKb4mImmw/xKjaqTT8A7vQq7j4ZK+jq+XIf4VGCxqJ+VK14GBOj5VMsezkCu9xh7inco6uKbiet5zYucrdYNZ7cv37Dl1N21WrtSF5jXcu20BWIM4jvSzjh0inMuCG8RBlTN/1bsh4xCJYX3bSwNBXzxk9lMesT/Fkws33WX2+i2f/UfYW+S/SmyYY8AWBlLie2rNRYNJRjAJylERXtJJqgQDltCOYWI8tjsG2SZzdGv5a1l6PhWmzDyDYJPO5nVcjDcoBeKbMSbT+bbvlRodtvgRXYWMt2Ww2hu4VoqdgG97fWoHFnuWGc4vjGbRe2xrGIQUoWrWoP/JSaYzluWyb2uhPHoNX0TmYVcuasFvybSYNRuDG0jAHxtur+c8tj85inTaJtYyKe0l7eXofq8iXwh/sXyF8UerrVhXhOJLMhR/BcSztBrqVxzMVVX5/+mJ+K5S3qdK8gioIohgvNZnJzXlCvOzW+4/bJXbzXbEFgNku5qHeKAHalqgUflssSl+uUu7orjBoJnOVGRFOCpeao7C/8jQFqAz88744XPiKn/q9GHx9PdAyK4dRq7hmeZ/FBg83188Q/UmdYvrgLHY7GMkr5+yrwX2tWBV9bCPoY5SZ4jWpEZnfLqzeoiQkkeKndWpZznR6f54qsgI9sHlEAc0BgZwKQZJ8OZRp/bXYhtDbOI5yU0AtLVdunHzkJTaPvHiO0d8GJmCqZt3BXoby3K6regHgbGLjmowQaL+ccV/junwi8H00Gkn8mviR/XglyVV9x5FI0KYPB1MaVKw+5YYCBKTVtPTBhzAFG6I7rIsYbE4+qvL3rvoQkfaSgJHoYL7oA6xr8XZ3LetgDmVnJPBAOrW5qvGt5hYD+tb8aPm9Kbczmjxl7wKidMB0tjJN//YjPxTLGaVAoUoqwwj3p0ZXE+zKld8gKRHd/t1k3WaNydy7KItbBCOHJ63Dun4M5u+vB1kEpaYFjggcc3QGUJRXBH7VJbg4a6EzMzgaqXOFYQsRRJipj8MvS9CD5lyCWgHv+kncYk3K9fSYZvCfmGS2H8o5lw9EV5ZNGYEv6m68C2BYHAAEZ4yuO3jd9b6fIPRvTa1waM0WZqaK221OriU9tZ6EljYfkB0k8b/y1fBR8Kv/38oaovYQ+sXKd8ATUqDptJS0miim652EBkTcXzD/QwSECOS7HuxpaVbEQd9wOCMQ/dN4Qnji6rEwnppJWx5ee4XuQkFMzI8+B38maiEPyliMjfcAAbzO6bGpvmwYugAMNrtz1s9yhbG/tRXcovKkivFDUDL75lpfOIC8et2m6ufDAvgAqdqzjCmV20FkM4GB2yOTwPjYHeR65ZEg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR01MB5463.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(7416005)(52116005)(1800799015)(366007)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: r2A3dMv8bQjeL1FrflgoDPHZvkRUvjA1SjMiQq4B/RIC2XizqfD3fdTe9qZnSfriyseZSZqKnhSKo+lfmp23fLNAAq9rO7v8OViT0yfwtZFa+bV4Q6nwUbjekp17m1W9xk0vJx+apR6kjYlMA9J4JJ2LvU8WY32GMsTo6amEkhxf8rObIPjTQrOld7sd3palkXZBvAyI60dXqdaUhNhCJT9AMvCef+HnJus1hHOliqyNgNI9kSux6pUJTNavKeaCHMtykrSBJYY5ZI9gI+nP8TFcxEVXZruVse2tM2WpYgoqfay2cCW8E5k9XoSOjtueysfCS0tqRmMpMFsRPUHJ0ZpwTumpZiTFQo5qQpLNa6txpgJ0AhFrGvJSD18AzomsvR7Q0bKkEyaESjqJprXnYa8mqwNkRzQ1CtQpI1IQK9iY2Xs8f6M2IpDbXD/jCRTXlmDvScOsrw2Jg9xLyPBfxK+eM6D9Y81Ga8kJYKnWVb/0UEZzR8VVLdfGPXhkucr5q+5MxiFL7YEhdgNH81qufb1QvSLhlefFGRdBi/xPxPHNtD1epaXMx/+9Z80KVYS3fEbKqp6xoXsDUKqrPb3w05P116W3WgXmrVgEFaaIELqIcCuaMQKwHXZx3eZ8JcjXVJzyNohBQrqHr71K9G9MTGf+5DIYCLYXxb/9K+jMUkSR0PJkxMZA1heRe8mDSw/GUEs7RbFK6qxqysYadkTxtx3bXH48H+AWwelCTpOVKJFRjiEJE3B+6nfM34ikU9f5Syf4Dj0wlFhQAih65ExYM9H6qAfXoCwHqPEhX6DV/845GES4G9NH7/grgmwI8/xTihB7wmI/VQ4zJQ0uQFq22HoeNw+I885r1skHsxbAJeOkNiWsvJry31fMRf3VtuZ/L/hYIR2IqqM1fHsEnxFS0Wi6xM04PYqZd1GVrWGGbLuZusdt9bBfr6eWs7wbflS2BHyG9juDSz9sWTs0tU9rc8OAVNyySiFmBuWVsCq2m5VGLDsHLzdPW12ifQlBwLGvP/jKTJMzu3nN83C6d4H8gR5CR5SfEFpJa0OXKFKy8RQUzpZ7Tsvi+kN6x+OTndbIWpjwBwt/f0PraloYNg+6w6+ar0bYVKa2VFT3OIFgBMXOVONgwovy+GjFA45QuUd8ToNXPh3ll1/xE7+aRLTnEGfxmBRiufU1HYPYG8JOo8qzHIpbhtckrZXiXJCBiIv0svioFyCtQvl1B3sp02vhsTulPEBaYAcJi9fKFuA2vD5pws8RTrtJOu43BAXRPq0WlWvsvz8AfTIsf0DfGJmFJErmysVstO/YhiMPb9YQ9cCUfo/bgnYMtPydZY+/PDlnpMYwmUk14Lnk/GTnxv13apIsqCJEacecYU/tcNjezRZAHuE6jQ5v1m6WgqgwOq0WkLVxVdNV0K3D9H/Hxb5SW1aHJOJgEcEg03jHEROBcX4D5pDmoUuuyTCdtSnHB7+hNcmWHDI8gqJ4CvHJ3T+mJJHi5qInHncwJBCy0YvYskazZw9vaneuha6F2GbrI8Is2KkAZrSKT9HCRdDhHh9n2iyi8KscW5hQZsa8ktcHs9dclwUEo8rjAvIgM+tvJ+8YSl7oObGPk/xmIUmaGpOcHFDwJSZmLhmwNmVv2v+X7Ik= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: e1b54cd6-d675-4c62-a66c-08dc84f0f133 X-MS-Exchange-CrossTenant-AuthSource: BYAPR01MB5463.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 23:49:15.5933 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6DxAejY2MnVmMjQ48imu483c/X2f9x1CdUzZDFIKwAw4Rs/4XQGSsCULOBbZ6dKOoxTGrNItR/frER4xrkKzag0yBOdp4N8sE8Vwq/XOxpI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR01MB8575 X-Rspamd-Queue-Id: E761D180014 X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: qgxokp4cwt31jkfiby9z3cdwxytt5fky X-HE-Tag: 1717544961-793941 X-HE-Meta: U2FsdGVkX18g2U4T/Xct8M8JOnlZ3DdaoWsj2eBuf9S4s0SMjo8st8IXU5Wd8guMFEvNHX4TJoGgNUiviR1hSaaQ/EBO0zmQACASzPfLT4xAnT0b2DHDML1odDmB+qZNf1Xqmk0ymdxswe5HnKgCzA0/F4X4r+KwKw/lYcIa7Pgv8E2rtVKpTzpE17Cl9KoJOVfszECLO7TDR2OBRfXPfIt3a1bN5CKlFds+5wWkbyXJ5aA2tlKcjFR2C2/aiv2ZkdWy1dgIjdglMooOEiSvv/byFPOgXy1WEAZsxFhPsEAyTkGUcN21o80dSP67OWWVcX/kGmsJhagFcw2WXwEiYNVuIzzy1cw1P2WfC9ZiiaTuVtNGM3ida2reckl+M4VPqhFK/+l0f9l/1B2NyZODND9+ZzOVoWmt1iaA8QPh/NVeKiY6pbWiD0ColaGKIwT9ZywmPtYv53v0AKoFgNUVrT4YRZE6dVh0tPaczFo/rypjWDVgxt435RKRMWbYLWbhjQJsP+K+YbM3a8ltbKvSHn1emwlodxNcSd5Rf1zWrPvKeKRgtSjDI6RONOuRBhqBRpA7yv7iOfi2ddFMgX2jEG3fGiELXPGKo0+eroBwQdh6uNUQWo3WGI6e1XQcjz2weIUSA651/zdJTRRz0VL6MQ5e6dfXmdkbZyOud7tdXstR1Bqm2rW/jBjSKAlYOe6InkTENT0+dZ2MyzfVzzQjpM+cRJAGUoAmbbxlVPnGPPCWLsH7y3sF58tM7O21YczYti4YV2WWNlKn+7r4BwQpOv1UQjfdpH1WIz+zEWwzptYDaXi2QDD2myoMlSw5QLMqp1fpUEzgTs+reWz15tqs3lsk/9thmdFJIM540KiviLM2eA9WO2Itn20svz80B4PuV/OmK4tfSHZkB2Zu7RjPorawzu7b90Cd5lKVa4qDB+yQTxJvgNeQ1+YTeDNBg/g3kYei4OVd+e7Zbp8sSJ3 Z6v4UcgT 4UzlrNMp4Lvzuk0zjtXNUb15cW8r5bGVtQAQ1ndbBozBjzz/JvcOlvI0/RHAQdkxtDuZhDbnBi6tNh0flKM1LEYsdor8aL9dVIfWaEXz+tj4l8bUyTN0BXJOau0l8e90WKJE89ptyW+6VbfXhLwEmSJEWKc4//hjbipncsctt34bxcNUgR+QgpqCRmMuTxsIOv1PJJDdodBOnFKFY2CZC4kaHQOARTPN3aIz2lxqJO83wA+b0d+KaPuQMKrtBUzM3PUMKrDxGv+7k+Jk5GEOwYijBINRsYROAjdE7CrhHP5YaY7uSF/xRCdJMGCxKSN5azjZPtQU1ZlDp2e7Y6N+huv02t3kbHq0q1H2Xmfos4cvJXxYUhYEqjXXSFDnrDaNokKp0HVmMktz/j3h/v1uzhP5WjM4WRjqN0AsrSjHJiLxUBB67gD3Pzz+sMI+s49XpKJ3hyC6lqkAneTaXGnskrDh1FYdMXEHncbY5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The below bug was reported on a non-SMP kernel: [ 275.267158][ T4335] ------------[ cut here ]------------ [ 275.267949][ T4335] kernel BUG at include/linux/page_ref.h:275! [ 275.268526][ T4335] invalid opcode: 0000 [#1] KASAN PTI [ 275.269001][ T4335] CPU: 0 PID: 4335 Comm: trinity-c3 Not tainted 6.7.0-rc4-00061-gefa7df3e3bb5 #1 [ 275.269787][ T4335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 275.270679][ T4335] RIP: 0010:try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.272813][ T4335] RSP: 0018:ffffc90005dcf650 EFLAGS: 00010202 [ 275.273346][ T4335] RAX: 0000000000000246 RBX: ffffea00066e0000 RCX: 0000000000000000 [ 275.274032][ T4335] RDX: fffff94000cdc007 RSI: 0000000000000004 RDI: ffffea00066e0034 [ 275.274719][ T4335] RBP: ffffea00066e0000 R08: 0000000000000000 R09: fffff94000cdc006 [ 275.275404][ T4335] R10: ffffea00066e0037 R11: 0000000000000000 R12: 0000000000000136 [ 275.276106][ T4335] R13: ffffea00066e0034 R14: dffffc0000000000 R15: ffffea00066e0008 [ 275.276790][ T4335] FS: 00007fa2f9b61740(0000) GS:ffffffff89d0d000(0000) knlGS:0000000000000000 [ 275.277570][ T4335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 275.278143][ T4335] CR2: 00007fa2f6c00000 CR3: 0000000134b04000 CR4: 00000000000406f0 [ 275.278833][ T4335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 275.279521][ T4335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 275.280201][ T4335] Call Trace: [ 275.280499][ T4335] [ 275.280751][ T4335] ? die (arch/x86/kernel/dumpstack.c:421 arch/x86/kernel/dumpstack.c:434 arch/x86/kernel/dumpstack.c:447) [ 275.281087][ T4335] ? do_trap (arch/x86/kernel/traps.c:112 arch/x86/kernel/traps.c:153) [ 275.281463][ T4335] ? try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.281884][ T4335] ? try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.282300][ T4335] ? do_error_trap (arch/x86/kernel/traps.c:174) [ 275.282711][ T4335] ? try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.283129][ T4335] ? handle_invalid_op (arch/x86/kernel/traps.c:212) [ 275.283561][ T4335] ? try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.283990][ T4335] ? exc_invalid_op (arch/x86/kernel/traps.c:264) [ 275.284415][ T4335] ? asm_exc_invalid_op (arch/x86/include/asm/idtentry.h:568) [ 275.284859][ T4335] ? try_get_folio (include/linux/page_ref.h:275 (discriminator 3) mm/gup.c:79 (discriminator 3)) [ 275.285278][ T4335] try_grab_folio (mm/gup.c:148) [ 275.285684][ T4335] __get_user_pages (mm/gup.c:1297 (discriminator 1)) [ 275.286111][ T4335] ? __pfx___get_user_pages (mm/gup.c:1188) [ 275.286579][ T4335] ? __pfx_validate_chain (kernel/locking/lockdep.c:3825) [ 275.287034][ T4335] ? mark_lock (kernel/locking/lockdep.c:4656 (discriminator 1)) [ 275.287416][ T4335] __gup_longterm_locked (mm/gup.c:1509 mm/gup.c:2209) [ 275.288192][ T4335] ? __pfx___gup_longterm_locked (mm/gup.c:2204) [ 275.288697][ T4335] ? __pfx_lock_acquire (kernel/locking/lockdep.c:5722) [ 275.289135][ T4335] ? __pfx___might_resched (kernel/sched/core.c:10106) [ 275.289595][ T4335] pin_user_pages_remote (mm/gup.c:3350) [ 275.290041][ T4335] ? __pfx_pin_user_pages_remote (mm/gup.c:3350) [ 275.290545][ T4335] ? find_held_lock (kernel/locking/lockdep.c:5244 (discriminator 1)) [ 275.290961][ T4335] ? mm_access (kernel/fork.c:1573) [ 275.291353][ T4335] process_vm_rw_single_vec+0x142/0x360 [ 275.291900][ T4335] ? __pfx_process_vm_rw_single_vec+0x10/0x10 [ 275.292471][ T4335] ? mm_access (kernel/fork.c:1573) [ 275.292859][ T4335] process_vm_rw_core+0x272/0x4e0 [ 275.293384][ T4335] ? hlock_class (arch/x86/include/asm/bitops.h:227 arch/x86/include/asm/bitops.h:239 include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/locking/lockdep.c:228) [ 275.293780][ T4335] ? __pfx_process_vm_rw_core+0x10/0x10 [ 275.294350][ T4335] process_vm_rw (mm/process_vm_access.c:284) [ 275.294748][ T4335] ? __pfx_process_vm_rw (mm/process_vm_access.c:259) [ 275.295197][ T4335] ? __task_pid_nr_ns (include/linux/rcupdate.h:306 (discriminator 1) include/linux/rcupdate.h:780 (discriminator 1) kernel/pid.c:504 (discriminator 1)) [ 275.295634][ T4335] __x64_sys_process_vm_readv (mm/process_vm_access.c:291) [ 275.296139][ T4335] ? syscall_enter_from_user_mode (kernel/entry/common.c:94 kernel/entry/common.c:112) [ 275.296642][ T4335] do_syscall_64 (arch/x86/entry/common.c:51 (discriminator 1) arch/x86/entry/common.c:82 (discriminator 1)) [ 275.297032][ T4335] ? __task_pid_nr_ns (include/linux/rcupdate.h:306 (discriminator 1) include/linux/rcupdate.h:780 (discriminator 1) kernel/pid.c:504 (discriminator 1)) [ 275.297470][ T4335] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4300 kernel/locking/lockdep.c:4359) [ 275.297988][ T4335] ? do_syscall_64 (arch/x86/include/asm/cpufeature.h:171 arch/x86/entry/common.c:97) [ 275.298389][ T4335] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4300 kernel/locking/lockdep.c:4359) [ 275.298906][ T4335] ? do_syscall_64 (arch/x86/include/asm/cpufeature.h:171 arch/x86/entry/common.c:97) [ 275.299304][ T4335] ? do_syscall_64 (arch/x86/include/asm/cpufeature.h:171 arch/x86/entry/common.c:97) [ 275.299703][ T4335] ? do_syscall_64 (arch/x86/include/asm/cpufeature.h:171 arch/x86/entry/common.c:97) [ 275.300115][ T4335] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:129) This BUG is the VM_BUG_ON(!in_atomic() && !irqs_disabled()) assertion in folio_ref_try_add_rcu() for non-SMP kernel. The process_vm_readv() calls GUP to pin the THP. An optimization for pinning THP instroduced by commit 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") calls try_grab_folio() to pin the THP, but try_grab_folio() is supposed to be called in atomic context for non-SMP kernel, for example, irq disabled or preemption disabled, due to the optimization introduced by commit e286781d5f2e ("mm: speculative page references"). The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") is not actually the root cause although it was bisected to. It just makes the problem exposed more likely. The follow up discussion suggested the optimization for non-SMP kernel may be out-dated and not worth it anymore [1]. So removing the optimization to silence the BUG. However calling try_grab_folio() in GUP slow path actually is unnecessary, so the following patch will clean this up. [1] https://lore.kernel.org/linux-mm/821cf1d6-92b9-4ac4-bacc-d8f2364ac14f@paulmck-laptop/ Fixes: 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") Reported-by: kernel test robot Cc: linux-stable v6.6+ Signed-off-by: Yang Shi Acked-by: Peter Xu Acked-by: David Hildenbrand --- include/linux/page_ref.h | 49 ++-------------------------------------- mm/filemap.c | 10 ++++---- mm/gup.c | 2 +- 3 files changed, 8 insertions(+), 53 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 1acf5bac7f50..490d0ad6e56d 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -258,54 +258,9 @@ static inline bool folio_try_get(struct folio *folio) return folio_ref_add_unless(folio, 1, 0); } -static inline bool folio_ref_try_add_rcu(struct folio *folio, int count) -{ -#ifdef CONFIG_TINY_RCU - /* - * The caller guarantees the folio will not be freed from interrupt - * context, so (on !SMP) we only need preemption to be disabled - * and TINY_RCU does that for us. - */ -# ifdef CONFIG_PREEMPT_COUNT - VM_BUG_ON(!in_atomic() && !irqs_disabled()); -# endif - VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - folio_ref_add(folio, count); -#else - if (unlikely(!folio_ref_add_unless(folio, count, 0))) { - /* Either the folio has been freed, or will be freed. */ - return false; - } -#endif - return true; -} - -/** - * folio_try_get_rcu - Attempt to increase the refcount on a folio. - * @folio: The folio. - * - * This is a version of folio_try_get() optimised for non-SMP kernels. - * If you are still holding the rcu_read_lock() after looking up the - * page and know that the page cannot have its refcount decreased to - * zero in interrupt context, you can use this instead of folio_try_get(). - * - * Example users include get_user_pages_fast() (as pages are not unmapped - * from interrupt context) and the page cache lookups (as pages are not - * truncated from interrupt context). We also know that pages are not - * frozen in interrupt context for the purposes of splitting or migration. - * - * You can also use this function if you're holding a lock that prevents - * pages being frozen & removed; eg the i_pages lock for the page cache - * or the mmap_lock or page table lock for page tables. In this case, - * it will always succeed, and you could have used a plain folio_get(), - * but it's sometimes more convenient to have a common function called - * from both locked and RCU-protected contexts. - * - * Return: True if the reference count was successfully incremented. - */ -static inline bool folio_try_get_rcu(struct folio *folio) +static inline bool folio_ref_try_add(struct folio *folio, int count) { - return folio_ref_try_add_rcu(folio, 1); + return folio_ref_add_unless(folio, count, 0); } static inline int page_ref_freeze(struct page *page, int count) diff --git a/mm/filemap.c b/mm/filemap.c index 9fe5c02ae92e..0fb5f3097094 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1847,7 +1847,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index) if (!folio || xa_is_value(folio)) goto out; - if (!folio_try_get_rcu(folio)) + if (!folio_try_get(folio)) goto repeat; if (unlikely(folio != xas_reload(&xas))) { @@ -2001,7 +2001,7 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max, if (!folio || xa_is_value(folio)) return folio; - if (!folio_try_get_rcu(folio)) + if (!folio_try_get(folio)) goto reset; if (unlikely(folio != xas_reload(xas))) { @@ -2181,7 +2181,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, if (xa_is_value(folio)) goto update_start; - if (!folio_try_get_rcu(folio)) + if (!folio_try_get(folio)) goto retry; if (unlikely(folio != xas_reload(&xas))) @@ -2313,7 +2313,7 @@ static void filemap_get_read_batch(struct address_space *mapping, break; if (xa_is_sibling(folio)) break; - if (!folio_try_get_rcu(folio)) + if (!folio_try_get(folio)) goto retry; if (unlikely(folio != xas_reload(&xas))) @@ -3473,7 +3473,7 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, continue; if (folio_test_locked(folio)) continue; - if (!folio_try_get_rcu(folio)) + if (!folio_try_get(folio)) continue; /* Has the page moved or been split? */ if (unlikely(folio != xas_reload(xas))) diff --git a/mm/gup.c b/mm/gup.c index e17466fd62bb..17f89e8d31f1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -78,7 +78,7 @@ static inline struct folio *try_get_folio(struct page *page, int refs) folio = page_folio(page); if (WARN_ON_ONCE(folio_ref_count(folio) < 0)) return NULL; - if (unlikely(!folio_ref_try_add_rcu(folio, refs))) + if (unlikely(!folio_ref_try_add(folio, refs))) return NULL; /* From patchwork Tue Jun 4 23:48:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13685988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AEDEC25B78 for ; Tue, 4 Jun 2024 23:49:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DBFD6B0095; Tue, 4 Jun 2024 19:49:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 08BDE6B0096; Tue, 4 Jun 2024 19:49:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E20466B0098; Tue, 4 Jun 2024 19:49:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BF7D36B0095 for ; Tue, 4 Jun 2024 19:49:38 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6DB8540534 for ; Tue, 4 Jun 2024 23:49:38 +0000 (UTC) X-FDA: 82194850836.22.C5B965E Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2125.outbound.protection.outlook.com [40.107.93.125]) by imf07.hostedemail.com (Postfix) with ESMTP id A11864000D for ; Tue, 4 Jun 2024 23:49:35 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b=VSd5X7kg; spf=pass (imf07.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.107.93.125 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717544975; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kgFAk0t8buK12Swo9M8E2WKPM2udA1VmV+n2vpvLnjw=; b=sBWeg9JWQLSLuoTgyt2WYILHxq/DNIZ1FJEb8QhseI6HTohiTHfWLwiaqwM2SSFvgrgAJ6 S0zeuSzSNqIfjeaF78GWVSM+vjRPKgVQHrzwsG+csSq1/dXwQ/oLxKQDB+LGuOTHi3mfuh Hk/hzqBTMKbPRYiSzRXHRdd4BCAcqeM= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1717544975; a=rsa-sha256; cv=pass; b=L8oK1EsSVhoKiskiRvBdUgasm7DVQodCH5aM92OW6mJkYXLRr8rxpnW2T9xlWKQ/rSTQJx Q+J9lyiJNICybe3auzlbqv6uq+IS4sK6Zyuy1VUnoaxkgb4kjEbMPTTxBsDjws5E8xOPrv Uvy47L3XBre8NWZ1EDEOoIKJ2pDJ7iY= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b=VSd5X7kg; spf=pass (imf07.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.107.93.125 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mDuaOUlpxkIVNXPHMkDdHNupXrkr8JGB93xeFe7mar7Qu0dN8z2mKxzsGEacLLqILJmBZkOfWFNevFG+oyrp/Sk1km/v8jOpGhz3cSQ3wOKVGsg/31/OpO2pvkybXhgcciNq4rdDHzI3aHkaplwdJjQonRdgWiXlwYc1sZ5i/RuDSSWkcouTgPH2LHsA02fabbNSbinLNYTq2/r4/kh+54Cxluru3bBjkbX/lPz3smdIND71NXC7AidssmlyrQWp02xdZQwJBJDbxUU8f4/Yu/ajZXkvbLmgflhFcZ4ArEFlIxZyaDeZK8HFT0C20ezgLYnE9QnFExgncSmmdL16XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kgFAk0t8buK12Swo9M8E2WKPM2udA1VmV+n2vpvLnjw=; b=URAzFUId16kziYkYuyqawAuDePeAWdKhXDVcAPHysNluPLjv6BHlny8IvmsAZDky4oGWV5et/nZ0H16efMWrlUMH/S5goDvExN6hnDGCtWwX1xQ4X6hCLVFYQbDqwWkaveq1Rd7MVqlC6tNskDgaLtpA51Sh9krvYWW+EeK0Ql92cm6LcQK5ZySlnFFExtWG/WCYmXjoiN08CO0EZtOvU10x72qgAWQ2MgAJGxoApDcRrfcTJMGqz5EQQGM2Nml7sjXnsMResikgmJd6ZE9s6Mal0FrNgZpse99nZDckhK7WZXsGWdkRElgO4y7jvWbctR4fvVMNj8liM/avwWL0Eg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kgFAk0t8buK12Swo9M8E2WKPM2udA1VmV+n2vpvLnjw=; b=VSd5X7kgcMCUg5RXMhznOZ+YBbf09YapDJw1IJ2j+B3ZaZKxfr7/+D0IqfOrSFEJE7Um7KaIsjZS5vkNOFzbh/EwYLJDDhSarLt5v5HR2W+wN533LB9HQBSHPwZkcP8bLy5Gh4veA4YBzspQ8WTADaTD2oCBVnhG4+diHk8lhiw= Received: from BYAPR01MB5463.prod.exchangelabs.com (2603:10b6:a03:11b::20) by CH3PR01MB8575.prod.exchangelabs.com (2603:10b6:610:168::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.30; Tue, 4 Jun 2024 23:49:17 +0000 Received: from BYAPR01MB5463.prod.exchangelabs.com ([fe80::4984:7039:100:6955]) by BYAPR01MB5463.prod.exchangelabs.com ([fe80::4984:7039:100:6955%6]) with mapi id 15.20.7633.021; Tue, 4 Jun 2024 23:49:17 +0000 From: Yang Shi To: peterx@redhat.com, oliver.sang@intel.com, paulmck@kernel.org, david@redhat.com, willy@infradead.org, riel@surriel.com, vivek.kasireddy@intel.com, cl@linux.com, akpm@linux-foundation.org Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Subject: [PATCH 2/2] mm: gup: do not call try_grab_folio() in slow path Date: Tue, 4 Jun 2024 16:48:58 -0700 Message-ID: <20240604234858.948986-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240604234858.948986-1-yang@os.amperecomputing.com> References: <20240604234858.948986-1-yang@os.amperecomputing.com> X-ClientProxiedBy: CH0PR13CA0024.namprd13.prod.outlook.com (2603:10b6:610:b1::29) To BYAPR01MB5463.prod.exchangelabs.com (2603:10b6:a03:11b::20) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR01MB5463:EE_|CH3PR01MB8575:EE_ X-MS-Office365-Filtering-Correlation-Id: d15a5653-7896-4278-c0af-08dc84f0f260 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|376005|7416005|52116005|1800799015|366007|38350700005; X-Microsoft-Antispam-Message-Info: D6ta4Xbon/sK6Ku+F2J75YdSUHgkfRu5FxiCDoWizJFqeAZLek69oVUBN9IKqF9qpTD4es9d1MQDXKeQ87GIijvJt/lp/wjjdDskfsV7CjZC5wU1GXQ9yAWlQE5gUvXJRmjjQVW1BhUQiRjs+L//nwNaA5O89g0ewRk380zwEfZe89ieM/r1z0GwUjTXM7YjymuRkZdQVh0OXNaKZkNQRqh9BTLn1mkMTdnck4K3SmhaKIC20TeFCnwOWCk+m16GhIeyqMeo/XDKPQkM1tgZvrkMPH2GFilsX04ScL5vPIs3AcMAhGv6Va30JVhoIVUg1DmJjksRFfMdlJ9pVPu12Hd0/1cq5RsHqRJJEmcT4qnUiKF/4paC0KW14dGZxGhng1xjW083XJXzzwVIJXz61+Jg/jsFAj9oJfOJQq+YtHaHSFvKTWvoIiHfMcj/qUT27ye/as+v3j+Mr2yCNqCceB48AT5qBdMWPNyDA2frfYxWRA8hCLtjNsYdi5rZp975GAQyr+O3j+/iULQYdUEKEWlTJUhw7Hsm5oLX1Wp/4CXczdMKmO15LHYRHXSfIx3MwT5oYvdk9pgQZzU8Qo2OWmyUdfotNtytSlPHewqOF0eUeXxDzWmKhziPBr0h4B8qsbOKTFxxyOikkHRtoPgIHD03iHEYKxkJZ6rAby/BnB8pY67sxptpN9etYepbJ6tMufb4GQqERH8Kk+qstUOKrgGEBtGkY368PNnAJfyRypLD6mF7ZGyS2xK+F2koxUSPDiUVHdL5TDJeKaG9tvgFabmN41ifLlWlbT2rQodd7vduZ84fFaaZnGCgqXp4yX9usZOkUfve54Nsc3hCMEKbYaf6hGNOHAliVEQwOIT0MZUirE1bAR/FUXzdl10gzbrL4zTltUYPCCh93ezOwtewLaJaCsLb1jwll+8h73fa3Gytk121D76DNCTs3SQAeB6Io48b/TRrfacWleGVsQMJr3HcAqzYmGa/Bcfw0JnlDcLvsi23Oozvu9Dxwq1Ovrt6vxcYOjuc7cmHZwdKUY4gu8uWU3WiX13xc3c/iRn+uhWWnCbPUhGbqjVTpwGVnNbWC8NLaR7oDWy+/ng6gChcBEcD1SB62vERiemfIejCuZiiUItEasSUjoPLfCOKdDjX/xepLjvME3xeQ74fV5m3PeRI3mNcXhCThA7AHz+uun0bRXSVuCGrjdbuRhPq3ZrdVX53ql3r2UgFYiFWMA2ZltuZU9PjO9o1w3q0Eav38li4mmZyz8/zZADD6M8S+QATUah/S1oIhy+9FNOHMi+NiSZktEEZyzu/oIyKl8qnmLXFBwteYFeJmf+YvVOEp01vpXqKWIF87SbJMZo+F234mQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR01MB5463.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(7416005)(52116005)(1800799015)(366007)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IqNCUt5jU3DZILqCqh6xjvtoiRDu+aTjsllLfdlZFU4Y1DZ9ReDU7TgArp+pjKOg1uWFcuHgajZc3GwXP7nNWWdCLKjt+TI9za03uDTxl7W8w17agk4ZYMIezSw6KaYRKMnbf2PSjBUWG533KYmXCA1ET7nnSBs/pKXYDjgBXjumZfm1v2bwHJdafdT8Mm/yfUqzb5QFzAJC+S19wQUYZO7JKoFwQz0nL0mT0u/ZINSRkeae/rn33+ubhu4P2WCVgT2jhaQgv0/8Eztbi6ZOdlWww9oJQVsdPW2zZZFs1VJxMf6VUAsn6fRYos0ZtejdybyEltUGIcTRYd0EP2Z7KXtMGuzAa/Z47UOkLnTCpsJb9NmZRO08RLVxOLkGFKgT1FZVU/OBkIrxFBREPGGYqV2NokNRdgspYiWXApwFftRFXka40l9Tl1NanwvsEPDArD70g8e03cm/EVJTxcZTYA15Ep0wGx+b0HMUsqT+Vw2SM2LqD+psOjWviiFrLa4JexzM04pX4D20ylux5OlUpbECE+g4FZpq7UlntA+r4dy6j7gVncSFzWV3AJl/ykiX+w7xWOaNjdqJt1scRpdedfQTLGkJ9gJ5X4rzVTfKhFSTse/fnVn4U9QjOaGkq65cKClJ4DMgI3IXjEAVL+T8TCxdR5k9qT4tpvmCtwgaPCYu4Z4Oh5Xm6r6HbMjZTfjrYEmtY8N6mHzN6CfkMbmUYrRQrZ/iMEgK8SsrVTB9KAtRy/KcMR20k0t8KHrp4KcfWiMOUWOgpjw/zGoiGnLzPfsF2wKNU2c5Jo+5dusyEtiN2yoRpw+vspruMHbStprm8Y1TBwtuDqHqAHU7lSGCVKouhFlveNOac4fJlrBLpwi7MofDYAuLSMwsVN+AbG0AErqOANwmeS66TZhMgTUH3d4kEV2e4SJhBvVk5YvKb77u1LT0dyB2HMLKDLbE7IwjlAQe6LcHUjTQZxEEoZlFHhki6hneETvmEEtxTDb0FCACz8PATSS9v/qfLkab5RHN+yBLy8xyPwwpZPTsCdys5WO7VbIJkwAO9l5v3c7A8rQ3usb7yrifDbg/b/5YRORODNCNON5MINF6fLxO3LeGlqHRzMK6QaZ/6GYAXl10C218R/6uOgYQOXScdc3QotcWEM5cBeQFLtZthq6s2fHGOEEeIuwYft2O5hTO3lHUpvXLUAPTtKJWFO9XvrQpKadcp8g9UZ07iGXtpcMrc4I/3JLB6ghi2WFXFtAULZDsVbrF48AEnzOrZ87z0+5shZWuOSdJrLoQm+1UJRJ011weRDCycc5WO9w0BjiI+3rhu2zHJhTJr928n/Ua9NIPWzlwY5n2KjmNxUJJQpYWGx1hWiBURL2yqR75DWwSXl8FMQmmHoxeV/POLW5slhvpIEXRr0LFYdo0JB5qLGs864TEZP5d8hq9WczIxc8sgtMZJSmTcgAVeeMa1sz/bUmII7VOqqFHjVaGF+kYVOkdkqmq+UKtRPUmkMtlV8wGi5DHjYBAOZILuGyNwwXeVDBWwbGJkpp/B/AilAf5i7nJKMpyGmEmXjM2gQe82ENeydGqJpUOzGBjCRWKWtjbAXZXGk5eD2syTy5we6dSGlhITNzIJNVZXRcNQQcLbOcJvjTTgl0= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: d15a5653-7896-4278-c0af-08dc84f0f260 X-MS-Exchange-CrossTenant-AuthSource: BYAPR01MB5463.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 23:49:17.4401 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SY8jnAeEbt/CO1nATvG5ycAHPbQOPq98IWwVcx2bonorw0GFvK9qMjw0Q8rFgknRami1zDSYkU0cqfZQRyNJ16Hem112lJGdd1+2Ga/UBPE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR01MB8575 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: A11864000D X-Rspam-User: X-Stat-Signature: 8rbjbsuz934pjnj3zwuksuakqi4pt6ir X-HE-Tag: 1717544975-308689 X-HE-Meta: U2FsdGVkX1+jo8CX7xldF2LYPzi5aH6lTtO0Uc2vFwM64xq7U/hXKkBBhWVONTRdEwpO8PpRrUEt5Lx7sds3dfuggu6MMOqJ+gvaedaVHviksy6Xpxcna73qTzEb3k8lmLMfFS2GdhTi/vMC9k1lXs30ZEUYzbm71svb3CAbXDjwOeLJmkESRCBzxXVJxusSXEkuhwvDAMAqokgL+RVtVtMbDc3UBPPMB2mzH6xI5bV5qy7PL63CXxgurVV8V+myAlp0xpIVS6+bPUhL6obUfHyE3AUlgdmPMdPYLzrRyZkyPov8y1XIEvSiimR57csVsR6wXZCG8LPnmxquyPSS4CLcOzKdI2kyoBGASdbZ2xamBdNzHCLI5nAYy5UePjjRNP6M5gRuY7//dlFESQ+Aq2mox+JDjaB1HlAo5yICu4GlpFNCT8n/xFduYt/Wux93dl2yM1+IWPgJKCf59x+1W1TfkI4PKn2297zYGySuPOtgRtKJ78u06vROweFeLG02gnOL2eJhERBhRu5eXkbkblr00XF+GL9iClp6waDDiQa1OW/WO4/F3tkMe1x/kFwa62BaA0xla9atVQc4TR+3FZ7TSeVKqdJp6+fnpnT4r2OjgPBtnIdDkge9ArzqO1SK1nxLwnxtAPPdlUDfrOaQMndnJPjT6JYsC4OS9nR10LOglgQYoLUG4dt8srxW1BDcnmOqX/WR+OlJ5qXWPStWFdFWoeDIkCW6flEFC8xBfvRhDRnYVnQN8/dH0czslFtZr4AS3Hb3ct8ow30jaSVEdeYxeYMOhxsh+1ONiDy2DUxGigtX9LrsJuW3nWe0bb6ms2xCsbNwTNn/nRMN7WztIxkEvZGDPFuFSFFtDY2r5JPysHvBIVA29ThPX9J7q8hjdsCRupMOgzkpXJWwn6phfpCiI7gpMRXpzmjmP+2gKYU0H1EtoXF0pUTBtMVclfVAt9m7Mic6oTt3Ki2hHTk LcgnUihi D5X9GeJ53mmJhXQ204T2Sk3PGK1rwLtqd8gVKxGnsMXConx2QXZHRuxcxqQiOPPh4BJsaABySp3jMRtIiU4ZPLJRwkGjAizGEpc4Gcg3BSR6tF8QXgb07MjFcw4mt3Xf6pMmNgMvGanhRtfryFdKmACI9mjc8/2X5gkVzzHgS4NtsRlaxrr2d14KAfx7/FjEsH99opIxgNKHEU4tLwqQzlNfGXxWn31p9I6+fxbAW6QCQVGf0lAb0PONTe+VQEfvOQIwuF8QIXWyYGVixvdOK3NvLfDWDaWQpZx115ZuMQUug9UoPa0Tho0pl6czvX4IIBRaOqqJetPI+oVAneOeDmhkgi/xofhk+ecxcFyaYnoDiGkDuxM0krC4oAD/ktZaKpt3qeJs9vYPAl4o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The try_grab_folio() is supposed to be used in fast path and it elevates folio refcount by using add ref unless zero. We are guaranteed to have at least one stable reference in slow path, so the simple atomic add could be used. The performance difference should be trivial, but the misuse may be confusing and misleading. Signed-off-by: Yang Shi --- mm/gup.c | 112 +++++++++++++++++++++++++++-------------------- mm/huge_memory.c | 2 +- mm/internal.h | 3 +- 3 files changed, 66 insertions(+), 51 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 17f89e8d31f1..a683e7ac47b5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -100,7 +100,7 @@ static inline struct folio *try_get_folio(struct page *page, int refs) } /** - * try_grab_folio() - Attempt to get or pin a folio. + * try_grab_folio_fast() - Attempt to get or pin a folio in fast path. * @page: pointer to page to be grabbed * @refs: the value to (effectively) add to the folio's refcount * @flags: gup flags: these are the FOLL_* flag values. @@ -124,11 +124,18 @@ static inline struct folio *try_get_folio(struct page *page, int refs) * incremented) for success, or NULL upon failure. If neither FOLL_GET * nor FOLL_PIN was set, that's considered failure, and furthermore, * a likely bug in the caller, so a warning is also emitted. + * + * It uses add ref unless zero to elevate the folio refcount and must be called + * in fast path only. */ -struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) +static struct folio *try_grab_folio_fast(struct page *page, int refs, + unsigned int flags) { struct folio *folio; + /* Raise warn if it is not called in fast GUP */ + VM_WARN_ON_ONCE(!irqs_disabled()); + if (WARN_ON_ONCE((flags & (FOLL_GET | FOLL_PIN)) == 0)) return NULL; @@ -205,28 +212,31 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) } /** - * try_grab_page() - elevate a page's refcount by a flag-dependent amount - * @page: pointer to page to be grabbed - * @flags: gup flags: these are the FOLL_* flag values. + * try_grab_folio() - add a folio's refcount by a flag-dependent amount + * @folio: pointer to folio to be grabbed + * @refs: the value to (effectively) add to the folio's refcount + * @flags: gup flags: these are the FOLL_* flag values. * * This might not do anything at all, depending on the flags argument. * * "grab" names in this file mean, "look at flags to decide whether to use - * FOLL_PIN or FOLL_GET behavior, when incrementing the page's refcount. + * FOLL_PIN or FOLL_GET behavior, when incrementing the folio's refcount. * * Either FOLL_PIN or FOLL_GET (or neither) may be set, but not both at the same - * time. Cases: please see the try_grab_folio() documentation, with - * "refs=1". + * time. * * Return: 0 for success, or if no action was required (if neither FOLL_PIN * nor FOLL_GET was set, nothing is done). A negative error code for failure: * - * -ENOMEM FOLL_GET or FOLL_PIN was set, but the page could not + * -ENOMEM FOLL_GET or FOLL_PIN was set, but the folio could not * be grabbed. + * + * It is called when we have a stable reference for the folio, typically in + * GUP slow path. */ -int __must_check try_grab_page(struct page *page, unsigned int flags) +int __must_check try_grab_folio(struct folio *folio, int refs, unsigned int flags) { - struct folio *folio = page_folio(page); + struct page *page = &folio->page; if (WARN_ON_ONCE(folio_ref_count(folio) <= 0)) return -ENOMEM; @@ -235,7 +245,7 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) return -EREMOTEIO; if (flags & FOLL_GET) - folio_ref_inc(folio); + folio_ref_add(folio, refs); else if (flags & FOLL_PIN) { /* * Don't take a pin on the zero page - it's not going anywhere @@ -245,18 +255,18 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) return 0; /* - * Similar to try_grab_folio(): be sure to *also* - * increment the normal page refcount field at least once, + * Increment the normal page refcount field at least once, * so that the page really is pinned. */ if (folio_test_large(folio)) { - folio_ref_add(folio, 1); - atomic_add(1, &folio->_pincount); + folio_ref_add(folio, refs); + atomic_add(refs, &folio->_pincount); } else { - folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); + folio_ref_add(folio, + refs * GUP_PIN_COUNTING_BIAS); } - node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, 1); + node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); } return 0; @@ -584,7 +594,7 @@ static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, */ static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz, unsigned long addr, unsigned long end, unsigned int flags, - struct page **pages, int *nr) + struct page **pages, int *nr, bool fast) { unsigned long pte_end; struct page *page; @@ -607,9 +617,15 @@ static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz page = pte_page(pte); refs = record_subpages(page, sz, addr, end, pages + *nr); - folio = try_grab_folio(page, refs, flags); - if (!folio) - return 0; + if (fast) { + folio = try_grab_folio_fast(page, refs, flags); + if (!folio) + return 0; + } else { + folio = page_folio(page); + if (try_grab_folio(folio, refs, flags)) + return 0; + } if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { gup_put_folio(folio, refs, flags); @@ -637,7 +653,7 @@ static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz static int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, - struct page **pages, int *nr) + struct page **pages, int *nr, bool fast) { pte_t *ptep; unsigned long sz = 1UL << hugepd_shift(hugepd); @@ -647,7 +663,7 @@ static int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, ptep = hugepte_offset(hugepd, addr, pdshift); do { next = hugepte_addr_end(addr, end, sz); - ret = gup_hugepte(vma, ptep, sz, addr, end, flags, pages, nr); + ret = gup_hugepte(vma, ptep, sz, addr, end, flags, pages, nr, fast); if (ret != 1) return ret; } while (ptep++, addr = next, addr != end); @@ -674,7 +690,7 @@ static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, ptep = hugepte_offset(hugepd, addr, pdshift); ptl = huge_pte_lock(h, vma->vm_mm, ptep); ret = gup_hugepd(vma, hugepd, addr, pdshift, addr + PAGE_SIZE, - flags, &page, &nr); + flags, &page, &nr, false); spin_unlock(ptl); if (ret == 1) { @@ -691,7 +707,7 @@ static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, static inline int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, - struct page **pages, int *nr) + struct page **pages, int *nr, bool fast) { return 0; } @@ -778,7 +794,7 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, gup_must_unshare(vma, flags, page)) return ERR_PTR(-EMLINK); - ret = try_grab_page(page, flags); + ret = try_grab_folio(page_folio(page), 1, flags); if (ret) page = ERR_PTR(ret); else @@ -855,7 +871,7 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma, VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && !PageAnonExclusive(page), page); - ret = try_grab_page(page, flags); + ret = try_grab_folio(page_folio(page), 1, flags); if (ret) return ERR_PTR(ret); @@ -1017,8 +1033,8 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && !PageAnonExclusive(page), page); - /* try_grab_page() does nothing unless FOLL_GET or FOLL_PIN is set. */ - ret = try_grab_page(page, flags); + /* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */ + ret = try_grab_folio(page_folio(page), 1, flags); if (unlikely(ret)) { page = ERR_PTR(ret); goto out; @@ -1282,7 +1298,7 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address, goto unmap; *page = pte_page(entry); } - ret = try_grab_page(*page, gup_flags); + ret = try_grab_folio(page_folio(*page), 1, gup_flags); if (unlikely(ret)) goto unmap; out: @@ -1685,20 +1701,19 @@ static long __get_user_pages(struct mm_struct *mm, * pages. */ if (page_increm > 1) { - struct folio *folio; + struct folio *folio = page_folio(page); /* * Since we already hold refcount on the * large folio, this should never fail. */ - folio = try_grab_folio(page, page_increm - 1, - foll_flags); - if (WARN_ON_ONCE(!folio)) { + if (try_grab_folio(folio, page_increm - 1, + foll_flags)) { /* * Release the 1st page ref if the * folio is problematic, fail hard. */ - gup_put_folio(page_folio(page), 1, + gup_put_folio(folio, 1, foll_flags); ret = -EFAULT; goto out; @@ -3041,7 +3056,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, VM_BUG_ON(!pfn_valid(pte_pfn(pte))); page = pte_page(pte); - folio = try_grab_folio(page, 1, flags); + folio = try_grab_folio_fast(page, 1, flags); if (!folio) goto pte_unmap; @@ -3128,7 +3143,7 @@ static int gup_fast_devmap_leaf(unsigned long pfn, unsigned long addr, break; } - folio = try_grab_folio(page, 1, flags); + folio = try_grab_folio_fast(page, 1, flags); if (!folio) { gup_fast_undo_dev_pagemap(nr, nr_start, flags, pages); break; @@ -3217,7 +3232,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, page = pmd_page(orig); refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); - folio = try_grab_folio(page, refs, flags); + folio = try_grab_folio_fast(page, refs, flags); if (!folio) return 0; @@ -3261,7 +3276,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, page = pud_page(orig); refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); - folio = try_grab_folio(page, refs, flags); + folio = try_grab_folio_fast(page, refs, flags); if (!folio) return 0; @@ -3301,7 +3316,7 @@ static int gup_fast_pgd_leaf(pgd_t orig, pgd_t *pgdp, unsigned long addr, page = pgd_page(orig); refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); - folio = try_grab_folio(page, refs, flags); + folio = try_grab_folio_fast(page, refs, flags); if (!folio) return 0; @@ -3355,7 +3370,7 @@ static int gup_fast_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, * pmd format and THP pmd format */ if (gup_hugepd(NULL, __hugepd(pmd_val(pmd)), addr, - PMD_SHIFT, next, flags, pages, nr) != 1) + PMD_SHIFT, next, flags, pages, nr, true) != 1) return 0; } else if (!gup_fast_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) @@ -3385,7 +3400,7 @@ static int gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, return 0; } else if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) { if (gup_hugepd(NULL, __hugepd(pud_val(pud)), addr, - PUD_SHIFT, next, flags, pages, nr) != 1) + PUD_SHIFT, next, flags, pages, nr, true) != 1) return 0; } else if (!gup_fast_pmd_range(pudp, pud, addr, next, flags, pages, nr)) @@ -3412,7 +3427,7 @@ static int gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, BUILD_BUG_ON(p4d_leaf(p4d)); if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) { if (gup_hugepd(NULL, __hugepd(p4d_val(p4d)), addr, - P4D_SHIFT, next, flags, pages, nr) != 1) + P4D_SHIFT, next, flags, pages, nr, true) != 1) return 0; } else if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) @@ -3441,7 +3456,7 @@ static void gup_fast_pgd_range(unsigned long addr, unsigned long end, return; } else if (unlikely(is_hugepd(__hugepd(pgd_val(pgd))))) { if (gup_hugepd(NULL, __hugepd(pgd_val(pgd)), addr, - PGDIR_SHIFT, next, flags, pages, nr) != 1) + PGDIR_SHIFT, next, flags, pages, nr, true) != 1) return; } else if (!gup_fast_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) @@ -3842,13 +3857,14 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, next_idx != folio_index(fbatch.folios[i])) continue; - folio = try_grab_folio(&fbatch.folios[i]->page, - 1, FOLL_PIN); - if (!folio) { + if (try_grab_folio(fbatch.folios[i], + 1, FOLL_PIN)) { folio_batch_release(&fbatch); goto err; } + folio = fbatch.folios[i]; + if (nr_folios == 0) *offset = offset_in_folio(folio, start); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8e49f402d7c7..b6280a01c5fd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1331,7 +1331,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, if (!*pgmap) return ERR_PTR(-EFAULT); page = pfn_to_page(pfn); - ret = try_grab_page(page, flags); + ret = try_grab_folio(page_folio(page), 1, flags); if (ret) page = ERR_PTR(ret); diff --git a/mm/internal.h b/mm/internal.h index 3419c329b3bc..dc358cd51135 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1215,8 +1215,7 @@ int migrate_device_coherent_page(struct page *page); /* * mm/gup.c */ -struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags); -int __must_check try_grab_page(struct page *page, unsigned int flags); +int __must_check try_grab_folio(struct folio *folio, int refs, unsigned int flags); /* * mm/huge_memory.c