From patchwork Sun Apr 13 03:59:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 14049180 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49B033D6A; Sun, 13 Apr 2025 04:05:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744517112; cv=fail; b=SyRueWiugxcbWkm80mI42GjPUyCvOtc4x9XcVO130H5iGY/Yf+c78eFanHYRpYF2JBDlmRbDRisdVL/rWxVv7AoZZmOKBWTX99n9DQxGijtLdhYPcZTk/eTVRJS+kgB9YiCP1Nr1YjfeTdaGatfbfuOPb1ZdIBEGJGgKPOwxvBc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744517112; c=relaxed/simple; bh=9nyYW31kUOCnW8G76VymNCKKLJXvp2sdPvDtxCB808M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=SytjNm4Qwdqb4di3jnlqeQAxH8acN14DPJ+J76tDWYo6thlaY1YoN25NWwHh/yshCGBwKQREKQoGcg3SmhWiTcndNdYwO03ZuSoFS4USfJOqT215Hh9enXPmVzHTMgnTmYglT7CxkXpYTjfyJsKq5mpf9Ir5ffU8OjRXMU2S5iQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=TUl1fuhh; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=w8bv8nQi; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="TUl1fuhh"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="w8bv8nQi" Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53D41GDE023002; Sun, 13 Apr 2025 04:05:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=IXrBW+j3KMd2rEHbx4JpJOyfU27YPpGfL4ef8mv6/T0=; b= TUl1fuhhLVagvCWaJGpHHYRlUjo5DjODY5jOwajlgo1v4/up9g5js0u0BOEqnepd QI83ddHyztE7mIERAQ4DkLOY43gIhGWnLeH9B70vtwkOgcs0YXtvCcRhVI/GqqgL 8iC6YDS/6If9HdLFQMgtXejDEZj9CFP8qiM9EuQyajzBdomVJWCDeloKp5rpz/22 E1AADVfQUerPsdeISqFVzO0ixeb5A/gXD0IlsXi4FqzOqdlyj12NLZcDiSy95mnx gsr+vcll/kaqOhdWzilUg4u2Tdz51tai/cV4lcbRGNdY3jXltZb/W8LuGGzNZz+a dbMmW75NRKqNGlP3g2EJgA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 4605g4g0d0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 13 Apr 2025 04:05:08 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53D3NOek004934; Sun, 13 Apr 2025 04:05:08 GMT Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012051.outbound.protection.outlook.com [40.93.20.51]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 45yem6h9g5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 13 Apr 2025 04:05:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=I7AJJsDTqz1nEmXolRXIyE7WgT6zOU4lZAzQSZFhzj1I5l68IFiOEe+9kXe2A9CXCVAwiEVxW0K2VX/+9XRKTdskKvqO11NCe5xaUbyuOIn/uRVabLuDUc1D4AuECKDQxGATF9/dyWtbDxysB28meieglW1tefKO2MX0TFfFV6DCjGqkyvxgmkxpNuSH/IMyzfJ9oS6ai2jy29uDLHw5PCIxb6Ux6dCk/oHIknjnX8FAz7can933nYNf06VRgsI70pMv960Z+0QrPZuEiSOESR4KyVA9L7y0XA9nfTBStRnNWzURjhpjPfRE1yJOj661Cu7ijzrL3Xg1TRNg0m9e+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IXrBW+j3KMd2rEHbx4JpJOyfU27YPpGfL4ef8mv6/T0=; b=riDQtjcvqyJqhNOnoZtL9RA70Asy0F0UymAc/3x4RXkmqkNTeiZZfW85Up4IqTFdBkvsPDdztfxMQenaHGAjvT9bafOZZyeMpbvaboeawy4R605NFWLTTO8ddcwOA6gx9yaTqMvTZpypoYdM9s1lzztMxp70FdSdiBbJhdTUcJSSHlgLnS3IpA39I/MTFaqbbgE1WIyDvQLBrrzs8p1BQg5aBrlLv61NvG3Is+FAafTmdOd0DeVEwJJf5w9P1EORk6IJi+yLP68p8++B7Ubqbhwur3io/9mbu9aR++OTUn5L/SgK21u7RDkRiBBNSbAP8f4RuYO0O3+HJtnGJPEgqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IXrBW+j3KMd2rEHbx4JpJOyfU27YPpGfL4ef8mv6/T0=; b=w8bv8nQigTZj67hlxfBw+64b/A5gP0rVhU7Ofsf/Apzd2msd/MOvDQf18PA/PgMQEF6BMiAeh6hdWKjlmoShYXCcAJD39SOl7JFUl1cZto/z/3vr9Y5TyZI6205xStJ2MN1yEiVtWw8MpPsEMLMpbkRQPauWwOWgzk9w8G5RtGI= Received: from CY8PR10MB7243.namprd10.prod.outlook.com (2603:10b6:930:7c::10) by CH2PR10MB4120.namprd10.prod.outlook.com (2603:10b6:610:7d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.32; Sun, 13 Apr 2025 04:05:05 +0000 Received: from CY8PR10MB7243.namprd10.prod.outlook.com ([fe80::b779:d0be:9e3a:34f0]) by CY8PR10MB7243.namprd10.prod.outlook.com ([fe80::b779:d0be:9e3a:34f0%5]) with mapi id 15.20.8632.030; Sun, 13 Apr 2025 04:05:05 +0000 From: Mike Christie To: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH 1/2] target: Move IO path stats to per cpu Date: Sat, 12 Apr 2025 22:59:50 -0500 Message-ID: <20250413040500.20954-2-michael.christie@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250413040500.20954-1-michael.christie@oracle.com> References: <20250413040500.20954-1-michael.christie@oracle.com> X-ClientProxiedBy: DM6PR07CA0080.namprd07.prod.outlook.com (2603:10b6:5:337::13) To CY8PR10MB7243.namprd10.prod.outlook.com (2603:10b6:930:7c::10) Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY8PR10MB7243:EE_|CH2PR10MB4120:EE_ X-MS-Office365-Filtering-Correlation-Id: 559b3eaa-5417-4fb0-d7ac-08dd7a405f21 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: ohejDK+Ja5rL01ZFErmODK35FVNLD//VR4dK7sL4Qu9nhZJ3o53//lO3nuCb/g9w92fTLiRgkLjXLUZpV9cnpWAl3NgOHtaHdvEwTJuqAD3heb62PRsF1lrYG0f2Xp3NExtRsV22EdGuvWhF8g/leyuGSGA7KoHfVYn9I5CZhLs6Cbduj55Ef/AitSGEmQha+qHc4iMYI1xxT4i5rGB/rDAr3dTi85KBPCw8gVtqVfwQZWB6aIuu7fhF9t9js5uzcRsz7Bw8KJ2LT/URwFd9QBLjFNjxKbKuCUkeUNrx582QRKQ6tytWh2kvWvIGwcDwvW+OexLLRmSacb6/LeWdifTwGbd6R9QL8vUPofkkZJqoqI1/iGzAs8meV5dQiV2cJHoGoYXFNZTK5AgvCsL50O5lVkWedZ3nswFinOtWdHMGdfrunGFH/97n9QSMQKNKp2GAornzvrEIn8GmL0G8hS2S59ZbOxvhOOxGKR3jQNBFVhtyKbjje1BzV1WBqdc6VyCSJBddEBdMT1W/rbdyXc7oYWKBMm4PWaaSlxTJRUMQAhZKBljH4HTVcRnBrfjJY6jpm3GrPtMlxgZWomm4EX3NfDY+fa27Wq++RwY7ewURJznoKivWz4aACIX85I02vwq/p0BaMX8/9wplyjI6rY6yb/1FJ8bHOaft9cbyB7GXwrP3mTik4L3koj48nuGgN+9xwA0DZZsN3u5CEYXuUOPETC1kbhHauj0JgKHb+RKt+F5Evr9aJqzl7MqHL2BtyPmsH4Qe4YCDx04x7UR9OnOA59r0SXL8NxQ/EqKB9hn8X4x7V9VcWTiI9X+4FzkX18Ti3IumCCxrktgNvUBRcE/NoLkOd30FQYrz98D5Q4SGALO3xWSTRNev6Adkmpzpi6kovydHTZTs2zNR8J5GbASnIVWDF6ftgVsHlDllHa+rusbj0X+JeLgFC6xFg0B96wQQUBh71qMWUjWmCy/J21vUSwTyybJgayJWgBiUBqCpIcsEtNEx3qAj9+pwTKv/jJkkybgTvcVyaejIwauP5jTjLCA09ywujqWss9YtSUYSAbZsQKotvoTmPUzYhXdFLfihnqJfL2BPy2fc90RGE67vnFkHJ3c/5kj8ixe38ilkfbI+UD5HTfQjE7yyOFjku5xCXSGUbwFGWX7dVhrWDnVnWVS3gutwQ4zvxkgHu3Gsc+9NoCLAZ6FWYv1YGZ9t6VVLfbDfkLNgozhHo+P1l8BcLj9a5jK75b9IQufKNGtNQwBriOjViAchUI/zMHAET57p4CiqQF22O+NirNbQ06ytzWtN3Y6Iv6UteTwSfIRKmHvPG0B5+Rkfo1xPzbgD2UXZV3Jd122cW9UtP7Xmc6D/BjHlajtiW2hpDufiuICJiKj2emxMQEnNU4sIo+nBrxodsXZLXF5wYbi8MSY8Xg5XNmuHy7AmAWRLVXzc+Nw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY8PR10MB7243.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pxggC3m6fBdbZLbTp8KGzFGCpdudA65beJz6t8owKPDZEMTP/+DzZCwhFmpDU0xRArRHPxtCaNCAfP46TsY4Z4+q0gwsIvnRwU2FL5cGNoLuRaleDHKbJ2S3vamYeLM2X41QKB2Aay2wjgFk05z0DPAPOb3iUN5VlBRAql7O4SXQSl/dYGCIGe4OCSubsVK1wKQWGN+Wu2FIZt+pImW3YqIntIO0A+7q4KVLrJTuBZ6DT1xYUhlMcKPwPO7VXFkKOmUDN+FHPwi2CDDJSsFRQ5MP6Pb9/RFOQgHUuyzsUofQ1GG508+h8gqcQ6vGPYXyUsWR1i0hbNEpLL+KycW8zssWdyybFeZkVxqmmXfQvwpuIV9u82zn5Ro43T2zKa3YVkrynVUxOuvXB32fwkI5NN9ikEPzHoYuJAmGF9PDDLqODbf4igKeHCCZz5+64Y7K37xZozFm2jJOy+I2WV5zxPXWNyfQI4uFFFEyNS/Vu59bSodmd+mqRbC4Bq7nPga3A64CepSK5KmL5kIkqiit9WSoMlg58ciF2mkla0cZOw4rFmsnvTbQw9OS3a9Ij/s1EUfZ5A3QYQp2YzKq9fNeHrU5cTscsJRCRysVyV/kX9i5C6Gd2KfI1eelWUH81tyi8rdIC+Fzle3ac6jKltDo3ELD0N4VY7c3nyaZIly3HFZM6w30rRxNXXPiRb6+ycMGgMfGUy5ayE34NNxR7774PH6ykbBKAmTUdVdKAfs4EpZp66JlUPK7l2tG3sTixSdQvTAr6tPCV7/v6ZtU0CaI0zUK0OhfcILRGSCBZu2JjX4IRNwkd/Z5H+0qNmA5yCHkGnrB/CVAwWiautmhijSVRe4GQmJTLgjHrDr+bR88BoY7hevbtYJdhl7J2zzMJTTZfCmj2zZeLQYLT/npOSJ0MGWxm6dny7b/oXafthgVGjeoDjKKOk3lGX5D4Z2MBS64ASbI8mH4BUnutsoLyoJELE2K9GPgToBgxoIHDwu/LMT629PndPC8MR2JOxWrUQ2C2QwviQ3Sx7gGdTzawCgEH3wQZ+NG21O+mnIYc4U6Q8Y6/3ow30B8hVCvZ9eABqjoJH/Nfn8vH8zJaK3S63eaWS+8D6k/sizyDm4zOLiJiKlxpuakPOavTDxd3L0Ihy9JvONn9SgrfnTZFRjFGsps9QLBSwB3XikAnFUjm3OEjIg9XqAXqhfG16rofV7pS9yNfTcG1nDF9YB9lCHlIwrME0XXWuVh42p5scIjlhVbHpommoTLNXTFmeDTHFGUQdW30DVGCJ/2z8+4Vfpqtvsc0uzlQGpbaMpiSpyWXBcuRzbgxYrzWIDKiWsPLykyffMXAMOr2c6nlz9RbayGBHKhkAJKGmpec5PYdiQiAAocANisPSoZy/ivMxTYsr8j+3tE8a6CUSYwfwXVePpaKqkMpCjiQE1n4zaqZmR78VCld/4U0sS1EkIXLvw1Ahe1q06gu5y725yjzcCOkWN3APbUsDoqI3f/odSzGXFSIp6nYEnczvw9j6R08t3zz48artw5LDjXtp6zVJPf2pUxF3mL/w5QdqSdfpWfm3cc+QH1b2SRza+/V+2PMd12qIkSTwwVp7OuzTfCQpI7JGUUxep9bw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: EQmAFDiX5D3lo/famgxakIbv21XpZ5S8Y6yv0thplu1gDj0iMr4x4E2iKt7sCKSdAXQLL7pMIbdjAV+/CYMwZ83wm5RrGlIdHAfLzw6s0WqnEfq23k5tBU3mPysX8PamYsOmaZvlON3GiahnBM1T80cCJb1/nspSNTjSnBq1IhyazgvNd9WB4rUDjreWNBJ5qXuxLzGu+71zwtQx5pBkTEjyxBuLPMdKa9IDgLrePn3ElO0EoOi9EbI4lB35MS9lQ9y/MKX6WwxqM/VtSZvVCORLZEvA6J7secm9INYmeXKAs6aOw8Q03CY+BYSrcSMR+Ep8s2MSWXg80ONtrh5QFJgjikotBrbqpMUyvIEpWGvEC1GIPBx7rjLaNxvnUUD7SYL2P1wCFJ3jJL8eXjXc2XeYenjjuGOvg97eN/TkAAJwvqT2RuhMgZcO7jEu79ww46Gk68bp0GIey3ZCtVHpGpIxo1+pKul5UGow+Bu5ic8chCjDpEKbpWtOM4Ytekr9wsOeCpguAdoGEq1r05FQIHjGJlFaDllOQw0DIeIg1f/7JuHzGS++JdzMXLxo+awj7sDqzHY+ZL5PATXtcmpkZQEAsouKlB8rfco+vd8F9O0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 559b3eaa-5417-4fb0-d7ac-08dd7a405f21 X-MS-Exchange-CrossTenant-AuthSource: CY8PR10MB7243.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Apr 2025 04:05:05.0669 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vNQg3wZlJZh8OIXvRZJYED3YSQVRiljd3lk+sTxBRMJ6HTfIcg/FpwyMGGwx+SA4LqoXs7lp2B5m+J8FpY4B+PBBXDy814gUcQfPRtthSpI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR10MB4120 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-13_02,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504130028 X-Proofpoint-GUID: YXItNpHCxwzPEM1pdvbbNNbF2EOfg3TA X-Proofpoint-ORIG-GUID: YXItNpHCxwzPEM1pdvbbNNbF2EOfg3TA The atomic use in the main IO path is causing perf issues when using higher performance backend devices and multiple queues. This moves the stats to per cpu. Combined with the next patch that moves the non_ordered/delayed_cmd_count to per cpu, IOPS by up to 33% for 8K IOS when using 4 or more queues. Signed-off-by: Mike Christie --- drivers/target/target_core_device.c | 69 +++++++++++++++++++++-------- drivers/target/target_core_stat.c | 69 ++++++++++++++++++++++++----- include/target/target_core_base.h | 20 ++++++--- 3 files changed, 121 insertions(+), 37 deletions(-) diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c index cc2da086f96e..39aad464c0bf 100644 --- a/drivers/target/target_core_device.c +++ b/drivers/target/target_core_device.c @@ -55,14 +55,14 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd) rcu_read_lock(); deve = target_nacl_find_deve(nacl, se_cmd->orig_fe_lun); if (deve) { - atomic_long_inc(&deve->total_cmds); + this_cpu_inc(deve->stats->total_cmds); if (se_cmd->data_direction == DMA_TO_DEVICE) - atomic_long_add(se_cmd->data_length, - &deve->write_bytes); + this_cpu_add(deve->stats->write_bytes, + se_cmd->data_length); else if (se_cmd->data_direction == DMA_FROM_DEVICE) - atomic_long_add(se_cmd->data_length, - &deve->read_bytes); + this_cpu_add(deve->stats->read_bytes, + se_cmd->data_length); if ((se_cmd->data_direction == DMA_TO_DEVICE) && deve->lun_access_ro) { @@ -126,14 +126,14 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd) * target_core_fabric_configfs.c:target_fabric_port_release */ se_cmd->se_dev = rcu_dereference_raw(se_lun->lun_se_dev); - atomic_long_inc(&se_cmd->se_dev->num_cmds); + this_cpu_inc(se_cmd->se_dev->stats->total_cmds); if (se_cmd->data_direction == DMA_TO_DEVICE) - atomic_long_add(se_cmd->data_length, - &se_cmd->se_dev->write_bytes); + this_cpu_add(se_cmd->se_dev->stats->write_bytes, + se_cmd->data_length); else if (se_cmd->data_direction == DMA_FROM_DEVICE) - atomic_long_add(se_cmd->data_length, - &se_cmd->se_dev->read_bytes); + this_cpu_add(se_cmd->se_dev->stats->read_bytes, + se_cmd->data_length); return ret; } @@ -322,6 +322,7 @@ int core_enable_device_list_for_node( struct se_portal_group *tpg) { struct se_dev_entry *orig, *new; + int ret = 0; new = kzalloc(sizeof(*new), GFP_KERNEL); if (!new) { @@ -329,6 +330,12 @@ int core_enable_device_list_for_node( return -ENOMEM; } + new->stats = alloc_percpu(struct se_dev_entry_io_stats); + if (!new->stats) { + ret = -ENOMEM; + goto free_deve; + } + spin_lock_init(&new->ua_lock); INIT_LIST_HEAD(&new->ua_list); INIT_LIST_HEAD(&new->lun_link); @@ -351,8 +358,8 @@ int core_enable_device_list_for_node( " for dynamic -> explicit NodeACL conversion:" " %s\n", nacl->initiatorname); mutex_unlock(&nacl->lun_entry_mutex); - kfree(new); - return -EINVAL; + ret = -EINVAL; + goto free_stats; } if (orig->se_lun_acl != NULL) { pr_warn_ratelimited("Detected existing explicit" @@ -360,8 +367,8 @@ int core_enable_device_list_for_node( " mapped_lun: %llu, failing\n", nacl->initiatorname, mapped_lun); mutex_unlock(&nacl->lun_entry_mutex); - kfree(new); - return -EINVAL; + ret = -EINVAL; + goto free_stats; } new->se_lun = lun; @@ -394,6 +401,20 @@ int core_enable_device_list_for_node( target_luns_data_has_changed(nacl, new, true); return 0; + +free_stats: + free_percpu(new->stats); +free_deve: + kfree(new); + return ret; +} + +static void target_free_dev_entry(struct rcu_head *head) +{ + struct se_dev_entry *deve = container_of(head, struct se_dev_entry, + rcu_head); + free_percpu(deve->stats); + kfree(deve); } void core_disable_device_list_for_node( @@ -443,7 +464,7 @@ void core_disable_device_list_for_node( kref_put(&orig->pr_kref, target_pr_kref_release); wait_for_completion(&orig->pr_comp); - kfree_rcu(orig, rcu_head); + call_rcu(&orig->rcu_head, target_free_dev_entry); core_scsi3_free_pr_reg_from_nacl(dev, nacl); target_luns_data_has_changed(nacl, NULL, false); @@ -689,11 +710,13 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) if (!dev) return NULL; + dev->stats = alloc_percpu(struct se_dev_io_stats); + if (!dev->stats) + goto free_device; + dev->queues = kcalloc(nr_cpu_ids, sizeof(*dev->queues), GFP_KERNEL); - if (!dev->queues) { - hba->backend->ops->free_device(dev); - return NULL; - } + if (!dev->queues) + goto free_stats; dev->queue_cnt = nr_cpu_ids; for (i = 0; i < dev->queue_cnt; i++) { @@ -707,6 +730,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) INIT_WORK(&q->sq.work, target_queued_submit_work); } + dev->se_hba = hba; dev->transport = hba->backend->ops; dev->transport_flags = dev->transport->transport_flags_default; @@ -791,6 +815,12 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) sizeof(dev->t10_wwn.revision)); return dev; + +free_stats: + free_percpu(dev->stats); +free_device: + hba->backend->ops->free_device(dev); + return NULL; } /* @@ -1001,6 +1031,7 @@ void target_free_device(struct se_device *dev) dev->transport->free_prot(dev); kfree(dev->queues); + free_percpu(dev->stats); dev->transport->free_device(dev); } diff --git a/drivers/target/target_core_stat.c b/drivers/target/target_core_stat.c index 210648a0092e..0aafc900c3aa 100644 --- a/drivers/target/target_core_stat.c +++ b/drivers/target/target_core_stat.c @@ -280,30 +280,51 @@ static ssize_t target_stat_lu_num_cmds_show(struct config_item *item, char *page) { struct se_device *dev = to_stat_lu_dev(item); + struct se_dev_io_stats *stats; + unsigned int cpu; + u32 cmds; + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(dev->stats, cpu); + cmds += stats->total_cmds; + } /* scsiLuNumCommands */ - return snprintf(page, PAGE_SIZE, "%lu\n", - atomic_long_read(&dev->num_cmds)); + return snprintf(page, PAGE_SIZE, "%u\n", cmds); } static ssize_t target_stat_lu_read_mbytes_show(struct config_item *item, char *page) { struct se_device *dev = to_stat_lu_dev(item); + struct se_dev_io_stats *stats; + unsigned int cpu; + u32 bytes; + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(dev->stats, cpu); + bytes += stats->read_bytes; + } /* scsiLuReadMegaBytes */ - return snprintf(page, PAGE_SIZE, "%lu\n", - atomic_long_read(&dev->read_bytes) >> 20); + return snprintf(page, PAGE_SIZE, "%u\n", bytes >> 20); } static ssize_t target_stat_lu_write_mbytes_show(struct config_item *item, char *page) { struct se_device *dev = to_stat_lu_dev(item); + struct se_dev_io_stats *stats; + unsigned int cpu; + u32 bytes; + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(dev->stats, cpu); + bytes += stats->write_bytes; + } /* scsiLuWrittenMegaBytes */ - return snprintf(page, PAGE_SIZE, "%lu\n", - atomic_long_read(&dev->write_bytes) >> 20); + return snprintf(page, PAGE_SIZE, "%u\n", bytes >> 20); } static ssize_t target_stat_lu_resets_show(struct config_item *item, char *page) @@ -1019,8 +1040,11 @@ static ssize_t target_stat_auth_num_cmds_show(struct config_item *item, { struct se_lun_acl *lacl = auth_to_lacl(item); struct se_node_acl *nacl = lacl->se_lun_nacl; + struct se_dev_entry_io_stats *stats; struct se_dev_entry *deve; + unsigned int cpu; ssize_t ret; + u32 cmds; rcu_read_lock(); deve = target_nacl_find_deve(nacl, lacl->mapped_lun); @@ -1028,9 +1052,14 @@ static ssize_t target_stat_auth_num_cmds_show(struct config_item *item, rcu_read_unlock(); return -ENODEV; } + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(deve->stats, cpu); + cmds += stats->total_cmds; + } + /* scsiAuthIntrOutCommands */ - ret = snprintf(page, PAGE_SIZE, "%lu\n", - atomic_long_read(&deve->total_cmds)); + ret = snprintf(page, PAGE_SIZE, "%u\n", cmds); rcu_read_unlock(); return ret; } @@ -1040,8 +1069,11 @@ static ssize_t target_stat_auth_read_mbytes_show(struct config_item *item, { struct se_lun_acl *lacl = auth_to_lacl(item); struct se_node_acl *nacl = lacl->se_lun_nacl; + struct se_dev_entry_io_stats *stats; struct se_dev_entry *deve; + unsigned int cpu; ssize_t ret; + u32 bytes; rcu_read_lock(); deve = target_nacl_find_deve(nacl, lacl->mapped_lun); @@ -1049,9 +1081,14 @@ static ssize_t target_stat_auth_read_mbytes_show(struct config_item *item, rcu_read_unlock(); return -ENODEV; } + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(deve->stats, cpu); + bytes += stats->read_bytes; + } + /* scsiAuthIntrReadMegaBytes */ - ret = snprintf(page, PAGE_SIZE, "%u\n", - (u32)(atomic_long_read(&deve->read_bytes) >> 20)); + ret = snprintf(page, PAGE_SIZE, "%u\n", bytes >> 20); rcu_read_unlock(); return ret; } @@ -1061,8 +1098,11 @@ static ssize_t target_stat_auth_write_mbytes_show(struct config_item *item, { struct se_lun_acl *lacl = auth_to_lacl(item); struct se_node_acl *nacl = lacl->se_lun_nacl; + struct se_dev_entry_io_stats *stats; struct se_dev_entry *deve; + unsigned int cpu; ssize_t ret; + u32 bytes; rcu_read_lock(); deve = target_nacl_find_deve(nacl, lacl->mapped_lun); @@ -1070,9 +1110,14 @@ static ssize_t target_stat_auth_write_mbytes_show(struct config_item *item, rcu_read_unlock(); return -ENODEV; } + + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(deve->stats, cpu); + bytes += stats->write_bytes; + } + /* scsiAuthIntrWrittenMegaBytes */ - ret = snprintf(page, PAGE_SIZE, "%u\n", - (u32)(atomic_long_read(&deve->write_bytes) >> 20)); + ret = snprintf(page, PAGE_SIZE, "%u\n", bytes >> 20); rcu_read_unlock(); return ret; } diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index 97099a5e3f6c..05e3673607b8 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -669,15 +669,19 @@ struct se_lun_acl { struct se_ml_stat_grps ml_stat_grps; }; +struct se_dev_entry_io_stats { + u32 total_cmds; + u32 read_bytes; + u32 write_bytes; +}; + struct se_dev_entry { u64 mapped_lun; u64 pr_res_key; u64 creation_time; bool lun_access_ro; u32 attach_count; - atomic_long_t total_cmds; - atomic_long_t read_bytes; - atomic_long_t write_bytes; + struct se_dev_entry_io_stats __percpu *stats; /* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */ struct kref pr_kref; struct completion pr_comp; @@ -800,6 +804,12 @@ struct se_device_queue { struct se_cmd_queue sq; }; +struct se_dev_io_stats { + u32 total_cmds; + u32 read_bytes; + u32 write_bytes; +}; + struct se_device { /* Used for SAM Task Attribute ordering */ u32 dev_cur_ordered_id; @@ -821,9 +831,7 @@ struct se_device { atomic_long_t num_resets; atomic_long_t aborts_complete; atomic_long_t aborts_no_task; - atomic_long_t num_cmds; - atomic_long_t read_bytes; - atomic_long_t write_bytes; + struct se_dev_io_stats __percpu *stats; /* Active commands on this virtual SE device */ atomic_t non_ordered; bool ordered_sync_in_progress; From patchwork Sun Apr 13 03:59:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 14049181 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F39643D6A; Sun, 13 Apr 2025 04:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744517116; cv=fail; b=KJ5fJXqITqmxl5LIBXBzpxStoMAZZO1xTH5fd4zeTThbXuyGNPSbFvs6oE9FnL50gwHs9b6Gjol6xT7Y/8170aqiVcijysxGAuCpq3fTEnNh4fLZUwRdj1sKs84yb2CuVdtJlQrVv2XV5qJ3jWBhMLiPr8A8bX5YtCjh8QKx06M= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744517116; c=relaxed/simple; bh=DnJOHvHVQX7/PYt0xbCywc7hK0vi78IimKay4yHFa9Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=qZvWl2NdRUG3oCQKq1auqPifU39s8QVOoFLOUA40crLA9MJMug+vUcE8g58rY6yvNljbcw9wcP36WEDGFLqlWFgzzAiF4ZdoGyZYjngrBdqBi8US7MNCaksmwycSsI5Jca8PMTu57qfqltPBxYMZ/Rn+zgNChlrNqyiDDqCKNMQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=FzTxnx3G; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=wP9g6SRf; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="FzTxnx3G"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="wP9g6SRf" Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53D371Pd010269; Sun, 13 Apr 2025 04:05:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=vsAeNcwlHMxYfkhNJfjCs/p9nIgvOmgvZgNkgdYX8CA=; b= FzTxnx3GRE3vsaRhrOqstyMkdhbsdo7ZscfMbNN64XLcC4JuGw/faYLJSsgyLy3T RsCkIIw+8D3JfE0QULEEVW1OlxZykOF9eCoQdQTpoDuJSh8CW6yBa4TVzs2QGLHs DFjfjx7oDOycYwzRQY9OYx6iOQSFrIvX/G0wP/E1k2P7T7Jkp1A6TuBtqxh1r8r4 tdlAtD2vOMRJsW7JOScyMa//NpSppJD8FDdPx4p/JwGzswkiVb9hvESWF7zo+WZz hToNqFnwKtnjenqvj67s4D+CDTKwKIrE6sTtmK/EYN/V3SnIHYOKgIe77wLBMtAI uqFl8xgHKks/zxgPR0qeHw== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 46052wr1fq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 13 Apr 2025 04:05:12 +0000 (GMT) Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53D3UXYK019917; Sun, 13 Apr 2025 04:05:11 GMT Received: from sa9pr02cu001.outbound.protection.outlook.com (mail-southcentralusazlp17011027.outbound.protection.outlook.com [40.93.14.27]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 45yem6sdhd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 13 Apr 2025 04:05:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AXjFsECE6cMjXdm0MtxrnnZ8qaqKQnFpS/c+QD/BQeM6E0/ZZJAroJTPx0KSkQLfffRCETqDG6F5D+WtmDn+w5wJYgmGuqTCexaunBoqjACeK9VEkgcISmJrDcA6URFo03GNGLiRH1V6NTGspL+Kdynni0VfdtRZrF0xxG/W1m5m3iwHUEExk9nnnkw/P14hVY1ctqZ31XIrdosHh/3IzKVcFeZf3mYc54n9KygxvhkJ2W4U1DKobn4Tmhdme5w476Cd8Xo6Yh/Y4bgyiuhOL8mGyYQF/zkzDy9lXc8XrtzKLTHzYQ9TFVHmYBdEv6TR5FwL3IZXr3bCCMsOEqFmOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vsAeNcwlHMxYfkhNJfjCs/p9nIgvOmgvZgNkgdYX8CA=; b=OZArD12q6p22OoMiXvQsVpYKB7jq6HIXE81N1eNXgBpXYa4sUlXyrHEom/IVOlNciOn1P6Sn0Af95Gq+r0uWI4K5/jyYvk+dIuSOCGGEYB2tsjpUrhzcT+KKWxYTYQ5LK74S8QEwjjTMR9Tkl6pAFcpP+xE+7e4/0yc+aL/U0ZNB3gUcfEa520ueyws5xTXZSI2HmxbjRoJc5aDra7VUYZ2t60TjygUPAVyhH5Ll+UuY8blPSVGkPJuZ4nzEL62zmt8wE+HRN85EM7EqdDUZuvS9v44E96ROoB2q262gF9U5uDQOiSYDxPwdxZQzl4I/RI1p466bD3nJzPoK4XTe9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vsAeNcwlHMxYfkhNJfjCs/p9nIgvOmgvZgNkgdYX8CA=; b=wP9g6SRffryaVRYFvhjqwGlLsA9DmCkDgQQQtIHE0SFDfMsWUKIxtmtA7HKEQjxg/vIOYHwagDu9cOZvspCZ5gSpCubFAy19s6EKKu2sdX9HG++7B9TCHK9b+mP3UVz9/XHaUP/J/1tU/ZJJSeMTE/hBJrgXSOdeT5KtjiI5m6A= Received: from CY8PR10MB7243.namprd10.prod.outlook.com (2603:10b6:930:7c::10) by CH2PR10MB4120.namprd10.prod.outlook.com (2603:10b6:610:7d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.32; Sun, 13 Apr 2025 04:05:07 +0000 Received: from CY8PR10MB7243.namprd10.prod.outlook.com ([fe80::b779:d0be:9e3a:34f0]) by CY8PR10MB7243.namprd10.prod.outlook.com ([fe80::b779:d0be:9e3a:34f0%5]) with mapi id 15.20.8632.030; Sun, 13 Apr 2025 04:05:06 +0000 From: Mike Christie To: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Cc: Mike Christie Subject: [PATCH 2/2] target: Move delayed/ordered tracking to per cpu Date: Sat, 12 Apr 2025 22:59:51 -0500 Message-ID: <20250413040500.20954-3-michael.christie@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250413040500.20954-1-michael.christie@oracle.com> References: <20250413040500.20954-1-michael.christie@oracle.com> X-ClientProxiedBy: DS7PR05CA0020.namprd05.prod.outlook.com (2603:10b6:5:3b9::25) To CY8PR10MB7243.namprd10.prod.outlook.com (2603:10b6:930:7c::10) Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY8PR10MB7243:EE_|CH2PR10MB4120:EE_ X-MS-Office365-Filtering-Correlation-Id: 68cd90c0-6124-4f92-f660-08dd7a40602d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: iX+NGN5J902+J/7Irsb66/OM+NAOSMjAvqSZJ/EIx5c9I7LKlvjWbXLOaJRaSNryCHjRJDBAQHG8ffh5Y6kiTBnm4dsxl4Ay9I2hAtC4q/fvkLkpDs/yscCApXpaCvmUPIj7znqUJNLVxWS0usjE8Wo9B8bqzxAtjB+b8kU2oIf4IaB3kaKPl3hZBeTBLWl6rXcw/pPEOw4VbgkA9a3Gj3QMP3NJZ47tU9WYP8u0cNjrK9Pl97F7JJT8aPOdWPqPFJCYNoOrU1HZR/sUPEGgR9hnYBbGgHLNzcX2HXbwpLwmPI3H6kxwct0LzON3ZOcNVjYfmHp4q6sl/mtoD7pbc8DfrBL8qOAgpl7tlfn4MS52fpmZIUikUE3k+6ywwVW2cX/cQM4XV9Zr8pQmGk7rkm/CMOvY/jta46B2g60Qf6RCJNgXK1PB3Lv9iU6d1g6LCYbaiEG8KptcMAiTW/7w/RytojuX2+gNzCHDfd5FPoV8IG2o8BFfBnz4x+b78ZJ5ZWb4d8jRZeJUvafbTB4sGati+Lk37MzAQCaODJlFjYvVHZE3qNc0fsdUcymhr6T8IC162MqHFBSZjJJhe6CrxgjzFCEqtjxWFqVtd4t0QhesoiDP47Ng+cTYYcsXZp8qSfy+IPhbu95QOqboQUF8U+7i/cCn9QaArPxBNtGZk2GVZOIKIsosSX53h+Mkj7vKoP0b8O27vMG2sIM/XjYPAyrWA2HvkWR0yK9nFX1QlEYTPsgz9bI/hZrQVIfiipUvcQY5F2+HxwiJD/E4SjUCoYxKts7W1xpuI3spyFsPN37bkosEGz1D36MPtFrPHQ7lGcx6jFVaXNsZBtrWBKU33/ZE03c7+XG0FfkXFQivFqGtJ44yXTqYlWUZZqwYlzh5iwY8fMBOWtPkaqSmX6MbAfiEm81csbp+/aULzxeC/6LsaVpundNW4PIkXrN1eNB59PvtvnrEFoIXCoMurh0wNBwAJGY7J4tnppw3HwXbw9O539IlOWQT8Y1BBG/YuscI4fJonMygtwY9ukfMrv7U4op/xR33WCCL7Om1wAhVQ0umCMp0BWYcMsaMGS3cvZf0UulRUhBc6Nqchj59nfT0WFJj9u0JJUUeu8MmibHUl046H5VRHqPxeJqpCorGvlRhcDdV3cV1BviAWAXD/vm7gofQVBZwKidZqRlFZ5Q3G/XB/yptYDeAE3LNLgMrhy8R4z34AiX+b7QrMW9zOCzELG7S+gLy76DaYzADee6y1CGmcWVur1Cs52YNrK6olIQSB8S+ZtIKp1rQC0zppughb36ctAC3TsLtd/HtPzh9PtXcvEvMAyVUkixdctoUlf70+2LJna5jtfGXy/q8TjgkFuwzNJUUoV99UnSOFWKN3Z0mUl8yK7e9R7Cz59wo0qjeomPmEJP2eIy1S4eN3F53OPKcdtmHlvoFPnvNcgTJ9Sg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY8PR10MB7243.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zF/ogP0T+x3E96PK0wCyXdBC1Oat3Jap5crplZ7gtuJy0RWfa0We5lFdIimJBDDEQy6qxbQf7kZc40r0xJTzSIXuvrbh+/8MEKP8+kg+e+twdMLP3e74VJWDwqelwM12t+alGmltvZp/aO6HKsLa8RQCf/DcH+QTVSXIAFj5GlUEA9/3JCry+zknEmcW4Vyhu/B/pusC5E9S6yZX8FXbZ8mTVXLLjPxQk6JhHu1JreVMgNpMDh1IJHPa1GCaJw+2TkAcgeRvtwHQfMBJBvtB9MpaHS/5bkVRCBqirkUFTbYuVMtpAUhWxpU05BG0bCqGUv7YMm1JbGqiZ6cDGvi/XI6+PSR53xCtJW/ctYuXHBx6jtXxJgeu8vj7nlh2B5JIUNrnLA4nSXTs65jPX5Mm9wvwUbbha3yWFgbNKcPHP2OEJ+ePOI+/q5wSzMuNWPYRsWBnATj0AUzj99hQPNkkgNpZf+bDl1JFSW5sdz8L9P8iOdjwjcX4SqLPew5VHQuuWVTyKsGVkP4ykj2npDhZoIROkHKkVNo0T8kINf1mOtf4aTq4kiqeJIZhNL+h+Dd7UB8a5oE2wM5+mPUITa4oLntgqJviPKcz8uW2pJz+AVBaVxcQyYapmeNX+/6YgIeMwNqechHl72Ww0Vu+4Zrp23vJd4XJoMi7pyRH6fz+HKiz4rBr6MLCoL1p4IfjGjHNeWwfrGJypeJ4ifl/L2IxnRc0LXZPplIDEM3weUXeUIN4Xbp6G/CFlUZQmkakaVtphVQWyxltHRXGKVgqJGnzk7yQzEjjP52Ej+U48nqvAliYBxrHfk8sc4GkpzN24omaby6E3slokPHTR+kJ8SrBbxGoivIWvmUms2ZH4+LmKL1qw4+HiKUMSYJHEzn/z8lPGgfcTXibX/Y6ThDks0T6a5zMgOIZ3C6NwakZhUXDkPLaBXtY5/taQqq/smTuX8xMHrphezmnqUH52b0N9jdBsbfDLXFSbg4A0OOc42TsNLlIVPC/IcbOPKhzZBGCKDdYjnBf11NoeKB3buFr+IJZSO0VitERCNh7TJGa6zp31uY6hHhOoyj+V/5kCP4XYuxiZXJUI/4PVM7YcHKTflaRycT6H6wKjaZdRwlMdCR4KOZ4aJyL0LWRFzTv4yHJrveOLkW4daU07PeQ/9o6OUPaj3+ltVLzLUoTcL0j3psR0LXdO05WM8nIzH/BxpQwZ7hgfF7McFyZSJwre7pkn3gPU7d7d+e43xVw4Pj2mi73+GwdUo/8uxgQq8y++v6/U9dw4bA/3IllU2JDfeNSI0mSgcCNxj774037SGJRrobpP5gZjx0segrQOK8ayW8pIyrzXFqlOLdArPk9//L2av24OAmQlhlGS8uEmgFuNqs17C0HPUJMdWz3+GKnRKoYJ8Qz6SDqvBagepon7at+V6Q77ZWfzdVkeHeqi3aazA4UQeO0BUuSMwBa0msEKW7e57iVDCs47QQtlzbEA9vRed91oWKVWHn3GvIc4f23w9aDzlB5ETayokdukDttBLDZXlvUWpIv1S2487M0+B1A/6Xo5JsKhnG4/xvgE/hmXwGFnzjixsjeY69L1WPAbqeMrA8TGEojGW7eWIC67g9E0t8AQQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: QDkKQov1bZ5nF9Tzgyo9bFG0DnP49Y6cQYTjCis8XX7j/z0EMxSnYXagb0ymwwwD/WRCSmXgSJYnykV6HJctdF5UrGr3yNyx2EMd/58GcMJBinyGifhnbiV6b0dmewIQcucC9nb3LI9gUxWWLBsJG9SsBn4HVw3qsMUhFH7psWIFtHf/94xx2MiqAbC6bhmoi1Im3iBdSByH2kkvwEsuj3NFNBfCVrBSb/HjdzsHlRjZaWmoqRfstafaKU9If4xnyCVv7ZiMCdt/Cj5gF60vU9nPaBXqNW4Af8cAW2Qel6Z8vb7DDJGag7v9uTxHMn1eGucISdEVb2AGpyFz2pKYuLu+oAxkEpqzbZ+XjAY7s7vtn7OlbERhz5DhPuBw5Esd97ZKoctu8M/+tUmPqmnHoPPkEaINqa6VWm5oB+1TXRCWj5Uq07mxPNApQN34YfmE5ZeDWWvl4M08pnUVYuX/x0nBtGTcvTyBKMZ3wcUaqUdM0/1QAvMtHB06DTXghmSIpnIT44CQg3FFuhm1dWWhQh49sigfDxv3XNFnD5asM7hPMqSuGkPgyfmmfjwbQxywqi9LIM3T00qfApw8VJyCyqdvAoNFe+pwur/B/ZMXYCc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 68cd90c0-6124-4f92-f660-08dd7a40602d X-MS-Exchange-CrossTenant-AuthSource: CY8PR10MB7243.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Apr 2025 04:05:06.7923 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QE0um4akv3KaHRrnrpwKiqp2c9s3CJRIDo9xY1H+LUyfPkgG6BHxQLkf/LcIYD5HMaVVMbe8XhbgrB2T9cNmxRja5rALhqDdV+qM3tjNl2U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR10MB4120 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-13_02,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 phishscore=0 adultscore=0 malwarescore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504130028 X-Proofpoint-ORIG-GUID: f-7K7bi9nar66VrTuglmfwQCGodCGex3 X-Proofpoint-GUID: f-7K7bi9nar66VrTuglmfwQCGodCGex3 The atomic use from the delayed/ordered tracking is causing perf issues when using higher perf backend devices and multiple queues. This moves the values to a per cpu counter. Combined with the per cpu stats patch, this improves IOPS by up to 33% for 8K IOS when using 4 or more queues from the initiator. Signed-off-by: Mike Christie --- drivers/target/target_core_device.c | 20 +++++ drivers/target/target_core_transport.c | 119 +++++++++++++------------ include/target/target_core_base.h | 4 +- 3 files changed, 83 insertions(+), 60 deletions(-) diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c index 39aad464c0bf..7bb711b24c0d 100644 --- a/drivers/target/target_core_device.c +++ b/drivers/target/target_core_device.c @@ -700,6 +700,18 @@ static void scsi_dump_inquiry(struct se_device *dev) pr_debug(" Type: %s ", scsi_device_type(device_type)); } +static void target_non_ordered_release(struct percpu_ref *ref) +{ + struct se_device *dev = container_of(ref, struct se_device, + non_ordered); + unsigned long flags; + + spin_lock_irqsave(&dev->delayed_cmd_lock, flags); + if (!list_empty(&dev->delayed_cmd_list)) + schedule_work(&dev->delayed_cmd_work); + spin_unlock_irqrestore(&dev->delayed_cmd_lock, flags); +} + struct se_device *target_alloc_device(struct se_hba *hba, const char *name) { struct se_device *dev; @@ -730,6 +742,9 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) INIT_WORK(&q->sq.work, target_queued_submit_work); } + if (percpu_ref_init(&dev->non_ordered, target_non_ordered_release, + PERCPU_REF_ALLOW_REINIT, GFP_KERNEL)) + goto free_queues; dev->se_hba = hba; dev->transport = hba->backend->ops; @@ -816,6 +831,8 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) return dev; +free_queues: + kfree(dev->queues); free_stats: free_percpu(dev->stats); free_device: @@ -1010,6 +1027,9 @@ void target_free_device(struct se_device *dev) WARN_ON(!list_empty(&dev->dev_sep_list)); + percpu_ref_exit(&dev->non_ordered); + cancel_work_sync(&dev->delayed_cmd_work); + if (target_dev_configured(dev)) { dev->transport->destroy_device(dev); diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index 05d29201b730..0a76bdfe5528 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -2213,6 +2213,7 @@ static int target_write_prot_action(struct se_cmd *cmd) static bool target_handle_task_attr(struct se_cmd *cmd) { struct se_device *dev = cmd->se_dev; + unsigned long flags; if (dev->transport_flags & TRANSPORT_FLAG_PASSTHROUGH) return false; @@ -2225,13 +2226,10 @@ static bool target_handle_task_attr(struct se_cmd *cmd) */ switch (cmd->sam_task_attr) { case TCM_HEAD_TAG: - atomic_inc_mb(&dev->non_ordered); pr_debug("Added HEAD_OF_QUEUE for CDB: 0x%02x\n", cmd->t_task_cdb[0]); return false; case TCM_ORDERED_TAG: - atomic_inc_mb(&dev->delayed_cmd_count); - pr_debug("Added ORDERED for CDB: 0x%02x to ordered list\n", cmd->t_task_cdb[0]); break; @@ -2239,29 +2237,29 @@ static bool target_handle_task_attr(struct se_cmd *cmd) /* * For SIMPLE and UNTAGGED Task Attribute commands */ - atomic_inc_mb(&dev->non_ordered); - - if (atomic_read(&dev->delayed_cmd_count) == 0) +retry: + if (percpu_ref_tryget_live(&dev->non_ordered)) return false; + break; } - if (cmd->sam_task_attr != TCM_ORDERED_TAG) { - atomic_inc_mb(&dev->delayed_cmd_count); - /* - * We will account for this when we dequeue from the delayed - * list. - */ - atomic_dec_mb(&dev->non_ordered); + spin_lock_irqsave(&dev->delayed_cmd_lock, flags); + if (cmd->sam_task_attr == TCM_SIMPLE_TAG && + !percpu_ref_is_dying(&dev->non_ordered)) { + spin_unlock_irqrestore(&dev->delayed_cmd_lock, flags); + /* We raced with the last ordered completion so retry. */ + goto retry; + } else if (!percpu_ref_is_dying(&dev->non_ordered)) { + percpu_ref_kill(&dev->non_ordered); } - spin_lock_irq(&cmd->t_state_lock); + spin_lock(&cmd->t_state_lock); cmd->transport_state &= ~CMD_T_SENT; - spin_unlock_irq(&cmd->t_state_lock); + spin_unlock(&cmd->t_state_lock); - spin_lock(&dev->delayed_cmd_lock); list_add_tail(&cmd->se_delayed_node, &dev->delayed_cmd_list); - spin_unlock(&dev->delayed_cmd_lock); + spin_unlock_irqrestore(&dev->delayed_cmd_lock, flags); pr_debug("Added CDB: 0x%02x Task Attr: 0x%02x to delayed CMD listn", cmd->t_task_cdb[0], cmd->sam_task_attr); @@ -2313,41 +2311,52 @@ void target_do_delayed_work(struct work_struct *work) while (!dev->ordered_sync_in_progress) { struct se_cmd *cmd; - if (list_empty(&dev->delayed_cmd_list)) + /* + * We can be woken up early/late due to races or the + * extra wake up we do when adding commands to the list. + * We check for both cases here. + */ + if (list_empty(&dev->delayed_cmd_list) || + !percpu_ref_is_zero(&dev->non_ordered)) break; cmd = list_entry(dev->delayed_cmd_list.next, struct se_cmd, se_delayed_node); + cmd->se_cmd_flags |= SCF_TASK_ORDERED_SYNC; + cmd->transport_state |= CMD_T_SENT; - if (cmd->sam_task_attr == TCM_ORDERED_TAG) { - /* - * Check if we started with: - * [ordered] [simple] [ordered] - * and we are now at the last ordered so we have to wait - * for the simple cmd. - */ - if (atomic_read(&dev->non_ordered) > 0) - break; - - dev->ordered_sync_in_progress = true; - } + dev->ordered_sync_in_progress = true; list_del(&cmd->se_delayed_node); - atomic_dec_mb(&dev->delayed_cmd_count); spin_unlock(&dev->delayed_cmd_lock); - if (cmd->sam_task_attr != TCM_ORDERED_TAG) - atomic_inc_mb(&dev->non_ordered); - - cmd->transport_state |= CMD_T_SENT; - __target_execute_cmd(cmd, true); - spin_lock(&dev->delayed_cmd_lock); } spin_unlock(&dev->delayed_cmd_lock); } +static void transport_complete_ordered_sync(struct se_cmd *cmd) +{ + struct se_device *dev = cmd->se_dev; + unsigned long flags; + + spin_lock_irqsave(&dev->delayed_cmd_lock, flags); + dev->dev_cur_ordered_id++; + + pr_debug("Incremented dev_cur_ordered_id: %u for type %d\n", + dev->dev_cur_ordered_id, cmd->sam_task_attr); + + dev->ordered_sync_in_progress = false; + + if (list_empty(&dev->delayed_cmd_list)) + percpu_ref_resurrect(&dev->non_ordered); + else + schedule_work(&dev->delayed_cmd_work); + + spin_unlock_irqrestore(&dev->delayed_cmd_lock, flags); +} + /* * Called from I/O completion to determine which dormant/delayed * and ordered cmds need to have their tasks added to the execution queue. @@ -2360,30 +2369,24 @@ static void transport_complete_task_attr(struct se_cmd *cmd) return; if (!(cmd->se_cmd_flags & SCF_TASK_ATTR_SET)) - goto restart; - - if (cmd->sam_task_attr == TCM_SIMPLE_TAG) { - atomic_dec_mb(&dev->non_ordered); - dev->dev_cur_ordered_id++; - } else if (cmd->sam_task_attr == TCM_HEAD_TAG) { - atomic_dec_mb(&dev->non_ordered); - dev->dev_cur_ordered_id++; - pr_debug("Incremented dev_cur_ordered_id: %u for HEAD_OF_QUEUE\n", - dev->dev_cur_ordered_id); - } else if (cmd->sam_task_attr == TCM_ORDERED_TAG) { - spin_lock(&dev->delayed_cmd_lock); - dev->ordered_sync_in_progress = false; - spin_unlock(&dev->delayed_cmd_lock); + return; - dev->dev_cur_ordered_id++; - pr_debug("Incremented dev_cur_ordered_id: %u for ORDERED\n", - dev->dev_cur_ordered_id); - } cmd->se_cmd_flags &= ~SCF_TASK_ATTR_SET; -restart: - if (atomic_read(&dev->delayed_cmd_count) > 0) - schedule_work(&dev->delayed_cmd_work); + if (cmd->se_cmd_flags & SCF_TASK_ORDERED_SYNC) { + transport_complete_ordered_sync(cmd); + return; + } + + switch (cmd->sam_task_attr) { + case TCM_SIMPLE_TAG: + percpu_ref_put(&dev->non_ordered); + break; + case TCM_ORDERED_TAG: + /* All ordered should have been executed as sync */ + WARN_ON(1); + break; + } } static void transport_complete_qf(struct se_cmd *cmd) diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index 05e3673607b8..a52d4967c0d3 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -157,6 +157,7 @@ enum se_cmd_flags_table { SCF_USE_CPUID = (1 << 16), SCF_TASK_ATTR_SET = (1 << 17), SCF_TREAT_READ_AS_NORMAL = (1 << 18), + SCF_TASK_ORDERED_SYNC = (1 << 19), }; /* @@ -833,9 +834,8 @@ struct se_device { atomic_long_t aborts_no_task; struct se_dev_io_stats __percpu *stats; /* Active commands on this virtual SE device */ - atomic_t non_ordered; + struct percpu_ref non_ordered; bool ordered_sync_in_progress; - atomic_t delayed_cmd_count; atomic_t dev_qf_count; u32 export_count; spinlock_t delayed_cmd_lock;