From patchwork Wed May 3 14:51:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 9709953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CA56060362 for ; Wed, 3 May 2017 14:55:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9BA128628 for ; Wed, 3 May 2017 14:55:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ADA9228631; Wed, 3 May 2017 14:55:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED,DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 97E9B28628 for ; Wed, 3 May 2017 14:55:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LYLJgaW1pXMSMdVQHr7DlHQ+WsqCTIIb4ppJJOE/IWY=; b=iBCaki0yo0k6bv 8txW+87QiG/aPBUsbRb7yxmgCttyzwrzy4VPCzoCblg4nt+dX57VyQ0zVQhwvd9/pXO7J31obrcrd KRHpB5tJ5yjyo3ZUkAHhI1uWt/YzyoYR7b7QW3983djWzSWVMOP14wC6j6b9ffo9AHxn7JfhKMtQ0 Bx/WE3IcyUI3nMHJnDAZ8OUR2E0kEkE3YjbmW/l0kUkzxe6y42sRSSxZuC6CAHGDacPmqA50mm29G e9mnXdj9Bv1RL7mBH/5nw7lEv0eUlJdcQqBoF8v+J4+LNHOkId/2YUQlvlOmgAKFwXvZVfSt/QeWx JmPBYj5cnQxbk9zpKo6w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1d5vgz-0004Ox-1O; Wed, 03 May 2017 14:55:49 +0000 Received: from mail-bn3nam01on0045.outbound.protection.outlook.com ([104.47.33.45] helo=NAM01-BN3-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1d5veh-0002Iy-Jt for linux-arm-kernel@lists.infradead.org; Wed, 03 May 2017 14:53:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=b6gqZSKFuouWpBvaOG42hsqLiZ1bN/ehNeYCh23P4cU=; b=PfE1r7AydbLZazB+aRXdklzy/hWQMqZrO6YIzIby4csSTGYm5/KgHeIGhqR32odLYZr7fx4OrX6Hun7VbAL0xiy6CwwWPTZKEI8d45xX4Y9i7elPtFfewZlHL9/uNQy+O4l3GpleAvz78r4pPQraGtD6+fTqWo6pMQ4OlQnOCgY= Authentication-Results: arm.com; dkim=none (message not signed) header.d=none; arm.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from localhost (176.59.53.57) by BN3PR0701MB1268.namprd07.prod.outlook.com (10.160.118.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1061.12; Wed, 3 May 2017 14:53:04 +0000 From: Yury Norov To: Will Deacon , Peter Zijlstra , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH 3/3] arm64/locking: qspinlocks and qrwlocks support Date: Wed, 3 May 2017 17:51:41 +0300 Message-Id: <20170503145141.4966-4-ynorov@caviumnetworks.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170503145141.4966-1-ynorov@caviumnetworks.com> References: <20170503145141.4966-1-ynorov@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [176.59.53.57] X-ClientProxiedBy: VI1PR0602CA0015.eurprd06.prod.outlook.com (10.175.26.153) To BN3PR0701MB1268.namprd07.prod.outlook.com (10.160.118.142) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e7a91c5b-15a5-4756-a7fc-08d492341aad X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:BN3PR0701MB1268; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1268; 3:Wnqyd7oscOYM/7dEL66c0csRdZohj+RO4KQQGSc5kWDdPfNPTo5l4JXjj4Cmke0Z8Hcbxdqkr82RU6Jsch4qCk0CqaxTH6QlKMYorD4m41MTA2nlLNPC2LKPvpVOHfPE7tTGuM9ZEnE/vMXDWq4M7Abx0G8M4NAR3FbcjM8aUc4xEp4IW6t+OmFxwoU3oDv6tL2ReY4k8LZ8l+k8gPxIdIPuJDm7pwjlyooFvCfjDBxtHMlC+qCDzsoC++DL8iPHKTJ9azQV28EHzmWTI6HA2orqrOfmp3Z0Z2XivW6o2kJi97iq8y3x4mUph/kE2INsPJ9HAiiNTd8IzEnph6Jysw==; 25:H6PxfvCFsbawLOqOKSrZHBVGn/q130P/xkc2/PPl3DuFpuxqCBO+qzlWiZvQ26WUyMtvqGIWcJUFF36inap1GchVIwPr7Hev+j9v4W7KM3x0/2q5Yq8ti84FY8w7laeVe0JsA4UppfGb27qO4SHayFRIK60m4s9WO13qZlKS1/Ns/7zt7VNGUcwsh4HXfPUvrBa+JQ7+KyUAgqq0lyzO4QvRQeL9QSedIATuFAEa1iLgdGvxOFZhr+j9j0L01V4a6vuaWAnPHR1C5OkLXSTLa0PMBkiSziIu69jvtFeG+pOcUwSXbMnQAxaCht0v//TU7GZLxszhvIeWQNFgshpNofmUBFpoKwIIvhn+6gj9aFeC7p1szbK53VMfbFOKCps9BmHDAi0xs0CR7h/gqDhOo5h9PeUMIXbZ/fqleARZEOPm8KoujcmR+2pd1qRsmGMandRF7p5C+rjI/3pfNznUjg== X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1268; 31:NakcYaETmrViOiGcjb9YoH6x5LAflqRIScdp5ILZsGgxmpdaPTnBCBUgBn1ooYDjCiaTWwgZ2b8ByCrq8Etk91FSkk8EzP+oNaRk8o5LxQk9FNtJWfRSGhS4Tu4wxTfcqb12zFhLXMJm28LezUvHodrm+iKLjo8mA6mmRTMqe802veezzNNQ8v2o69c91YJlH26s3Sv7pn82fQgaUXm9OgfZSBjjAjkKKeLknRoY0zl2NvYrtyKIG1k4oKDPMLWL; 20:V9VGxKYjIZvgOmMT/0bZK/Z0G84/lAm3sY3qVwcoyW0E2+hkCV54Yytu2qY/KwPdYhLCiV496gptwSIHJCN1dXMVrbvy0JcUPr4nCB5kfXfceR1UfK3jtNkCkwlKECMaiErIdb4cGHOicISzU5LLGMpFB9rlTrkbG+H392pRV8rKM7V+4GARrozMUiPrxbjcEYbtvceKuU+Ngq2b8dlU5+HcegxexjqjvfNcz/pBeeOFIeY3BSxInMCIhgJekY6GvU99QiUs3mbNAUihr8tcXzrJCIcUPXo2CRcO3FWSDcrqMi6Ut6m6sHUc7z+3LQekHAPVHKouM6wYGVqrVM7l/OV869eeWshBgrmqFpPGC+mnes5rQQTPghfi9vsmns+xZlKLd6PQL+asN+bQo5hQV9hKGY3ftLuDAmnr6CcZZAkl5HBHAMF0UCJWtUESSChwU9o54RbbIUqE+0l3qqG1wPkm5EXBFmX/hdgpq3oDf8YT9THNyeRdqSbGBTHPH7niw2OrbndwKePMBAouwsXvJDmShLptM/NBxSpQKc2Klo+to6ekuOa2F+WWJA+1dhD/9GBCAx4lzNt2+oST3rrMaC4zXLWrdybc0UJfc4x3w8E= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(190756311086443); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(6041248)(20161123555025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(6072148); SRVR:BN3PR0701MB1268; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1268; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1268; 4:GZEti4y+KLQTWAUTVXnM2fLXLIpBB/fxOlwyUp12SudxyRA+CjbLcQgyW9c1NDroE6bgfBSP+iVWpLBvF9/X3riFGXV2ndDBgyS19mHKUxsMdP1+5LJmnWWjz2PXOe8kCQfYQTy0kL++mBzUI2H7bXBJHr8OQQVMwsu4DYGRicnkE/MuAj0VLUpg6a1cJ04EBdjW59jKjdVzpqUnT9U509ARmmJKwqEIe/8Zs7/CM0nneko5dr/x7AH9yeANRQCMehF0o2c8MaClW2T2jTEM0gDDAWTUomx87WssF9JtnFffcawdxSR24mEWJb209IT34HHtg+Iz0LFEytWEj2wqxVZbDcNGv6lNwmzCDNPjHPbOU4uKTEBR8xw7C9zGNrAF46g1J4DBnV2/32zOhAff2nCO+7IvDAHQLgpEWEua3r+/NtJaLfI55uqXdZmsgyUNb7ocjZVfTnJY1pIIOoaUcOuwEkdNrUAQZbZ0tKnhPExQFC3cGCFJRr7g02Nnndsr0C2li09F28dKAz+x2WCeNzK9n4YcWEVo3sU1WuKrOt8xRhykw283V7fyxHVzXlEiVOsn+OjDz0klV71hkujmpmxNn9uuA6d1/9MotCK14Irm3Ch7o+ypOB3avFHgLLObB1Bii++Kn4akIGLuoqE/QR54mywxo66h5r/D82Yqc7JO4vs+RHpaEisOgTGXVp4W0KfOYY/0Z008CWkEqkfoHj4zc6EKUJVZ0+Gii8eq3ex2eJCwKhz4aLOP4jjYdDV6iff2ZNIMHxh05P5JdITYG0G0z+EN3cNfUknBk/rlbdqCgHqTlJO9RBND2jIdmKt2 X-Forefront-PRVS: 029651C7A1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(39850400002)(39450400003)(39840400002)(39400400002)(2950100002)(7416002)(5660300001)(25786009)(42882006)(1076002)(6666003)(189998001)(305945005)(478600001)(5003940100001)(4326008)(36756003)(575784001)(6116002)(50986999)(2906002)(48376002)(7736002)(50466002)(3846002)(33646002)(53936002)(47776003)(66066001)(6486002)(38730400002)(8676002)(50226002)(54906002)(6496005)(81166006)(76176999)(42186005); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1268; H:localhost; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1268; 23:qg+pw5DjF6DFw1r/JCOO/XWsFkUIhqFVBVc84Qx?= =?us-ascii?Q?+hjq2eoF9SnfOLuGrzT7qRsQ2DOVmM7viI/opZswlpk3lT581XZTQZAD6fQS?= =?us-ascii?Q?wgnmbYikr8rQNXqGoE5CWg1rSbUtFWRydf43zXq18kLsjKmDIJhp5859DJAb?= =?us-ascii?Q?MVf/+ENrsIPazpv+UW7rSW16holj9fYPyou5wi/8x55qCuM8HVT9DcXZLQAU?= =?us-ascii?Q?Ze0ychQSS5Wlz3Y/vlSpSECDFXwaMp5yAiWtNHD9/5scgzMihX0eup4MmPs6?= =?us-ascii?Q?xvxYzvKhPqlflhVJtcgmRmPswgT+ca/vInjMc3jlG3C24p2b3r1ktnq5mAgf?= =?us-ascii?Q?oUpGd21y1XzDJUED80MzG6e8xNJ+tVrRicYhjwgQ2QBAsZCjh2DvG8xCpLP5?= =?us-ascii?Q?Bsaw7qgyYcsyzdwbFfUFYFlcX5K7QYO0FKvRckHTIKTjdF6vXpVrgGeGdEtD?= =?us-ascii?Q?vM5gZXMQjuwO29yMXxBKr5LfjzFpIrWbwdEopYI2XP0RVxxKZNwupDJ/E3fx?= =?us-ascii?Q?M/k2lAbT+MaJxmcAua+LxO3n+1I8VX08pBorF/oi3W/ct3xp8wOpnIsKj1SH?= =?us-ascii?Q?2rWF6QX4SPs2FDss4UvxwA/2UrX/4z2Tnt8sUHTpLddMTCa5gACBx6m8noSP?= =?us-ascii?Q?DtStal9uVpyr1RrQezRgW00EKzJc8B5hXBOInX+NBlAz8aeENRph1fJqiEOq?= =?us-ascii?Q?IqzyQPfq0fIXXv+9swYodTbMzdG1zCUU7sISWgisOm1yrnD7/Po1EG/UvQKJ?= =?us-ascii?Q?Vs5hRU/XpzxkW0NVHeOQWNJ9cXA4YGgUcP3JA8MOMfrBjGt6rTeUJEnVinT9?= =?us-ascii?Q?c3fIOTBbkpPcN93UE2kDuTzz5pKJuLt2teRyN0iMee4Wo2mnriEx7M3kMj1R?= =?us-ascii?Q?4GPHbGdapTI6oOxQlSRQsXfk/2U7JagRu8BDyeGmaudruYcSynq4/ImzDWw+?= =?us-ascii?Q?wwQs3AVvkUnvyXF1Fp2VkZIsGIZPDJLoUsc64lpfp3RVY0anViYRRnoBs+RU?= =?us-ascii?Q?2746Rmy+qcihxCics9z5VTiHgA5lez/KOzstmH+UHiOmCtw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1268; 6:vpSdyy/NUAyv7tghLFsijGYJRr/9CR7sIyVMRysVyMBYxQP+5YnCR3kZgtBjuJ+6dK03ALaiezg9YJKhhJk0nww9ilp9NywvRmEkAWzexviDJi+orEtVa8rSb0BCvXOsjPZvlH81lS5rGH98ZAdd2gQSHsXTtP6O8d7lxEZxArgfjTxskWONjYOj1u61qCBv7E0puYVWe652WfH0BySnxe0hoSmwviuH4HtqjEkHLYCzlJRNiJi8OSz59EcAlefE4GvpsPvT3YeKRIG7jvOjigMT5BG43kaRkRWhEaxNwa95/X9RNUxMOdU22avIVj9TLOgyqHa9RYw7WB6Hh9AE7fjVjqfQ009k322YJV4sDjLs3hcaPxSuGyiSovUYZ0l50TVKOkQo25wX3s1xMPlA1CBuoImWBaF7mMAoUJPf4/6dbcBVLnCbN7cT/5m48va6lDp3GuDuR847CkU/MI/n4+cXKiAbApyeXnHiK7bBLISZK6e1K9F2uW3mScVU3Sv0q6jwuXJ4Bag7Lu+2VV+8Dw==; 5:9dj5bjqzqsCBn4ZXW+JHxL82zrQgsh7lrkCY7dSDgAgRuK6SCgv8XPRR8g1Vk3BeNRwYTl92LD87p53EdWGmQIRtdgji+8ZaoQ3Swfs4v1D0M07nuijpTCFJFQ30Fxjk9Reh5/xJlhgm3aMFnl0Yyw==; 24:dcPkY+5wVRng5ACERph80+LcAJqx0DoU3s+y0rkq4Rs+Ir7PdqbcZBMLki/jJ5qgFnz5kJAssppVcauU2JhOuI0hZOdfz5UbnlM1tpjqTf8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1268; 7:6r0jmxk3va+eMQsWIh0GIJwBD3mIHyLJmIQNVy3TFqkANc1ynWCQ6OV8cfxhrefqWk6dBhI8BFW+m9jKscJIaseDDHphmm31VoX5wW9aYkQcTIT49T9tMdyucIl2PwlygcJ7uw7kHCjkYB845M3WxHn488rZ3hRXUeCA64FT4R69Xc7f49jvbCbBO1pHpwndDnoxLpvvsflv7RMCcNS5+TA0/5UD6I3wjJBCzZbMLIRrIboHSdZhDnW40LYhuvIKbM4Cr9NzslRsGNUeOoL+/1+kiIXdA/fLUm86dShtweNXveoikr/Kp8m0DNH2l7id3UaJggJOx+LxDLeyBUXJKw== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2017 14:53:04.5719 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1268 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170503_075327_789008_360E1A6E X-CRM114-Status: GOOD ( 19.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Andrew Pinski , Pan Xinhui , Arnd Bergmann , Catalin Marinas , Adam Wallis , Yury Norov , Jan Glauber , Ingo Molnar Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jan Glauber Ported from x86_64 with paravirtualization support removed. Signed-off-by: Jan Glauber Note. This patch removes protection from direct inclusion of arch/arm64/include/asm/spinlock_types.h. It's done because kernel/locking/qrwlock.c file does it thru the header include/asm-generic/qrwlock_types.h. Until now the only user of qrwlock.c was x86, and there's no such protection too. I'm not happy to remove the protection, but if it's OK for x86, it should be also OK for arm64. If not, I think we'd fix it for x86, and add the protection there too. Yury Signed-off-by: Yury Norov --- arch/arm64/Kconfig | 24 +++++++++++++++++++ arch/arm64/include/asm/qrwlock.h | 7 ++++++ arch/arm64/include/asm/qspinlock.h | 42 +++++++++++++++++++++++++++++++++ arch/arm64/include/asm/spinlock.h | 12 ++++++++++ arch/arm64/include/asm/spinlock_types.h | 14 ++++++++--- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/qspinlock.c | 34 ++++++++++++++++++++++++++ 7 files changed, 131 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/qrwlock.h create mode 100644 arch/arm64/include/asm/qspinlock.h create mode 100644 arch/arm64/kernel/qspinlock.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 22dbde97eefa..db24b3b3f3c6 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -25,6 +25,8 @@ config ARM64 select ARCH_WANT_COMPAT_IPC_PARSE_VERSION select ARCH_WANT_FRAME_POINTERS select ARCH_HAS_UBSAN_SANITIZE_ALL + select ARCH_USE_QUEUED_RWLOCKS if QUEUED_LOCKS + select ARCH_USE_QUEUED_SPINLOCKS if QUEUED_LOCKS select ARM_AMBA select ARM_ARCH_TIMER select ARM_GIC @@ -692,6 +694,28 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +choice + prompt "Locking type" + default TICKET_LOCKS + help + Choose between traditional ticked-based locking mechanism and + queued-based mechanism. + +config TICKET_LOCKS + bool "Ticket locks" + help + Ticked-based locking implementation for ARM64 + +config QUEUED_LOCKS + bool "Queued locks" + help + Queue-based locking mechanism. This option improves + locking performance in case of high-contended locking + on many-cpu machines. On low-cpu machines there is no + difference comparing to ticked-based locking. + +endchoice + source "mm/Kconfig" config SECCOMP diff --git a/arch/arm64/include/asm/qrwlock.h b/arch/arm64/include/asm/qrwlock.h new file mode 100644 index 000000000000..626f6ebfb52d --- /dev/null +++ b/arch/arm64/include/asm/qrwlock.h @@ -0,0 +1,7 @@ +#ifndef _ASM_ARM64_QRWLOCK_H +#define _ASM_ARM64_QRWLOCK_H + +#include +#include + +#endif /* _ASM_ARM64_QRWLOCK_H */ diff --git a/arch/arm64/include/asm/qspinlock.h b/arch/arm64/include/asm/qspinlock.h new file mode 100644 index 000000000000..09ef4f13f549 --- /dev/null +++ b/arch/arm64/include/asm/qspinlock.h @@ -0,0 +1,42 @@ +#ifndef _ASM_ARM64_QSPINLOCK_H +#define _ASM_ARM64_QSPINLOCK_H + +#include +#include + +extern void queued_spin_unlock_wait(struct qspinlock *lock); +#define queued_spin_unlock_wait queued_spin_unlock_wait + +#define queued_spin_unlock queued_spin_unlock +/** + * queued_spin_unlock - release a queued spinlock + * @lock : Pointer to queued spinlock structure + * + * A smp_store_release() on the least-significant byte. + */ +static inline void queued_spin_unlock(struct qspinlock *lock) +{ + smp_store_release((u8 *)lock, 0); +} + +#define queued_spin_is_locked queued_spin_is_locked +/** + * queued_spin_is_locked - is the spinlock locked? + * @lock: Pointer to queued spinlock structure + * Return: 1 if it is locked, 0 otherwise + */ +static __always_inline int queued_spin_is_locked(struct qspinlock *lock) +{ + /* + * See queued_spin_unlock_wait(). + * + * Any !0 state indicates it is locked, even if _Q_LOCKED_VAL + * isn't immediately observable. + */ + smp_mb(); + return atomic_read(&lock->val); +} + +#include + +#endif /* _ASM_ARM64_QSPINLOCK_H */ diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h index cae331d553f8..37713397e0c5 100644 --- a/arch/arm64/include/asm/spinlock.h +++ b/arch/arm64/include/asm/spinlock.h @@ -20,6 +20,10 @@ #include #include +#ifdef CONFIG_QUEUED_SPINLOCKS +#include +#else + /* * Spinlock implementation. * @@ -187,6 +191,12 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock) } #define arch_spin_is_contended arch_spin_is_contended +#endif /* CONFIG_QUEUED_SPINLOCKS */ + +#ifdef CONFIG_QUEUED_RWLOCKS +#include +#else + /* * Write lock implementation. * @@ -351,6 +361,8 @@ static inline int arch_read_trylock(arch_rwlock_t *rw) /* read_can_lock - would read_trylock() succeed? */ #define arch_read_can_lock(x) ((x)->lock < 0x80000000) +#endif /* CONFIG_QUEUED_RWLOCKS */ + #define arch_read_lock_flags(lock, flags) arch_read_lock(lock) #define arch_write_lock_flags(lock, flags) arch_write_lock(lock) diff --git a/arch/arm64/include/asm/spinlock_types.h b/arch/arm64/include/asm/spinlock_types.h index 55be59a35e3f..0f0f1561ab6a 100644 --- a/arch/arm64/include/asm/spinlock_types.h +++ b/arch/arm64/include/asm/spinlock_types.h @@ -16,9 +16,9 @@ #ifndef __ASM_SPINLOCK_TYPES_H #define __ASM_SPINLOCK_TYPES_H -#if !defined(__LINUX_SPINLOCK_TYPES_H) && !defined(__ASM_SPINLOCK_H) -# error "please don't include this file directly" -#endif +#ifdef CONFIG_QUEUED_SPINLOCKS +#include +#else #include @@ -36,10 +36,18 @@ typedef struct { #define __ARCH_SPIN_LOCK_UNLOCKED { 0 , 0 } +#endif /* CONFIG_QUEUED_SPINLOCKS */ + +#ifdef CONFIG_QUEUED_RWLOCKS +#include +#else + typedef struct { volatile unsigned int lock; } arch_rwlock_t; #define __ARCH_RW_LOCK_UNLOCKED { 0 } +#endif /* CONFIG_QUEUED_RWLOCKS */ + #endif diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 9d56467dc223..f48f6256e893 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -56,6 +56,7 @@ arm64-obj-$(CONFIG_KEXEC) += machine_kexec.o relocate_kernel.o \ arm64-obj-$(CONFIG_ARM64_RELOC_TEST) += arm64-reloc-test.o arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o arm64-obj-$(CONFIG_CRASH_DUMP) += crash_dump.o +arm64-obj-$(CONFIG_QUEUED_SPINLOCKS) += qspinlock.o obj-y += $(arm64-obj-y) vdso/ probes/ obj-$(CONFIG_ARM64_ILP32) += vdso-ilp32/ diff --git a/arch/arm64/kernel/qspinlock.c b/arch/arm64/kernel/qspinlock.c new file mode 100644 index 000000000000..924f19953adb --- /dev/null +++ b/arch/arm64/kernel/qspinlock.c @@ -0,0 +1,34 @@ +#include +#include + +void queued_spin_unlock_wait(struct qspinlock *lock) +{ + u32 val; + + for (;;) { + smp_mb(); + val = atomic_read(&lock->val); + + if (!val) /* not locked, we're done */ + goto done; + + if (val & _Q_LOCKED_MASK) /* locked, go wait for unlock */ + break; + + /* not locked, but pending, wait until we observe the lock */ + cpu_relax(); + } + + for (;;) { + smp_mb(); + val = atomic_read(&lock->val); + if (!(val & _Q_LOCKED_MASK)) /* any unlock is good */ + break; + + cpu_relax(); + } + +done: + smp_acquire__after_ctrl_dep(); +} +EXPORT_SYMBOL(queued_spin_unlock_wait);