From patchwork Tue Sep 5 21:42:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB0C0CA100D for ; Tue, 5 Sep 2023 21:42:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE87B8E000A; Tue, 5 Sep 2023 17:42:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D97AD8D0001; Tue, 5 Sep 2023 17:42:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C38238E000A; Tue, 5 Sep 2023 17:42:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B4E8A8D0001 for ; Tue, 5 Sep 2023 17:42:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 88FB7A071A for ; Tue, 5 Sep 2023 21:42:43 +0000 (UTC) X-FDA: 81203868606.17.93880CD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 7EB3C16001B for ; Tue, 5 Sep 2023 21:42:41 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cl1++Ayt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yL/LhjyJgEaE1ERsjh3smTILDBi2p+nbzGXe8vQHohc=; b=Fo6DGQMXNp+v0c057XB9c00EFG2IjrwivQO4+hKrjT5IvwQGa/OdMdan+319PQyMExAFLK GuE6Vy+6wSBwt/tPWslUnNJpU0/cDS3ZtsRKwE19q54Q+NWSOH6OYGdztVEc0Q77KyFbdO gr8Y6Oo0e6XsaBVwaGv6lOphTZOa/xE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cl1++Ayt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950161; a=rsa-sha256; cv=none; b=Z6ciEasAPVfYMFWzzbYwGn2QNcb5xZPo9Yy+orbThY/aXaz8XDCxms4loyTi74km1MMieT 8QNZmmY7Jtg5r5jLUwy001N+Pd+BeMhl5ACwTzhzzmzA9TUlDnjTFAzXT00PURjjn1cVN4 ZYgUCrgfePW0x8T1+fhIVSDdiT82tM4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yL/LhjyJgEaE1ERsjh3smTILDBi2p+nbzGXe8vQHohc=; b=cl1++AytMnm7VxG1lbJ1+L8bKs+Kn0XKnFNCOvAOu3JmBK8sKUm2NoL1tTl/MJXjvlWgTs VMYYPqJG5U1CFS5CgisYRYfI8pLThXVvwEDSjPDr2vHqSF6foJCK5kibYBnsaK/HSCoS+i ndANBZ3WWyR615H8PNJwI4Wvop67uJs= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-84-1UXV2iZSPDu7-PCLodrrNw-1; Tue, 05 Sep 2023 17:42:39 -0400 X-MC-Unique: 1UXV2iZSPDu7-PCLodrrNw-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-76f0d1c4045so44101785a.1 for ; Tue, 05 Sep 2023 14:42:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950159; x=1694554959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yL/LhjyJgEaE1ERsjh3smTILDBi2p+nbzGXe8vQHohc=; b=BZ5dqTJpHBbVIEXPWTOAkZ/73/VRLBuUZEylsQYAlHWHRlZic1QzavEMywBXVMYmBi j02UeD+yEzA9RE3pv+zAumHyZa0tyTs/S874uJ4MkjBI0LiJPazriW8XaC9WHZNLF6wi fuT20FgQ6HkuxReuSErFWjl75L8dd4km3YDHKJVKLHcteLzd3P6KzCALVu7eG/53bQCT BhxoJCckTmNm8bMzp+Z6P+rMo4tiIoCnRRaMy+OLy04qZ/30nOSjRkipYUPJ2ulO41co mQCaXbl0G6qwjIHvRcnwV3jk4VoJ1FffpzngHYnrw2AwHsId1yUFyZw0wvR4O9/1CpaH Ww+A== X-Gm-Message-State: AOJu0YwYaDNf/FwvxYc9c1U1agwbl+7s5/A1MYdG+BPxdBHmnTIiftp5 HbUBZoL9nV0SQb8Kpm4N/VFLkYtchW8yIh7z7tzMUCEDgmP+ScQZVinH19FZdi9cw17FKSgLc4m yfvFJ87Zb5Jo= X-Received: by 2002:a05:620a:1aa4:b0:76f:1614:577d with SMTP id bl36-20020a05620a1aa400b0076f1614577dmr16479345qkb.4.1693950158927; Tue, 05 Sep 2023 14:42:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEdEZxCLbQuYXO7d4fPer9qvq4VofEooWqnv//Ff0/KOa/Ui0Gkdskd8CCG0Sg9WRZVsAXIfA== X-Received: by 2002:a05:620a:1aa4:b0:76f:1614:577d with SMTP id bl36-20020a05620a1aa400b0076f1614577dmr16479333qkb.4.1693950158658; Tue, 05 Sep 2023 14:42:38 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:38 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 1/7] mm/userfaultfd: Make uffd read() wait event exclusive Date: Tue, 5 Sep 2023 17:42:29 -0400 Message-ID: <20230905214235.320571-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 7EB3C16001B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5k5cn64ecmued98rh8whds4udkhc8cca X-HE-Tag: 1693950161-817748 X-HE-Meta: U2FsdGVkX18YwF11UeZ51HqeNIr54NgjvBFwClulVMOPRLXIthD9NnWpXjUuU3tvjOcDAQ/VRK/dntFNBHj7pE5XEpFVhkHQXBerBavG4Fgl/uNXTATOCm9OJAkbrd8Bb1KdROoh1GQf/aCadrAPGD1ZmS/U9A5y2hooJDqGGZlcDRi15e7IZ9fHiwc+3IFVDCnqo2xcPXMzn9YAul1szW/zIfn2PuVkRZex5C1zOZiGxtUGp7mAJ5218LPWV6qqaZ8gtHRE4KDVXJmJSi5+588LkS/UKx3sh1Hf4uCRK4RzqaM8kaRhjBd74kUKjHCzjjaS2Gp03pVkLiAINtjkaH6pu2Lr2CCMFE9YfLvJtPD42PyD+FRI1CfV+b5Nyy/QdlVp8hCZciugF7/5MF4tr5y2CglJD5uXEScK6eGxKu80O1IQx2W53QSmlwB2X2hLlWx7jrOQbWqI+eLRRP22dDMXWerLqd2R2LIrCxEF0MoGO+W1QuZwGEUqvyXJ+ET9jO/8XxdbkY6QBu/BD8oUtBEMBrjBHYZIW4vRvN/JctZW7sUP1E7E+8gizjY1szVmiQ2gg+IcWBXzBsy/YwYhyUSBQ0VXmP5E852MMaRa40T+bDpuF5+Cql7hlO71f+2XOBqQdnOTQf64TbbWSb0vyTl9JPDRgqoDbpjeQVD+pePnssj3hGTXA+F4AgwLsN4N/U5UpBQirpD+Ey1SmA44PCRLJJdKcyqfhAEuZpTcUr02beLzFKgKMOeXBeRDK2DVhP1hWMVYzsS6uh+PE+H+/VEQ7gv5wn4PrS6N7jrhMyVGZL7kx0PM5OmFQOuqZtO6Olh/d36a+t47LPkAXUU6KV47fEFA+cWJZpxEfNrrge3Sk/rGRS8jUOPer+HlCb0Z2R0zPqP4W5y8d13iEeHJkvz+f0MseDY7DEdeWTorFYBnvct3OgX1YIuudWDFZ1MCK+/cl95X9jXRLCX20b3 EDQ7yIJo LQWAfyHDxkN9ElHU37LKiuinJXkIMzWPxc9Ih0dRE7qUJj5tieTfHqN+R+3VtecgFilrNAf5jKPxjLEg50k2KSdmRvIm7R4GfhjmMQkZBkXuCQzuH1tWfy/G/EZoz15aPRr2b3plYuFxG+/DgufQSuqnIYtzGaet5npAiRDoUiMOTl4VfHCJytpXEeWIe0kkglUav1Ib29qeqtFK3tBYKDjpVJ/w3pYyamqEScI3RbdJHK5hCr8Z/XBpi5h68r2XItnV+Cay/7CO+LzzDQxZQhf2a4hG61FoDdqYNGj/sZWGNyGUWVcX3BH8jJFTU3V43wZ4vrxCYL4az1I6tAf8v7UXWGRJWNbMwPyRg0JI1h0HmCWFSYmKEf51oAIJaX59ivEbmZnRlRGydjL3qWr5fFKp9/zlrap6FYZJgXvf3E0gPPsDj1Vj/Z912uZCS2i/OWn176K489B9uRqU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrea Arcangeli When a new message is generated for an userfaultfd, instead of waking up all the readers, we can wake up only one exclusive reader to process the event. Waking up >1 readers for 1 message will be a waste of resource, where the rest readers will see nothing again and re-queue. This should make userfaultfd read() O(1) on wakeups. Note that queuing on head is intended (rather than tail) to make sure the readers are waked up in LIFO fashion; fairness doesn't matter much here, but caching does. Signed-off-by: Andrea Arcangeli [peterx: modified subjects / commit message] Signed-off-by: Peter Xu --- fs/userfaultfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 56eaae9dac1a..f7fda7d0c994 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1061,7 +1061,11 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait, /* always take the fd_wqh lock before the fault_pending_wqh lock */ spin_lock_irq(&ctx->fd_wqh.lock); - __add_wait_queue(&ctx->fd_wqh, &wait); + /* + * Only wake up one exclusive reader each time there's an event. + * Paired with wake_up_poll() when e.g. a new page fault msg generated. + */ + __add_wait_queue_exclusive(&ctx->fd_wqh, &wait); for (;;) { set_current_state(TASK_INTERRUPTIBLE); spin_lock(&ctx->fault_pending_wqh.lock); From patchwork Tue Sep 5 21:42:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50B1BCA1008 for ; Tue, 5 Sep 2023 21:42:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F2518E000C; Tue, 5 Sep 2023 17:42:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02AF58E000B; Tue, 5 Sep 2023 17:42:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE7A68E000C; Tue, 5 Sep 2023 17:42:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D16618E000B for ; Tue, 5 Sep 2023 17:42:45 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A0103120AC5 for ; Tue, 5 Sep 2023 21:42:45 +0000 (UTC) X-FDA: 81203868690.21.B3CC7EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id E090614001E for ; Tue, 5 Sep 2023 21:42:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FnIpWh2+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950162; a=rsa-sha256; cv=none; b=78Os8T9DJOA4SvqXngV2NZ1JUK91/1nYV6aA8s+gH8YmmcvxZDUepBwdm7V7wmRJR+kwjD 1Szyf/hLKDgUtROO7vO2r+NTSf2AQas4Ub4Cxo1i6VXsdt484tpPPHxWo8Js1uuk6HiDMx iSP+MOyUMLPEQWojrsM4ozguQgglHNY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FnIpWh2+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iFjFRpr2tLumBDFqT/D9Qiro5EdrVpMUq8hDA8RT3eI=; b=wdY40VeO+PTnSASsWEsFzOuafRrwXwZ+tSIuro2RCBsvqxL9+PHCpSzvBVe9OPsdc43Zv+ MrcdF9iG4wXZ36uhDAF7tENSP3YcVKd17zT3OcXD7CnSE6sXbEETrKcb1MfCP6Ve6EZyNP YOYt82ujjz/GNu4LY5OLjU4Jko26u/A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iFjFRpr2tLumBDFqT/D9Qiro5EdrVpMUq8hDA8RT3eI=; b=FnIpWh2+3dEgghJAUd7IzNsGWC9/+r+PpT3uoY23yk+g85h0vDQx0fUol0FJoyVSe6wTL+ z16b9KvrSw7RMWlDjfTH23X84Un7Qb9ulJxxdL4QB1ladqy+caO55iNuPJ80Cx6HLuPPDH rZTRr/LbEqwxDzMheC+9FSOATF0rLIg= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-111-O2-Wz8zQNbmF8AXMMhDqDA-1; Tue, 05 Sep 2023 17:42:41 -0400 X-MC-Unique: O2-Wz8zQNbmF8AXMMhDqDA-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-76efdcb7be4so82010085a.1 for ; Tue, 05 Sep 2023 14:42:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950160; x=1694554960; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iFjFRpr2tLumBDFqT/D9Qiro5EdrVpMUq8hDA8RT3eI=; b=UMVrEZV0pp8b9ZDseHFTcs5s0oGOrBpzzHxEhc4TBL7A9Xb4ZdQ12AjUYP3PXt0+nf GMX6j21/53KLETG3v9r+Kc83eDWHU2u5WIWXMEnvKMxdMaVnWfa+GTujlty3LcSIuZ72 K99xjUDEykPJ6ID0RKc6jXIyHcuy8vSn9q3d1jw/k8CSQHaTkwMjxi0kPjP0IOBlXSHD Fu89/0t/TbFkMfpd36NwraG01sKTDBn6eW7SdC586YaG/IEim5Pod4b1uNPCnKWjWhFY llJV60HngJUrt2cenLnRkhW3cHPDsNEu8+ZXju7CxxwxZxTr1Dv6KSdML9GIRQdxC+vs tw0g== X-Gm-Message-State: AOJu0YyfPtsEuUX0Re19nX3J3i4HwxMz+ZJdXTkSfDy5GUUeqkYwwtTU bgRbxLsaQrQ/vqXrowgykYPASBTi/2O9n+9sk76tKqM9nafwzQyBJXrYTrGaLfxcpXLqHnEOyrd 19OE3E96tyCs= X-Received: by 2002:a05:620a:1a92:b0:76c:ea67:38e2 with SMTP id bl18-20020a05620a1a9200b0076cea6738e2mr16297936qkb.2.1693950160427; Tue, 05 Sep 2023 14:42:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGb0/uwjCE+wD8vjFeylTm+w9KR1ry9a3YxdBWKPhIRxUJR2OeC5hjEf8cXax4t1I0LDP54fg== X-Received: by 2002:a05:620a:1a92:b0:76c:ea67:38e2 with SMTP id bl18-20020a05620a1a9200b0076cea6738e2mr16297913qkb.2.1693950160176; Tue, 05 Sep 2023 14:42:40 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:39 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc() Date: Tue, 5 Sep 2023 17:42:30 -0400 Message-ID: <20230905214235.320571-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E090614001E X-Stat-Signature: krg9ibzy1uhe5ti46q5me41uu6eym94p X-HE-Tag: 1693950162-215662 X-HE-Meta: U2FsdGVkX1+vJb5F+kXWCbIaY1e23mwbj8tGPXdg0m32lw2jLM/0uyA8Xd7I2YwtN8y6EzMhoy7G/7KqKQ9rk8vFkQMXmGfkPu1QBgI7nGoGbWMeEMHJy5qKGLBKTCEwYOx0KZJgn3+pPNvn5+U5Ubg6+VMVPn4kvbahlrC9KzV+Wc3wWNOBdxSFShQ4QOpcPqAueB+zHHmmfv3YAHAW1NHVdyLVwtvq01j9bnEZ41DGyc74tDtlBrskhciuURnLg96A1zWViebYBYJzVN3P5rdBCzHJOXgrl4sEHtP45N0IsY7vkycz3iMimTZs0Rk1+eV/P79XMrpOAtoJJTzfqznSAlp1inPk2okbAEyOmQwSUGEz7GsnilMxfvJiy9KCxW1yY+WN9MnAF1U5TzgS2O5hQCDo7jGa4MuB2BiPq6uuveNuyQQ/XZCF4OYfLa5++OEfHhAyZvO3KN1ULcKaz0EDpnCpwpZYjBJLIqFtT/HC33DY7ANl63YsO9W2pNTPD+6fqo9udzhkodII6H7ZBGpOGzYWxtqiRZQ5cd8hW72bU9kSmtusX0dLV3GJJIl2CuXaNVTob284qGVUlmrSnmg9uaHtC3TmbHjdERZeVOlE5pKTe7PxbYAe1WnJJLvlwDZzWu+YFgnGmY7HTtrZkX6M01lJ2/dkuKZhw2PCvzqGkkmWClXvtQYWYmaoZ1MUZunVjN57YOMTgHg9YNXpNM2z213uMwfP2JNz22ijeHsEm7YGT5BBFcTHp6T+G3EncEd9YPAc0SrYW7i32FwWFPcN73P11tVsyxipupbf8cRdb1h5c0cEljAHt+0Z3m2x7VZsv0mF/WRYhLSERF3Toj7sjaN1ALEX3sT2/akxqb+ii/kaqGXIgT1urTjxsbc5ERBwITH2xUaic5awlAUEteDtUj7j5O2UVBXZHchh6RlGloFaCsT+x91nlFCTkOD9HbIPKu4+c+37sXoHXtp r4AT45EU avzZ9vTX+rWU3mGqji73liMIpPKyDxyaajzgyPB2C5FOGPmaPw+hh8p7yyEJpiED5Yb1gyyPDYyea7U7yF1AcSdnmhS4Hp+YC9S5UR2HjTklhcZ7ltMPDxAJwxGr8KklbdlrEI0cOQUVifhAvn3Y9N2pSGmEO0ox+Mp7PKgQqww5qqdaDexUDYJg/2wbOIC9AW0LAi2Wef3yHbsIYny/LJo04O1JE+VsWRLhEEwNWN1XWbjWYd9Btmyv0VSSJ/RuCv7afShPVhY39yHYdLpyJCGWktvJ/+KGRVdUi2d+K8BGtDT8iK1BWIhAUx7zTQB3O+SAbMQ4llnX/jOCzk7Re0BOo8bgsLx+iiF7dy8gmQE+2Hm4Q2bSDnxj32dSWqfSv7HqFSIAC8X8uIpiE5Mvgy+HC73S3ojsChO4uFBTuYM4fwwFErf+sGAVX5IuFViTIcJVkXPrqAamLq/x7hoEp839vIXGr90Ni6CW6cXS7q3w9fIcPlVRwtnD4Y/ChbrwjPb6K X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allows the poll enqueue function to pass over a flag into it. Signed-off-by: Peter Xu Signed-off-by: Peter Xu --- drivers/vfio/virqfd.c | 4 ++-- drivers/vhost/vhost.c | 2 +- drivers/virt/acrn/irqfd.c | 2 +- fs/aio.c | 2 +- fs/eventpoll.c | 2 +- fs/select.c | 4 ++-- include/linux/poll.h | 7 +++++-- io_uring/poll.c | 4 ++-- mm/memcontrol.c | 4 +++- net/9p/trans_fd.c | 3 ++- virt/kvm/eventfd.c | 2 +- 11 files changed, 21 insertions(+), 15 deletions(-) diff --git a/drivers/vfio/virqfd.c b/drivers/vfio/virqfd.c index 29c564b7a6e1..4b817a6f4f72 100644 --- a/drivers/vfio/virqfd.c +++ b/drivers/vfio/virqfd.c @@ -75,8 +75,8 @@ static int virqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void return 0; } -static void virqfd_ptable_queue_proc(struct file *file, - wait_queue_head_t *wqh, poll_table *pt) +static void virqfd_ptable_queue_proc(struct file *file, wait_queue_head_t *wqh, + poll_table *pt, poll_flags flags) { struct virqfd *virqfd = container_of(pt, struct virqfd, pt); add_wait_queue(wqh, &virqfd->wait); diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index c71d573f1c94..02caad721843 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -162,7 +162,7 @@ static void vhost_poll_func(struct file *file, wait_queue_head_t *wqh, } static int vhost_poll_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, - void *key) + void *key, poll_flags flags) { struct vhost_poll *poll = container_of(wait, struct vhost_poll, wait); struct vhost_work *work = &poll->work; diff --git a/drivers/virt/acrn/irqfd.c b/drivers/virt/acrn/irqfd.c index d4ad211dce7a..9b79e4e76e49 100644 --- a/drivers/virt/acrn/irqfd.c +++ b/drivers/virt/acrn/irqfd.c @@ -94,7 +94,7 @@ static int hsm_irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode, } static void hsm_irqfd_poll_func(struct file *file, wait_queue_head_t *wqh, - poll_table *pt) + poll_table *pt, poll_flags flags) { struct hsm_irqfd *irqfd; diff --git a/fs/aio.c b/fs/aio.c index a4c2a6bac72c..abb5b22f4fdf 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1823,7 +1823,7 @@ struct aio_poll_table { static void aio_poll_queue_proc(struct file *file, struct wait_queue_head *head, - struct poll_table_struct *p) + struct poll_table_struct *p, poll_flags flags) { struct aio_poll_table *pt = container_of(p, struct aio_poll_table, pt); diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1d9a71a0c4c1..c74d6a083fd1 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1270,7 +1270,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * target file wakeup lists. */ static void ep_ptable_queue_proc(struct file *file, wait_queue_head_t *whead, - poll_table *pt) + poll_table *pt, poll_flags flags) { struct ep_pqueue *epq = container_of(pt, struct ep_pqueue, pt); struct epitem *epi = epq->epi; diff --git a/fs/select.c b/fs/select.c index 0ee55af1a55c..0433448481e9 100644 --- a/fs/select.c +++ b/fs/select.c @@ -117,7 +117,7 @@ struct poll_table_page { * poll table. */ static void __pollwait(struct file *filp, wait_queue_head_t *wait_address, - poll_table *p); + poll_table *p, poll_flags flags); void poll_initwait(struct poll_wqueues *pwq) { @@ -220,7 +220,7 @@ static int pollwake(wait_queue_entry_t *wait, unsigned mode, int sync, void *key /* Add a new entry */ static void __pollwait(struct file *filp, wait_queue_head_t *wait_address, - poll_table *p) + poll_table *p, poll_flags flags) { struct poll_wqueues *pwq = container_of(p, struct poll_wqueues, pt); struct poll_table_entry *entry = poll_get_entry(pwq); diff --git a/include/linux/poll.h b/include/linux/poll.h index a9e0e1c2d1f2..cbad520fc65c 100644 --- a/include/linux/poll.h +++ b/include/linux/poll.h @@ -27,12 +27,15 @@ #define DEFAULT_POLLMASK (EPOLLIN | EPOLLOUT | EPOLLRDNORM | EPOLLWRNORM) +typedef unsigned int poll_flags; + struct poll_table_struct; /* * structures and helpers for f_op->poll implementations */ -typedef void (*poll_queue_proc)(struct file *, wait_queue_head_t *, struct poll_table_struct *); +typedef void (*poll_queue_proc)(struct file *, wait_queue_head_t *, + struct poll_table_struct *, poll_flags); /* * Do not touch the structure directly, use the access functions @@ -46,7 +49,7 @@ typedef struct poll_table_struct { static inline void poll_wait(struct file * filp, wait_queue_head_t * wait_address, poll_table *p) { if (p && p->_qproc && wait_address) - p->_qproc(filp, wait_address, p); + p->_qproc(filp, wait_address, p, 0); } /* diff --git a/io_uring/poll.c b/io_uring/poll.c index 4c360ba8793a..c3b41e963a8d 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -533,7 +533,7 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, } static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, - struct poll_table_struct *p) + struct poll_table_struct *p, poll_flags flags) { struct io_poll_table *pt = container_of(p, struct io_poll_table, pt); struct io_poll *poll = io_kiocb_to_cmd(pt->req, struct io_poll); @@ -644,7 +644,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, } static void io_async_queue_proc(struct file *file, struct wait_queue_head *head, - struct poll_table_struct *p) + struct poll_table_struct *p, poll_flags flags) { struct io_poll_table *pt = container_of(p, struct io_poll_table, pt); struct async_poll *apoll = pt->req->apoll; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ecc07b47e813..97b03ab30d5e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4877,7 +4877,9 @@ static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode, } static void memcg_event_ptable_queue_proc(struct file *file, - wait_queue_head_t *wqh, poll_table *pt) + wait_queue_head_t *wqh, + poll_table *pt, + poll_flags flags) { struct mem_cgroup_event *event = container_of(pt, struct mem_cgroup_event, pt); diff --git a/net/9p/trans_fd.c b/net/9p/trans_fd.c index c4015f30f9fa..91f9f474ab01 100644 --- a/net/9p/trans_fd.c +++ b/net/9p/trans_fd.c @@ -550,7 +550,8 @@ static int p9_pollwake(wait_queue_entry_t *wait, unsigned int mode, int sync, vo */ static void -p9_pollwait(struct file *filp, wait_queue_head_t *wait_address, poll_table *p) +p9_pollwait(struct file *filp, wait_queue_head_t *wait_address, poll_table *p, + poll_flags flags) { struct p9_conn *m = container_of(p, struct p9_conn, pt); struct p9_poll_wait *pwait = NULL; diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 89912a17f5d5..645b5d155386 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -246,7 +246,7 @@ irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void *key) static void irqfd_ptable_queue_proc(struct file *file, wait_queue_head_t *wqh, - poll_table *pt) + poll_table *pt, poll_flags flags) { struct kvm_kernel_irqfd *irqfd = container_of(pt, struct kvm_kernel_irqfd, pt); From patchwork Tue Sep 5 21:42:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E327CA100D for ; Tue, 5 Sep 2023 21:42:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 895758E000E; Tue, 5 Sep 2023 17:42:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8417B8E000B; Tue, 5 Sep 2023 17:42:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BC788E000E; Tue, 5 Sep 2023 17:42:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 595A08E000B for ; Tue, 5 Sep 2023 17:42:49 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 32917A0AFC for ; Tue, 5 Sep 2023 21:42:49 +0000 (UTC) X-FDA: 81203868858.19.6C7D920 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 2BBA6140006 for ; Tue, 5 Sep 2023 21:42:46 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IwdjEhtT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950167; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BAc+MDavbWwJ6795LOxgDkxGvxMJL+vGI+ILCcyBkDQ=; b=drM/L2uJYUrl0gzl2OFpwzeVavTWk26sALwhubrSZkP1G3CpS8mbMKxevsXjnMap8JHo2e toJ2v8wAB+JlJutMYkpMhpUH8s250NO2mns41rIDLBruhFE/U9ldLu8C6ofmmpnCxAaPsE 6hz5avmNB5ppnxsltGv0E9mhjINJzb4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IwdjEhtT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950167; a=rsa-sha256; cv=none; b=wL8zbomYoKJofAeb/8csjRyiHGF+XXqya+CO+v5l004YaMC+7Hoq2qjVWHO79Qo7dNgvAd VOPsibQPgWBq0CQBXSBXnCl6sW/l9C9dcqOPS47cdBN32B2RH4PwZbBOXP3hD4LGMpRbWi /HMlYQ6m7XLSNUKlRrNrPnL+732wq/Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BAc+MDavbWwJ6795LOxgDkxGvxMJL+vGI+ILCcyBkDQ=; b=IwdjEhtTpozqlt8KK3QPPWfb7w4CZnevm2Dy5pqlKnjmAU7CU3rl0PJtkD6skra+GGBbpT JRJGX0q0e07pq9WNkcYfPIOjcgsg5rZxwPI8UC9DD3f0HqPtyG/iqPgJs5vB4xIgoemSsJ r3W5uP9EhR797jze5FMlkVb7IIRL7v0= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-321-mDyZQ1UrOnGcHHP7QSNkpA-1; Tue, 05 Sep 2023 17:42:42 -0400 X-MC-Unique: mDyZQ1UrOnGcHHP7QSNkpA-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7708c1ae500so2639485a.0 for ; Tue, 05 Sep 2023 14:42:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950162; x=1694554962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BAc+MDavbWwJ6795LOxgDkxGvxMJL+vGI+ILCcyBkDQ=; b=le5k+6EEa7O9+fzVetA1Jg6278l3gxFgvS69AnrqJUGHRCGSji/8tH2j/Kj4+XHD1p yVUF+qoKpJAJl5+KZXdyuzp1kmZ2+usc/HDJqs+JmfaUMotKkPXQJ7ENl8Gv7bHd1msT 2WyvbjESSSe+NVHo7vDw/0EILJwsLnmrALjqPfywnU0UmSjFjdk84vhfMAsvaetkSevb VshWzPkNW1rGJwWjzdDN6SBbtodhlT3ZpiX6xwJlF+xTxut/BF1shT2PcLfFHPgoXrLv RDZ2hNuXYtq8bftYzQ9IURiktVf+za701oT/1HeAJQqB2dvZ3c6DHPG0QN6XGpL216nz FZJQ== X-Gm-Message-State: AOJu0YxWFQ5oqjrWO1okuTahLPJ093dZ1o6or9wY8/E4svuDa8cF3PbE 7zvQ28udnQnV1YnQtgI2G/4ie4E7/1EzaR78r7bgLYRd2XYIF2DGazPXzgfS7lO0Z3fUz3UknkF FIy7uafft9cORYSQa+1c= X-Received: by 2002:a05:620a:1a26:b0:76c:ed4e:ac10 with SMTP id bk38-20020a05620a1a2600b0076ced4eac10mr16770014qkb.6.1693950161775; Tue, 05 Sep 2023 14:42:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHX81eRpvF4K65o6If4EOmXGS0BgJwHt7s35QflxCMt79X+rblwCZ36Xulk34UDBuMUSImVnw== X-Received: by 2002:a05:620a:1a26:b0:76c:ed4e:ac10 with SMTP id bk38-20020a05620a1a2600b0076ced4eac10mr16769997qkb.6.1693950161469; Tue, 05 Sep 2023 14:42:41 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:41 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 3/7] poll: POLL_ENQUEUE_EXCLUSIVE Date: Tue, 5 Sep 2023 17:42:31 -0400 Message-ID: <20230905214235.320571-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 2BBA6140006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 5m8nekbjhwthkrc3o83me6sn4nwxq9dx X-HE-Tag: 1693950166-43682 X-HE-Meta: U2FsdGVkX18OQEZw/rMJcnV9t7cdWk8TjEJr0nmera5Bc5lQ/y14FhxP1cJhcu0pqfRfbyBSj1pEWOOy+uhEXwWrjNxKwKc1U6qlz8m63UtOnV6hXot83QXaYf+C5bi3SLdWGutzvPpuz2yJsJT6nITMOzGE0ZyQ8qAwgimD63V6mqg7GYyFk3uiDRRaux3ooHOm9H+ND+EepsIgdvEeLpkQ2CmLESm+cPHVhPRRO/ThHV9pdlsAkfiePlDKxmepEqsQRP2gdcHbgq+nOCtQ3iLLvrG4422yQ2L5iCmLbGbVnblzN7MuvmoOYYg5oHuKx3+/GywX2vdUEDndeWeL7Mg1C+17GNBXH77RkgdbsRL+g6Xx2hQVsBidmkEGDF5sv07wbWruWP8q56IoBuM1NivHMdTllSOn+mRlNg8PjdDykVBOMVbaGJimSUnv6/aVnvhibqAUZEODtkFPcgOQW459qOXetTnPE3g6nAvu22+ml5c/13WlTXrTrBfV0b44N6I1YU/XvW5XD37DnT/lpmL6Y5csGZ9CrZkUNZcdyj1VADNxH4LkLlg2XXD8F17ykyduP1rUf4AQGu27SejwyhWyu3jGyMUWV06ce3HJCODczSnjYuGROFf8NBVpoPEET1cPgFyrMwTYmuBn3TvOxxgQMHAnYhNaSkAgX6jNcveQ3D5iQjPdS0Q+BBbRST5X+zMP+83s0BAtbsdGnNRyIVWuBEp9Y0bfErwlMfyAZFAfEB33IenrnipsFpXv3TchdUJr+JgQ9lWkuw4IcaqYdbZ6s7FNoSU6fING/W0njV2cAy/Eaxsq1Hns44ODxy2Ed9tE2+3jQJZf7KBB1jPzLj9u+9SGjadYySkMgbSAzYRhiY8Lh3W264AQywM8kgjJStse2n+4VJPMtGSGtLJYCqazHfBbGNcFG2KIbDQQFOipPqFML3wAH2KVlDOXCWnbTtTsUzJ++U0NwIoHz34 3GXyJqaC ZVJYpesdf2jl2l5fgd4bIehIm/SfdAvSX1lDVR5tI/QAtu6IbriDStCqwZq50Bhxc8avdzsbPVOx3Erhay2X6ySLPuZXpu6bClNsONNx/wusXq/sxWLVcbwgRS3RMpZjPCtgIidWoe8/w53p/61qiuLIv8uvWtI09RDOy2q1zEYaMdThnc7AM3GWqSrkcSxeVNDvaWMdKUss304VpIhbRzEHJJKewD+z1fra6QPV9Z8H1gsXtZTWlt7W1G9zHRFisfjCaYturZPWS6nk3Z5fV/84xPhf+gHjskeGYRHxx5zybscEiVqeM3BAZSoOelhv66QqeilS7vwm8qs6oKpTxhAPI0nlSpgA0t7jH7DSvlZidid/t+eUScwhKUFxcF2JuLrflEXEnfJq4PuGE3rSS+gFpyePKnHoFnVb3iFRvZM2Hu6/8mXPaOlc2DGqi1iAzhOXIyoF87g6Bvyxdx6HBrYXe6lCjusH2xreaz4pJ37b4T2dO+lImr7hRFTJW6/0FofHj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a flag for poll_wait() showing that the caller wants the enqueue to be exclusive. It is similar to EPOLLEXCLUSIVE for epoll() but grants kernel poll users to opt-in with more efficient exclusive queuing where applicable. Signed-off-by: Peter Xu --- fs/select.c | 5 ++++- include/linux/poll.h | 20 ++++++++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/fs/select.c b/fs/select.c index 0433448481e9..a3c9088e8d76 100644 --- a/fs/select.c +++ b/fs/select.c @@ -231,7 +231,10 @@ static void __pollwait(struct file *filp, wait_queue_head_t *wait_address, entry->key = p->_key; init_waitqueue_func_entry(&entry->wait, pollwake); entry->wait.private = pwq; - add_wait_queue(wait_address, &entry->wait); + if (flags & POLL_ENQUEUE_EXCLUSIVE) + add_wait_queue_exclusive(wait_address, &entry->wait); + else + add_wait_queue(wait_address, &entry->wait); } static int poll_schedule_timeout(struct poll_wqueues *pwq, int state, diff --git a/include/linux/poll.h b/include/linux/poll.h index cbad520fc65c..11af98ae579c 100644 --- a/include/linux/poll.h +++ b/include/linux/poll.h @@ -29,6 +29,8 @@ typedef unsigned int poll_flags; +#define POLL_ENQUEUE_EXCLUSIVE BIT(0) + struct poll_table_struct; /* @@ -46,10 +48,24 @@ typedef struct poll_table_struct { __poll_t _key; } poll_table; -static inline void poll_wait(struct file * filp, wait_queue_head_t * wait_address, poll_table *p) +static inline void __poll_wait(struct file *filp, wait_queue_head_t *wait_address, + poll_table *p, poll_flags flags) { if (p && p->_qproc && wait_address) - p->_qproc(filp, wait_address, p, 0); + p->_qproc(filp, wait_address, p, flags); +} + +static inline void poll_wait(struct file *filp, wait_queue_head_t *wait_address, + poll_table *p) +{ + __poll_wait(filp, wait_address, p, 0); +} + +static inline void poll_wait_exclusive(struct file *filp, + wait_queue_head_t *wait_address, + poll_table *p) +{ + __poll_wait(filp, wait_address, p, POLL_ENQUEUE_EXCLUSIVE); } /* From patchwork Tue Sep 5 21:42:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 965CDCA100D for ; Tue, 5 Sep 2023 21:42:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC4068E000D; Tue, 5 Sep 2023 17:42:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD6068E000B; Tue, 5 Sep 2023 17:42:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B01A18E000D; Tue, 5 Sep 2023 17:42:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 977178E000B for ; Tue, 5 Sep 2023 17:42:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6ACC0160ABD for ; Tue, 5 Sep 2023 21:42:47 +0000 (UTC) X-FDA: 81203868774.05.BCF9E30 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 614094000C for ; Tue, 5 Sep 2023 21:42:45 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lkvio2yb; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950165; a=rsa-sha256; cv=none; b=UIp5jjQ4A8TiTsftjfjgSLpHPckUfxz5phqPCs9BbfY25heSlpb0Mh97PtlFdvVguwGMhV GLLagkdWwy/rpe5Mv7o1Z7Zu9YS6mju/uy+ItxVYRoFgeuLXfPzJBWtlxGf5mcSd0ha9hH KA8p4STanVGO3hYs7kM5LVsuDhNfWMI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lkvio2yb; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i4qwgXUgzQeLfKMT3tv3YVc7S/hLt8ARXOPSgsfGIQ4=; b=8lUL/tZBSMISTs3X4gNLwAGvjyDPkEKjHyRjtYmdzigSppnE/ALkq1jJB1QcYonLvaEGN+ MbiKcV+fHr6Mgi3ogIn7ByvKJx2xqpoHxZn9Y+3sACv4+vk5HpUovaRMJyLSfEchBx6bIW 4ZF0ZMXasXwCFCLZN4WkEL5BjFkHBLU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i4qwgXUgzQeLfKMT3tv3YVc7S/hLt8ARXOPSgsfGIQ4=; b=Lkvio2ybCpoBmp/4VfNsaGEyf3H83DGRRbW+C5dM0mJUGf34WhOvyJ/eqmQ+VBuA0oVu8p D4hgRkIGzrsLIZ0O0O4vucc11gPHs85yQxUr0i8iszPoNGOFJyJlgvbVir30t97FpHSnYc LpkJoqACjcz9HKBr9/rmtE9IOZvtqXE= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-413-rdYBiinpNe2EEFTp6aEkkQ-1; Tue, 05 Sep 2023 17:42:43 -0400 X-MC-Unique: rdYBiinpNe2EEFTp6aEkkQ-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-63c9463c116so6468436d6.0 for ; Tue, 05 Sep 2023 14:42:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950163; x=1694554963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i4qwgXUgzQeLfKMT3tv3YVc7S/hLt8ARXOPSgsfGIQ4=; b=MIDVkD3K/wDnSF2TL7QhU8usd7KxVu5YAhUzCOHVNnxLx+5zgzX+nbA/aN71glDc96 3RDyuXw3ECFCiN94eFhhqASmyISOgsfvUG7HhQLaxosQqNm1yD0J4ZcNLiGCzgPIDHlj lwEHjE1GJ0U7HyV1xuDxNGaODmHVNtwUZEOEgoX8xN7bNYbJbRw+9mefkORSawhNToOa WykP9gSJEXGZ2wUeo5cJwzVr0UUkgxp4ygC/XGKTP+vAv6G7M5J5qJUhXIJLuqo6Kwcg 5qv4+WRLOVNt62fnjyOVXJkkVI4Ov3ijgIAb1NwDLYREJsXCvBYpLgMYAOgFeUfTNeGc 5Lmw== X-Gm-Message-State: AOJu0YzUdMPwc/3YYLv6vJ3F99L28lRplKC2TrHnbi1liNqPJhdoda9J UJe8ALSvVYDYgnvpcI/lcJJgNqahVUoPdMH21sw5G9ASlgmfl/Tr351W1vZauT8J92SH4l+/Dms YfdmY6gF6NP4= X-Received: by 2002:a05:6214:21e4:b0:64a:8d39:3378 with SMTP id p4-20020a05621421e400b0064a8d393378mr17304042qvj.4.1693950163015; Tue, 05 Sep 2023 14:42:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFlS4tLxX3RT0SNIZn/SUsnHB3xGIbADDb9sPouLrcBV//Rq31qTbeRmNhRZtJnSN5gXEiBhA== X-Received: by 2002:a05:6214:21e4:b0:64a:8d39:3378 with SMTP id p4-20020a05621421e400b0064a8d393378mr17304030qvj.4.1693950162797; Tue, 05 Sep 2023 14:42:42 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:42 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 4/7] fs/userfaultfd: Use exclusive waitqueue for poll() Date: Tue, 5 Sep 2023 17:42:32 -0400 Message-ID: <20230905214235.320571-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 614094000C X-Stat-Signature: amr169dg7x8rcxdaxqqhkd9homr6nrpn X-Rspam-User: X-HE-Tag: 1693950165-379225 X-HE-Meta: U2FsdGVkX1+Oiu07epmP6TfLiqSX3lIQ11zHudiCXGAfY2w8Or/ETgPrRZ5S3GYGqiOc1ZMqqzpSkx0JR4vibRp4YyeCtSlGQ0jb3Fbc/+m3LA2mlxIrJo/d5IYFbY8loXJRLD7MJpwOEwgPawKrlJNjZ4wFGdgqtn7V7TXePw4LkK/HPd9eaFuK3pPg6uLk67fpOPsp+EqGje6Hy/i268HX6xM9RONZLJpCoBn4z+XGWI0VKP0dbxFDG/6fLOCrMRxD/l+kBqd+Gijk8VGvXg+aJ1nWo2VwCV6IyrSYhy9Ouu36i/bjkpkAmxv++Gyt4m28QYf8xG0wA6Udn/WcsjPq3yUSDEMB4kDVZpy9mxPNWdGludbO2COYrPT6nRIrfuLunjeWNOvi7qFMY/6zjEy7vPhLj20KsWT3fqpYpUuaQOcJyK99rlz0LYTw5TCX3FeUxdgJuVjPzbic48XPSONdUbTK8+sRqy8IZPlD/V/c4XXPgBYMGNe5sMB7xdnj54LrTvO24UYUigRtgt9xnO4H0wZCV1q+XHIrRxO2C6Ei/f6G9KaPTC59KGB1/GZZYIUVNNDyaT4OnQ9g7tToVuNIORz9GMqMLDZKIjGcJLamcWk/qaJ89AJezLi73lQp4THELK3PNgoOl2aGdeLD6iZLgiaIfZohvXegdOZp+URzDQVboCCBcNbBDyUt2z5xzdvkzLV7fvjWsINxRy3GduvFyEikoi4nKq9qny+QQYa4Ixojj7fX1mJ4Qg+yDNnUijz8f1vG+02Qvl55CwiUvvFPmB9LDUwDrTHFi87G0Bk+U9iOC4VTYwIrMp6zcalNUimL+9eSRql8UqTXySJUTrEiYmPXmdWTpWuuA6iYY7ZPhnH1sLnVPg5V/Osyv7A7EOc0ZVpgkn8Wrc2GsQbu6Bb2gWAtOw+Bfv/Nh25Y4DqGPuTSAmYnunI4wBUC3kJPX0zAcQy7rKZG+KOqFsa Yod6uKVm //jgaj+bK49BgncVU22KkjrWSaqE4ScO77qOhta3MXz9ygP5gWhfwjNo4X/RYfXZuv49YJEm0fLMQyd9Cgb3ScIWmwRG0W1bkGpbC/o6YNlHun5B5UKuiXyU2T/Uh9otM1aQjSyW2I0cmkNG4m8f0gp7Th1x9bNArKWvH57yMXKJDih8GmE9PlU40XiDxYnZqwHuFq/J9qD2dDjBXmqLT2bq0FjA/d/E03bbHdgj4ZcFYeIyaId8Da0mz7JLkww0avP5zlmg80WQEZJmzA+GCebr/daBpET/sSDg2Up77Gawm6bxEWH0Dlzj2AXvccfreumKoAzP3DeXTH1s7WlyUgSIzsprwXgBDmDsBi+YU81PUWtyo4ngdfsHpUWOCjtB8318ZMVREVoxwbb3arXbx4ya8EvswTw3uTMQDsqOk+85nssaz3ICtPuYDemMdXPz4bBc8E+8pYcqvhaA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Userfaultfd is the kind of fd that does not need a wake all semantics when wake up. Enqueue using the new POLL_ENQUEUE_EXCLUSIVE flag. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index f7fda7d0c994..9c39adc398fc 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -994,7 +994,7 @@ static __poll_t userfaultfd_poll(struct file *file, poll_table *wait) struct userfaultfd_ctx *ctx = file->private_data; __poll_t ret; - poll_wait(file, &ctx->fd_wqh, wait); + poll_wait_exclusive(file, &ctx->fd_wqh, wait); if (!userfaultfd_is_initialized(ctx)) return EPOLLERR; From patchwork Tue Sep 5 21:42:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 720F8CA100D for ; Tue, 5 Sep 2023 21:42:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51D268E0010; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 499918E000B; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 312218E0010; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1B9238E000B for ; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F05C91CA7EF for ; Tue, 5 Sep 2023 21:42:50 +0000 (UTC) X-FDA: 81203868900.03.7864D9E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id BA5222002B for ; Tue, 5 Sep 2023 21:42:48 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bAO22uVa; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uQJce4v94fBnSsGo2eU4ZKQw/Wk8m14ksf+Px8TLSGM=; b=7UN6LRfo5kzYoFFaJ30i15/JdNRl1MU7zSZU4tW23y/bFMk8TqMo9PO+uECsIgDE0RJAco AAK7Lh06RBTlxSHro/2oy4jNa0ARW13qotizGzgkflzE3NlfQMdq6/YUig3jHBzruTRN0u 9w74D0QHcr255/8UDT3i45vK6uNQzvE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bAO22uVa; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950168; a=rsa-sha256; cv=none; b=Hx40Gsa0J+iKhERHoyVebFhDTCTTOAXo6S7haXSP2MmoC2zrae5s6TtVpPciIFsfN6Fz6e +LHGKROblDpcyg/mbB19hAlgVhHW6vckH65gI4+i74I/sI8OqQUtKl8c/UFqlv99XVJQBJ xw04/LHa5COQ/S29cOjcUTFp71TXYa0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uQJce4v94fBnSsGo2eU4ZKQw/Wk8m14ksf+Px8TLSGM=; b=bAO22uVaAb8ZdKkd8XzdL7plbo+Ncc0zUdp8qneJ9XBqMyfVSk8oZvDRhQOkYfI6R5VGmy bRfNkFpIg6ta65nfkaMNtoGPH5oNKgowkfkntlzCGVUvSE7tE3yPayGFcQ8umOOL6VSAEz lVXzb9ySxRjPcG27mQtjjco3dISwkZc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-125-v1eyI8SWPAGq_IhV48Opzw-1; Tue, 05 Sep 2023 17:42:44 -0400 X-MC-Unique: v1eyI8SWPAGq_IhV48Opzw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-76f0d1c4045so44104185a.1 for ; Tue, 05 Sep 2023 14:42:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950164; x=1694554964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uQJce4v94fBnSsGo2eU4ZKQw/Wk8m14ksf+Px8TLSGM=; b=lcPczEenmKfXyyNrgcUj8B1DNg4+UeP5RDfQBdi+F6wPrampB0BWkfr+meJ0LFaEom uZeNoW8WlkBWbt5y2vlfmSQnuAv0SDmIVSm4zUOZNbWNQaHcqC+yHELzuai5Fy6aRX3W RhKEabkR8S8GdJN00SspZoU64acYrGsleDTGFN9ndtds0Ps9Q+4OhhATIoeQETKXw3o/ caocZnifZorL2RKro3Y5NrvItIpBZ/c5lTgOA1mltJOV+okCTPa4nlGmtBR9q3OqJRuW NgFlIe920DEKcjmscavktTiYrwPZ3fKci2ep0+y5hOLBKMGyWHnfjwVTTKUdFfwuknyb RAlw== X-Gm-Message-State: AOJu0YzsxTHoEOtUDfPUwcybH0mOroXszh3GrPdV3MFFyziBnnveOup0 tNtR1Y+BWHNO9SnQeZCwrto57jekBte+M4w6G3og12YzdvEb3MahpDkEd6YFMWXvB1WEbCd5w3d 60vyxdlOWE2Y= X-Received: by 2002:a05:620a:1aa4:b0:76f:1614:577d with SMTP id bl36-20020a05620a1aa400b0076f1614577dmr16479566qkb.4.1693950164460; Tue, 05 Sep 2023 14:42:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH0CeamzOJT7Goeqq7g3UHhRHuitMC171tC//j49BSbILvNCZigPeU/QwAtYa7Vbv4u1ikW+A== X-Received: by 2002:a05:620a:1aa4:b0:76f:1614:577d with SMTP id bl36-20020a05620a1aa400b0076f1614577dmr16479552qkb.4.1693950164161; Tue, 05 Sep 2023 14:42:44 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:43 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 5/7] selftests/mm: Replace uffd_read_mutex with a semaphore Date: Tue, 5 Sep 2023 17:42:33 -0400 Message-ID: <20230905214235.320571-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: BA5222002B X-Rspam-User: X-Stat-Signature: ddgb4ccxqs9bbohdxn9snq3jo53ic8zp X-Rspamd-Server: rspam01 X-HE-Tag: 1693950168-232416 X-HE-Meta: U2FsdGVkX1/58mapCLzjbXSKuT23yqMFq3vyM/esCjR0jITIFkGwZY8Jx0l42rVW7KD5odKu0PQZ5Ko844od5NwMZJczmwIuEaAUjvg5B6FuBk6fCF7KmNjl7ATwlv28/LSUrU8Ojy2e8iw+Zbx4AH4EBqC1cWwXAv77q7/pmfWeohvmOnySfTV/pF01SP7I/5TwOtj5e2gYzphiDZIFi2LDKaNvE4r5ZE6hk+wpt8yfqXQAmrvYA0f+o8iwBLe5uTdyhxeNdeQMg/hOzqmQk7I00oK5j9VY98tTuy86VfBnon9P5xa+L2G+31eV5JFEN4v0XjQX+0TQAGb0hKBjLqvxbx2/Xxn8u3AbxzOPEWdQEuAJq/q3olsjyCVg+Wqn1hFTAQe1Jiwj1EGUcN8bYq/YbBqD7fy+ePRHP3koMa0LzUQD+nVqwIKgrqAvAC3Ny8qwtL82kaXBb6XDGae1CYunBPxx0PyBhxLYh1sjDL8J0lBWKBnHN/+RPyqZu9piGyV8mxkV/jr1o1jfUiFSZY8UAxAkaFNN4Wv7jJXW9v3WBaotBYuD1d6eagiPYq4rgLjmcwSYXF/+45jC/dl9SwqLxxdPcNganPCuRjHo2khc3Q/YoAEXyeU64iGq2hWR+V3YUFub0mTIfSr+AOH0KZvuFZnq0ApYH9RRHNQM6nsCoQwVXd+H9ZudYY23QtcdKhuTjCF3wlfAmPmSj0OukHpH3iJ3X4C6uOGL37WwIYQDE0ldmOvZEqa5xEXsfwp6Uibi4tm03uMUP+mdtKu00eVkSJZr6/IFgkY5fnbHXAk5bZk3PMP4UGkV5omxsLIkclIFiUKGbu/PcEunlzBYysDuzKxNNH9D3ATSEMFJP3adPGzNLnIWtAflOfaxqs7/mroN6fR0gVbMf5JrJeaS+I5eE2IIevZG8yUBfmQT8Xl3eNCBtP01zglYYRfv15zklqNAwGWsPjJYVLq8dPd mTyaGcpP wUtQfj+JPIFdiWZJ3D7LX2Nz0589xqgj8V6DKeTGKpvYRH7/JoQmc/b0AYsJL7QrjfwMoFLLPLM8D6SBh2MvK+Wul3yVlFcJT/Ki8hEDWq54l5TKB3FPegHwWGgiya7wJ/Lf0QdFWDfB3cJA5osfesSJW8/VIIRSpfvs5prxdkmxfT2Aj4hT1oMurv07pqYBuicJXxoyMknZOwRvYKrhPBnpKOhr2CJNXCdxQ3TTmRUjRsxEpLBe9eUxYshaIy33Dif+A2acFdu1ULNED4PpPl0Qnp3Ey8oWZsZIjhxng2ENVsBBCSYO12fsbpu4ZMav4Ks+5OH0ZeMSZkMvQp/XuJFsG6F8L+wq4sHUMkx+10VeC3JDLf1xDIwVTKAiSCMIn309ghoDc2qXDrmkDEfwPfQ2L1B2MbAxza9V7TjDz4r/FTHwoYaAGMI6UE+gswWMiJs8NnEjcrguedK/hN9C+9m6DARyxtX1ZbFNAyTvkPcM4HCVXInjd2larQ1AZ6lQfoazY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Each uffd read threads unlocks the read mutex first, probably just to make sure the thread is reaching a stage where pthread_cancel() can always work before the main thread moves on. However keeping the mutex locked always and unlock in the thread is a bit hacky. Replacing it with a semaphore which should be much clearer, where the main thread will wait() and the thread will just post(). Move it to uffd-common.* to be reused later. Signed-off-by: Peter Xu --- tools/testing/selftests/mm/uffd-common.c | 1 + tools/testing/selftests/mm/uffd-common.h | 2 ++ tools/testing/selftests/mm/uffd-stress.c | 8 +++----- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index 02b89860e193..aded06cab285 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -17,6 +17,7 @@ bool map_shared; bool test_uffdio_wp = true; unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; +sem_t uffd_read_sem; static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selftests/mm/uffd-common.h index 7c4fa964c3b0..521523baded1 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -32,6 +32,7 @@ #include #include #include +#include #include "../kselftest.h" #include "vm_util.h" @@ -97,6 +98,7 @@ extern bool map_shared; extern bool test_uffdio_wp; extern unsigned long long *count_verify; extern volatile bool test_uffdio_copy_eexist; +extern sem_t uffd_read_sem; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selftests/mm/uffd-stress.c index 469e0476af26..7219f55ae794 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -125,14 +125,12 @@ static int copy_page_retry(int ufd, unsigned long offset) return __copy_page(ufd, offset, true, test_uffdio_wp); } -pthread_mutex_t uffd_read_mutex = PTHREAD_MUTEX_INITIALIZER; - static void *uffd_read_thread(void *arg) { struct uffd_args *args = (struct uffd_args *)arg; struct uffd_msg msg; - pthread_mutex_unlock(&uffd_read_mutex); + sem_post(&uffd_read_sem); /* from here cancellation is ok */ for (;;) { @@ -196,7 +194,7 @@ static int stress(struct uffd_args *args) uffd_read_thread, (void *)&args[cpu])) return 1; - pthread_mutex_lock(&uffd_read_mutex); + sem_wait(&uffd_read_sem); } if (pthread_create(&background_threads[cpu], &attr, background_thread, (void *)cpu)) @@ -258,7 +256,7 @@ static int userfaultfd_stress(void) zeropage = area; bzero(zeropage, page_size); - pthread_mutex_lock(&uffd_read_mutex); + sem_init(&uffd_read_sem, 0, 0); pthread_attr_init(&attr); pthread_attr_setstacksize(&attr, 16*1024*1024); From patchwork Tue Sep 5 21:42:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB828CA100D for ; Tue, 5 Sep 2023 21:42:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C136E8E000F; Tue, 5 Sep 2023 17:42:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC5C38E000B; Tue, 5 Sep 2023 17:42:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EF468E000F; Tue, 5 Sep 2023 17:42:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8FAE88E000B for ; Tue, 5 Sep 2023 17:42:50 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5FBF9A092C for ; Tue, 5 Sep 2023 21:42:50 +0000 (UTC) X-FDA: 81203868900.03.E713EC6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 4DFAD40024 for ; Tue, 5 Sep 2023 21:42:48 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="faMc/Agw"; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cw3DqXN0nAqgICZcLU+deZuKVK3JziD/s0/YHmFn328=; b=n5RrNoyUIXQmyUjGwxWYasfrLflG+XlJZdcJopYYsuOYKm1c6SJq7oZIS3883GUGlWZ3X/ TekEbvWNxD1ftEvtjn5TAbciyBlQ+2cGxJeRA6LjnsJs2kPs7+heusvgMaJ9zv0NW0JBjo jIPjxtHN/qeECKI5NPu4dqlTS6TSskM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950168; a=rsa-sha256; cv=none; b=uuD17IQa+FrsgZ+ec3a9sND6HmtHZKgZFHSj6hk/KwUxBdd5GpPr52rJsQE3T+dRHjrWv7 1FjeNPBCbrragpBTL1uX/qrae/MTJfy1ryuyjaaX/8wWft5KQGMVyqDWQImNcvqr6zBWVE C78WerWLE6WSTex8YX30TGpSbYGvjNs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="faMc/Agw"; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950167; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cw3DqXN0nAqgICZcLU+deZuKVK3JziD/s0/YHmFn328=; b=faMc/Agws4WT7KtKSTPFZTEP3D8U+bE77P93vXUQAp6iBW1jeHcvnckl/zpuS23N6HzhoW AzzmRb1Q1p8ox9n2d3cFAZoIO8kBsw6xol0qppULsId30oqKBVVeF9pGkFVSLaekpsjPj9 YSauur6BIEzBLTihO0gvseCX/JvlUZ4= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-534-sGJfONGKPXKs7lFZG7c0RQ-1; Tue, 05 Sep 2023 17:42:46 -0400 X-MC-Unique: sGJfONGKPXKs7lFZG7c0RQ-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-76f1cc68e65so91077285a.1 for ; Tue, 05 Sep 2023 14:42:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950166; x=1694554966; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cw3DqXN0nAqgICZcLU+deZuKVK3JziD/s0/YHmFn328=; b=YtJvlgAWk/vpbzbkm9OGnN8BqqbI7GnWZ4sr/6+JxkjscS1cOMM/d0brFquX2Ip8AK b0KTO/VaIf0+co1OvFoYZRo9j32ru1zNDCmm6yTTHViAt3ZWDujEUNSKQ/OLEYUZUYoi EAqV4Z0v/3zxE4d5kcRBzoypbXV6I2DGllU6zj6pGgMITy2aPChUV3LWjk2Ib+cy3ulz m+F9k1LzINrdp+/2Dgynhmup2+SqtVNPGZzdJO6bPW5HHXoKvnhmj9J2XQTytNg3co3M dBx+prlZj1M6/T1rGzh6BA/HkYFniN5WyvERpnL9mwJjbUbdmp8hYgrwaRjgRn4hVq8V suVg== X-Gm-Message-State: AOJu0YyI3rPjs88gjOLic/yaTmGETuxTBfpxw8tyMSWsbQJAqc3a1WvC 0avtHXapHUoc+BGLN2ObivFzYHewoIKzCI8iAd0wnulfc2/t0vvLMapcSAtY6tSupmeAg5BuNaZ s+0LGB1ePHr6m55uQyZY= X-Received: by 2002:a05:620a:4712:b0:76f:1b38:e74a with SMTP id bs18-20020a05620a471200b0076f1b38e74amr15367605qkb.4.1693950165776; Tue, 05 Sep 2023 14:42:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEjHmAQEBZtigx4G4B3sp1ASlOU6CNSzPZeJM4WgpbfHkl1dq3l/QojNcctzg5rIDRzuMLaUA== X-Received: by 2002:a05:620a:4712:b0:76f:1b38:e74a with SMTP id bs18-20020a05620a471200b0076f1b38e74amr15367597qkb.4.1693950165524; Tue, 05 Sep 2023 14:42:45 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:45 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 6/7] selftests/mm: Create uffd_fault_thread_create|join() Date: Tue, 5 Sep 2023 17:42:34 -0400 Message-ID: <20230905214235.320571-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 4DFAD40024 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: c9exbrk1wtnufa7c8zzzzao9jbimiiru X-HE-Tag: 1693950168-693602 X-HE-Meta: U2FsdGVkX184lrZYsoT6WLUChX+Bkt/Fmu6ljh2Wpack+edRoYyGYh2bzseZYeN95GnahQQojdrbQa2cKH6LCpLTI6L1QLguzPzwKduPv8JL1ztcvGtwpjU5HaYH7oqa0JVZPlImyX3AxlriqEN2smqj4gZ0dq5V1JjdnXUGTisux9eoD5qisF3L6XwuzH5BfWXh6ueCT1l86CAsZzhSdGlXGPXV0m9yIwbh55BTXD6ysddw8aDWZI1WCyb6MkF4z3AI70xgSsjtwJf/3ozbzB02jPM2wJx8F0AyKjMdxKf9ltdK4GQ71Z5sEIK9BwfeiUdwEMUpodfE9DyF8YXaHfbQOBt9EATxxu4EWVWP5wUiVALonyhEvy5GPQKuEIzttlJfGmRBZ9lsRNhonF94sPAZxgViR3jrCbyPcDYT6jSyoqZGvZ0/AYVvNgeFIqkQtSLpyxMsKdfjZ2Dt+lZbrS2iy8A8dbS5dhWHeoeU4qWFSYo/+AxSKOpCQo7b4zt/PaS5p+JsxZNYELWT9iS+v1ODS+C4vORmaOJfCS3QZVfSdGdwVIUS3IbwW34B0fJfWtblfBuf73iVac8sV65eSbHE1Cmq0ew/CeS3f0v6eCRmjJIeUBKLIwluPACMTQ6wc5EXF9Pbb1v4SeZkbUfDmCQi9fAghzbhTuu6yPfz4NMNG8KBw5nOIeQop7RhICWS2J1hAZ+dVgT9DosYqCg+RxhHS4U1LlFAX9A5eY7nTM5xxwikVs3NlMPGcXt9cg4i8b5TgOElK/U52YwHAL7ZFN1GyB6EzQBnHQT/rD4wwosmJsfPJLnshSteLXru/fw5fLORd5OH29CEOdbM2PpK0sEYp0qZAzEmow5HHKcg5e/DXH5uFpgIzL+ChHzQMdr5o7TnzMtaHrCgsUtPcWkzEszBB57AP4I39bZIQdtLQo8ntvB40+os1S394GbbVRARqlJ/Yqf3ZhvkbBaXjPJ wXHGnxyz 8gLf7TQnmzOKlP2kktuuNGTK3EDq4UhfbQOmA4ZLsw+1UNEPsEU+qIC5iHwxvyVDygQiFHzzDFXR2aXH2l15XF3yqYLxLcmYCsfzQtaUI5p64DzRFI/kSUurjTdV7WGb39PviPvwxXHNlbjuWHYh1LGPX1KimR0rYQ1imdiJH5sntbJ9iDtRybbR0mNibAewMjNtqkeRGdh3wQizzehWzYitEEbVOXv7tl7/bldGyYLGekR3fo2ZtxdlubQwfsM4k7INpDDTK8eVBHQk+D14INat6dsufFhINQ0+TSSz5mdA/k7B1ejCA5LzUiKopme/TMZLETaGlNcwgaUiiI0WI+lVh3SD0++sh/ag2J/qQBudPBrj+viJymZsBMOVBQtzYL7uviSja32Bylvjb9jeKanUMlrYwwJG1DHZmRZzgYhor9iceoeCet5X2vIodnL+E8NdNYzFiQqO193E7ClLR8n4I276yTSxk0wGmMT4ZgGP4vL7LEam3nXOJHGrQ6gyi19eK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make them common functions to be reused. Signed-off-by: Peter Xu --- tools/testing/selftests/mm/uffd-common.c | 46 ++++++++++++++++++++++ tools/testing/selftests/mm/uffd-common.h | 4 ++ tools/testing/selftests/mm/uffd-stress.c | 49 ++++-------------------- 3 files changed, 57 insertions(+), 42 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index aded06cab285..851284395b29 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -555,6 +555,52 @@ void *uffd_poll_thread(void *arg) return NULL; } +void *uffd_read_thread(void *arg) +{ + struct uffd_args *args = (struct uffd_args *)arg; + struct uffd_msg msg; + + sem_post(&uffd_read_sem); + /* from here cancellation is ok */ + + for (;;) { + if (uffd_read_msg(uffd, &msg)) + continue; + uffd_handle_page_fault(&msg, args); + } + + return NULL; +} + +void uffd_fault_thread_create(pthread_t *thread, pthread_attr_t *attr, + struct uffd_args *args, bool poll) +{ + if (poll) { + if (pthread_create(thread, attr, uffd_poll_thread, args)) + err("uffd_poll_thread create"); + } else { + if (pthread_create(thread, attr, uffd_read_thread, args)) + err("uffd_read_thread create"); + sem_wait(&uffd_read_sem); + } +} + +void uffd_fault_thread_join(pthread_t thread, int cpu, bool poll) +{ + char c = 1; + + if (poll) { + if (write(pipefd[cpu*2+1], &c, 1) != 1) + err("pipefd write error"); + } else { + if (pthread_cancel(thread)) + err("pthread_cancel()"); + } + + if (pthread_join(thread, NULL)) + err("pthread_join()"); +} + static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, unsigned long offset) { diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selftests/mm/uffd-common.h index 521523baded1..9d66ad5c52cb 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -114,6 +114,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); int copy_page(int ufd, unsigned long offset, bool wp); void *uffd_poll_thread(void *arg); +void *uffd_read_thread(void *arg); +void uffd_fault_thread_create(pthread_t *thread, pthread_attr_t *attr, + struct uffd_args *args, bool poll); +void uffd_fault_thread_join(pthread_t thread, int cpu, bool poll); int uffd_open_dev(unsigned int flags); int uffd_open_sys(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selftests/mm/uffd-stress.c index 7219f55ae794..915795e33432 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -125,23 +125,6 @@ static int copy_page_retry(int ufd, unsigned long offset) return __copy_page(ufd, offset, true, test_uffdio_wp); } -static void *uffd_read_thread(void *arg) -{ - struct uffd_args *args = (struct uffd_args *)arg; - struct uffd_msg msg; - - sem_post(&uffd_read_sem); - /* from here cancellation is ok */ - - for (;;) { - if (uffd_read_msg(uffd, &msg)) - continue; - uffd_handle_page_fault(&msg, args); - } - - return NULL; -} - static void *background_thread(void *arg) { unsigned long cpu = (unsigned long) arg; @@ -186,16 +169,10 @@ static int stress(struct uffd_args *args) if (pthread_create(&locking_threads[cpu], &attr, locking_thread, (void *)cpu)) return 1; - if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[cpu])) - err("uffd_poll_thread create"); - } else { - if (pthread_create(&uffd_threads[cpu], &attr, - uffd_read_thread, - (void *)&args[cpu])) - return 1; - sem_wait(&uffd_read_sem); - } + + uffd_fault_thread_create(&uffd_threads[cpu], &attr, + &args[cpu], bounces & BOUNCE_POLL); + if (pthread_create(&background_threads[cpu], &attr, background_thread, (void *)cpu)) return 1; @@ -220,21 +197,9 @@ static int stress(struct uffd_args *args) if (pthread_join(locking_threads[cpu], NULL)) return 1; - for (cpu = 0; cpu < nr_cpus; cpu++) { - char c; - if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) != 1) - err("pipefd write error"); - if (pthread_join(uffd_threads[cpu], - (void *)&args[cpu])) - return 1; - } else { - if (pthread_cancel(uffd_threads[cpu])) - return 1; - if (pthread_join(uffd_threads[cpu], NULL)) - return 1; - } - } + for (cpu = 0; cpu < nr_cpus; cpu++) + uffd_fault_thread_join(uffd_threads[cpu], cpu, + bounces & BOUNCE_POLL); return 0; } From patchwork Tue Sep 5 21:42:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13375099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2697CA100D for ; Tue, 5 Sep 2023 21:42:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 229608E0011; Tue, 5 Sep 2023 17:42:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B24E8E000B; Tue, 5 Sep 2023 17:42:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF9058E0011; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DB7C58E000B for ; Tue, 5 Sep 2023 17:42:51 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AAD03140922 for ; Tue, 5 Sep 2023 21:42:51 +0000 (UTC) X-FDA: 81203868942.20.29C50F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 788974001B for ; Tue, 5 Sep 2023 21:42:49 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JEdXCOat; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693950169; a=rsa-sha256; cv=none; b=rmbXXrDqoobGkJpVubW6GQXy4CQUm5kouFiiPHznyramqWu1KpCA1e8C9qrfmaUGhM+lXE 6ibQJM0DRfMs28OpKioK5wO1uQ3h8Hq5AHFgRybdeWDMQpkbMrHKpb6fW0blnIPcTYuK2g HW721YXAZEhJQ+nXSzrRkKZqvn1C/UY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JEdXCOat; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eXvYIDhk9Xunttv7gLZeZ6m5tePV5F/iEIjQTI0WMa4=; b=rfmZ4ykL80spwnzbw+ANC2K5SSFWz23h2XeaOrFs4ccWMJDxWwoPgeUTNB7Sp+14yN/Hj5 U7ilzWseupB3+aQ2mmUs+u9x6yYwvg2XBYzLQNZqGCvOfsUuuxvLyd3D3De7HajPiPtnjn 6E1duldUqGHycM4ZGHz9ETAMqZFkB6E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693950168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eXvYIDhk9Xunttv7gLZeZ6m5tePV5F/iEIjQTI0WMa4=; b=JEdXCOatA2jMcoVeya76c9mQfTHTUSW3YrzXtrrtXbUmqOSnnKFGu6P7cE5ic+0U0rRdON 4/lZn8QdEExkOaU2MY2KCXhmEmsOPOXf7Iv0qNqWnDd8Epb6lxUjz4NTsdHaExhO7f3xfI eLzkyRLHiyfrK6gQk/yZigEQm/DUEEc= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-73-sVDiqNIaNXCyPXss2LVgrw-1; Tue, 05 Sep 2023 17:42:47 -0400 X-MC-Unique: sVDiqNIaNXCyPXss2LVgrw-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-76f097b28ecso19719385a.1 for ; Tue, 05 Sep 2023 14:42:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693950167; x=1694554967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eXvYIDhk9Xunttv7gLZeZ6m5tePV5F/iEIjQTI0WMa4=; b=fy78zhciwFUwf52RgDYbfA/luyYU+9TtIyT7mvJUCXKSJCDfPWm2ANjCZsYSmn1UxE viuOWxrqg9+r30DweVcJDk5hnmQI3TIKV0s5RRTzYc10mVvhukF3bG2nQM0pmbFveuDG gvREBzdMSE2ofXXKVlMchVhAwLD+lcX7ubzWFPHr6/9G7/QtTowFoLWoT+alHGVl4I5a IavVgFrP2z3EAoukrPkp2BnE/afrlc8P1uDRs3d0AQC2kA08gFWFWxSAnopZNyf3Yvmt bmOruAj6EL5SpAQn8guqjGJJ6f/DFcet9Dx7FU7ojlo1QpgqveYrZmw8qk6qTc7yPtNo CLLA== X-Gm-Message-State: AOJu0YzugkEKBL8EzzQfF6zkHEt1gUUXKlqGnfL94LERU6w13yM7+Cnb h0od/eOt8oqMrNMrdJ3hy/Eey0NKs9bKpWbfMWiMUj5IApd8PEPaK7ur0Dh5VH68XSBG9MPRTbx EFnEMjolm+JY= X-Received: by 2002:a05:620a:1a26:b0:76c:ed4e:ac10 with SMTP id bk38-20020a05620a1a2600b0076ced4eac10mr16770227qkb.6.1693950167013; Tue, 05 Sep 2023 14:42:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEGbQRi3lj2GCHZMrI9G7N8PmFSehzLMjlA7R138X4fKreXMS+kXyyfOvFWMrTvBreLVK2oMg== X-Received: by 2002:a05:620a:1a26:b0:76c:ed4e:ac10 with SMTP id bk38-20020a05620a1a2600b0076ced4eac10mr16770206qkb.6.1693950166761; Tue, 05 Sep 2023 14:42:46 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id i2-20020a37c202000000b007682af2c8aasm4396938qkm.126.2023.09.05.14.42.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Sep 2023 14:42:46 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Anish Moorthy , Axel Rasmussen , Alexander Viro , Mike Kravetz , Peter Zijlstra , Andrew Morton , Mike Rapoport , Christian Brauner , peterx@redhat.com, linux-fsdevel@vger.kernel.org, Andrea Arcangeli , Ingo Molnar , James Houghton , Nadav Amit Subject: [PATCH 7/7] selftests/mm: uffd perf test Date: Tue, 5 Sep 2023 17:42:35 -0400 Message-ID: <20230905214235.320571-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214235.320571-1-peterx@redhat.com> References: <20230905214235.320571-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 788974001B X-Stat-Signature: 3zi8rzpzf56e3y3sgzjj4awn6ipfh19y X-Rspam-User: X-HE-Tag: 1693950169-948210 X-HE-Meta: U2FsdGVkX19+a9NzoQsr4jRGwtSwKUC3arBWKX5YeMAYtcgibuft0zwceGtpon6PUXJNT3SnZhH/nGqKXm8zoa2S+VDo4N/13mlskvdCucHKyQZ6jqQbHJXHmurn7IFX2slt0jmub113tY1Dc7y5KQwyi4NTBxM6A2MY52F1dAFrxsHIPeqnAO0HziPty/cYgGVoGEv9hKcDxlm/qgp8hr/bQnIyBo47BQogDf3plRb885BomsyhoocdM1McDLPp9UEl6l2Y5XvAFTDVCM86mX260khY3stPxzlzIiHozdte+L9f+fwyYTEr45N1ia3BftJ0QChzCelP4O2gcNSWAeabVTzwzw6x7i75W+R7Si708jlrkyZC+iVsM0Kee4uzqukAiCEGZF7oPn1HiBBa29dep/mBnJZNNO/Bm1rskV24Pq2jbp5QUEvmu4lZirwh7ebD2gAFl/lieDZL7DGbz1+/fbOsw72LQBVWm6viRC53vES2LOJdFYrPnbKXVtU9T1N0ESVM0pb0OKOK8gme97tZ+ijIFjIN1H6zO5z0t/Eec0ZGwN/nH4/dDtnewsFcnV1ZyaifwHBYJkF2mLDNGdpGRFNae4k6789ONhFTWGbdu7hZl5ry9/x6z3x2cndMxltZVNeESp9R9Zos0ozp/2Qh9L4rM84yDXCNIUWrUZcA4O+7JwF/iJmBpRXe6RcnO/BI5k/q7Wx+NzYh/DpBzdjjoqlToJ1BDR1zH++4kcPCzWGZ6ggZ+yuQvbCMvLRYAuhTnZGTVlVdzAfvi9wrW1ajwCOAGkCaKhaVpD3I4Wf6WwRxYDsPs9lfC3SoS+VwyGTztd7ILNtiO0vzwPpDyO8DkfIRUc2O/YNvrRo4mL7pPx4jK2PONxlznEOMiGXeDFpp305YWDFvsyczRqNWzunTHqIFfQ66bhcWvnqyIeFSjzFkfJu0UNoK0ixDC+FfSFk8/h63qowxYt5wMPU WJaawhTN BB25UMahvn4PHQ5JM7+4u7U2HI2YaZbRLQ4KWSki5pf0hdxq4JjK2kJxfzfKx/uRB1gE13gs62MBG5OMqiLxr5ztg+8oxwBasOMZlGWOuByKp3IzqdZHg+NMiyr1m6RlUPn9Adlm5yod08eFoi4cCqOUsaH3Kj1oDe1O+DO3g7vEryFL5Y8ogIANXDd21Jlfz5vy8JkEjRdVQgthoESgAJ7TNSRV1YFzXMNuoqSqqziV/ESSH6JUFZbQ5cbMhGIzjsF8Da0FB66B6P+dUFtRyBq1/Xx2+gFf32yj/X7KawFZXvnm5dRhhNlPKMbnackocH7indkkpf3QpSwL1Eayobb9MTQcRbHl2hpkwxT8wsLyJL8127XFwIzbXj76djg+bRlH+YLGndaJg6TrDheb/T4sof8MB3Ig2c7bNfWWdvkGeTbNaI16hepEbHGIJewFVuKhrCC4dnkfeyF2VSgxNgZcOwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a simple perf test for userfaultfd missing mode, on private anon only. It mostly only tests the messaging, so memory type / fault type may not that much yet. Signed-off-by: Peter Xu --- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/uffd-common.c | 18 ++ tools/testing/selftests/mm/uffd-common.h | 1 + tools/testing/selftests/mm/uffd-perf.c | 207 +++++++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 tools/testing/selftests/mm/uffd-perf.c diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 6a9fc5693145..acb22517d37e 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -64,6 +64,7 @@ TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress TEST_GEN_FILES += uffd-stress TEST_GEN_FILES += uffd-unit-tests +TEST_GEN_FILES += uffd-perf TEST_GEN_FILES += split_huge_page_test TEST_GEN_FILES += ksm_tests TEST_GEN_FILES += ksm_functional_tests @@ -120,6 +121,7 @@ $(TEST_GEN_FILES): vm_util.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c +$(OUTPUT)/uffd-perf: uffd-common.c ifeq ($(ARCH),x86_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index 851284395b29..afbf2f7add56 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -725,3 +725,21 @@ int uffd_get_features(uint64_t *features) return 0; } + +uint64_t get_usec(void) +{ + uint64_t val = 0; + struct timespec t; + int ret = clock_gettime(CLOCK_MONOTONIC, &t); + + if (ret == -1) { + perror("clock_gettime() failed"); + /* should never happen */ + exit(-1); + } + + val = t.tv_nsec / 1000; /* ns -> us */ + val += t.tv_sec * 1000000; /* s -> us */ + + return val; +} diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selftests/mm/uffd-common.h index 9d66ad5c52cb..4273201ae19f 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -123,6 +123,7 @@ int uffd_open_dev(unsigned int flags); int uffd_open_sys(unsigned int flags); int uffd_open(unsigned int flags); int uffd_get_features(uint64_t *features); +uint64_t get_usec(void); #define TEST_ANON 1 #define TEST_HUGETLB 2 diff --git a/tools/testing/selftests/mm/uffd-perf.c b/tools/testing/selftests/mm/uffd-perf.c new file mode 100644 index 000000000000..eda99718311a --- /dev/null +++ b/tools/testing/selftests/mm/uffd-perf.c @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Userfaultfd performance tests. + * + * Copyright (C) 2023 Red Hat, Inc. + */ + +#include "uffd-common.h" + +#ifdef __NR_userfaultfd + +#define DEF_MEM_SIZE_MB (512) +#define MB(x) ((x) * 1024 * 1024) +#define DEF_N_TESTS 5 + +static volatile bool perf_test_started; +static unsigned int n_uffd_threads, n_worker_threads; +static uint64_t nr_pages_per_worker; +static unsigned long n_tests = DEF_N_TESTS; + +static void setup_env(unsigned long mem_size_mb) +{ + /* Test private anon only for now */ + map_shared = false; + uffd_test_ops = &anon_uffd_test_ops; + page_size = psize(); + nr_cpus = n_uffd_threads; + nr_pages = MB(mem_size_mb) / page_size; + nr_pages_per_worker = nr_pages / n_worker_threads; + if (nr_pages_per_worker == 0) + err("each worker should at least own one page"); +} + +void *worker_fn(void *opaque) +{ + unsigned long i = (unsigned long) opaque; + unsigned long page_nr, start_nr, end_nr; + int v = 0; + + start_nr = i * nr_pages_per_worker; + end_nr = (i + 1) * nr_pages_per_worker; + + while (!perf_test_started); + + for (page_nr = start_nr; page_nr < end_nr; page_nr++) + v += *(volatile int *)(area_dst + page_nr * page_size); + + return NULL; +} + +static uint64_t run_perf(uint64_t mem_size_mb, bool poll) +{ + pthread_t worker_threads[n_worker_threads]; + pthread_t uffd_threads[n_uffd_threads]; + const char *errmsg = NULL; + struct uffd_args *args; + uint64_t start, end; + int i, ret; + + if (uffd_test_ctx_init(0, &errmsg)) + err("%s", errmsg); + + /* + * By default, uffd is opened with NONBLOCK mode; use block mode + * when test read() + */ + if (!poll) { + int flags = fcntl(uffd, F_GETFL); + + if (flags < 0) + err("fcntl(F_GETFL) failed"); + + if (flags & O_NONBLOCK) + flags &= ~O_NONBLOCK; + + if (fcntl(uffd, F_SETFL, flags)) + err("fcntl(F_SETFL) failed"); + } + + ret = uffd_register(uffd, area_dst, MB(mem_size_mb), + true, false, false); + if (ret) + err("uffd_register() failed"); + + args = calloc(nr_cpus, sizeof(struct uffd_args)); + if (!args) + err("calloc()"); + + for (i = 0; i < n_uffd_threads; i++) { + args[i].cpu = i; + uffd_fault_thread_create(&uffd_threads[i], NULL, + &args[i], poll); + } + + for (i = 0; i < n_worker_threads; i++) { + if (pthread_create(&worker_threads[i], NULL, + worker_fn, (void *)(uintptr_t)i)) + err("create uffd threads"); + } + + start = get_usec(); + perf_test_started = true; + for (i = 0; i < n_worker_threads; i++) + pthread_join(worker_threads[i], NULL); + end = get_usec(); + + for (i = 0; i < n_uffd_threads; i++) { + struct uffd_args *p = &args[i]; + + uffd_fault_thread_join(uffd_threads[i], i, poll); + + assert(p->wp_faults == 0 && p->minor_faults == 0); + } + + free(args); + + ret = uffd_unregister(uffd, area_dst, MB(mem_size_mb)); + if (ret) + err("uffd_unregister() failed"); + + return end - start; +} + +static void usage(const char *prog) +{ + printf("usage: %s \n", prog); + puts(""); + printf(" -m: size of memory to test (in MB, default: %u)\n", + DEF_MEM_SIZE_MB); + puts(" -p: use poll() (the default)"); + puts(" -r: use read()"); + printf(" -t: test rounds (default: %u)\n", DEF_N_TESTS); + puts(" -u: number of uffd threads (default: n_cpus)"); + puts(" -w: number of worker threads (default: n_cpus)"); + puts(""); + exit(KSFT_FAIL); +} + +int main(int argc, char *argv[]) +{ + unsigned long mem_size_mb = DEF_MEM_SIZE_MB; + uint64_t result, sum = 0; + bool use_poll = true; + int opt, count; + + n_uffd_threads = n_worker_threads = sysconf(_SC_NPROCESSORS_ONLN); + + while ((opt = getopt(argc, argv, "hm:prt:u:w:")) != -1) { + switch (opt) { + case 'm': + mem_size_mb = strtoul(optarg, NULL, 10); + break; + case 'p': + use_poll = true; + break; + case 'r': + use_poll = false; + break; + case 't': + n_tests = strtoul(optarg, NULL, 10); + break; + case 'u': + n_uffd_threads = strtoul(optarg, NULL, 10); + break; + case 'w': + n_worker_threads = strtoul(optarg, NULL, 10); + break; + case 'h': + default: + /* Unknown */ + usage(argv[0]); + break; + } + } + + setup_env(mem_size_mb); + + printf("Message mode: \t\t%s\n", use_poll ? "poll" : "read"); + printf("Mem size: \t\t%lu (MB)\n", mem_size_mb); + printf("Uffd threads: \t\t%u\n", n_uffd_threads); + printf("Worker threads: \t%u\n", n_worker_threads); + printf("Test rounds: \t\t%lu\n", n_tests); + printf("Time used (us): \t"); + + for (count = 0; count < n_tests; count++) { + result = run_perf(mem_size_mb, use_poll); + sum += result; + printf("%" PRIu64 ", ", result); + fflush(stdout); + } + printf("\b\b \n"); + printf("Average (us): \t\t%"PRIu64"\n", sum / n_tests); + + return KSFT_PASS; +} + +#else /* __NR_userfaultfd */ + +#warning "missing __NR_userfaultfd definition" + +int main(void) +{ + printf("Skipping %s (missing __NR_userfaultfd)\n", __file__); + return KSFT_SKIP; +} + +#endif /* __NR_userfaultfd */