From patchwork Thu Feb 6 03:08:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 11367599 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4828014D5 for ; Thu, 6 Feb 2020 03:13:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0718920720 for ; Thu, 6 Feb 2020 03:13:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0718920720 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 400B26B0007; Wed, 5 Feb 2020 22:13:03 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 389F16B000A; Wed, 5 Feb 2020 22:13:03 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24F026B000C; Wed, 5 Feb 2020 22:13:03 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0155.hostedemail.com [216.40.44.155]) by kanga.kvack.org (Postfix) with ESMTP id 0A0C56B0007 for ; Wed, 5 Feb 2020 22:13:03 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B3BFF181AEF10 for ; Thu, 6 Feb 2020 03:13:02 +0000 (UTC) X-FDA: 76458230604.06.trade25_71d61e2f58813 X-Spam-Summary: 2,0,0,5091869980584a31,d41d8cd98f00b204,leonardo@linux.ibm.com,:benh@kernel.crashing.org:paulus@samba.org:mpe@ellerman.id.au:arnd@arndb.de:akpm@linux-foundation.org:aneesh.kumar@linux.ibm.com:npiggin@gmail.com:christophe.leroy@c-s.fr:steven.price@arm.com:robin.murphy@arm.com:leonardo@linux.ibm.com:mahesh@linux.vnet.ibm.com:bsingharora@gmail.com:arbab@linux.ibm.com:tglx@linutronix.de:allison@lohutok.net:gregkh@linuxfoundation.org:rppt@linux.ibm.com:msuchanek@suse.de:linuxppc-dev@lists.ozlabs.org:linux-kernel@vger.kernel.org:kvm-ppc@vger.kernel.org:linux-arch@vger.kernel.org:,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1261:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:5007:6261:6737:7875:7903:8634:8784:10004:11026:11232:11473:11657:11658:11914:12043:12048:12297:12438:12555:12663:12895:12986:13146:13230:13255:13894:14096:14181:14394:14721:2 1080:213 X-HE-Tag: trade25_71d61e2f58813 X-Filterd-Recvd-Size: 6835 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Thu, 6 Feb 2020 03:13:02 +0000 (UTC) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 01639EMI096736; Wed, 5 Feb 2020 22:12:53 -0500 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 2xyhn60kte-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 22:12:53 -0500 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 0163Bgpt101390; Wed, 5 Feb 2020 22:12:52 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0b-001b2d01.pphosted.com with ESMTP id 2xyhn60kt2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 05 Feb 2020 22:12:52 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 0163BmQl013924; Thu, 6 Feb 2020 03:12:51 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma01dal.us.ibm.com with ESMTP id 2xykc9489r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Feb 2020 03:12:51 +0000 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0163Cnb937028238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Feb 2020 03:12:50 GMT Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DE62EBE053; Thu, 6 Feb 2020 03:12:49 +0000 (GMT) Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A1A82BE051; Thu, 6 Feb 2020 03:12:33 +0000 (GMT) Received: from LeoBras.aus.stglabs.ibm.com (unknown [9.85.163.250]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 6 Feb 2020 03:12:33 +0000 (GMT) From: Leonardo Bras To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Arnd Bergmann , Andrew Morton , "Aneesh Kumar K.V" , Nicholas Piggin , Christophe Leroy , Steven Price , Robin Murphy , Leonardo Bras , Mahesh Salgaonkar , Balbir Singh , Reza Arbab , Thomas Gleixner , Allison Randal , Greg Kroah-Hartman , Mike Rapoport , Michal Suchanek Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kvm-ppc@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v6 10/11] powerpc/mm: Adds counting method to track lockless pagetable walks Date: Thu, 6 Feb 2020 00:08:59 -0300 Message-Id: <20200206030900.147032-11-leonardo@linux.ibm.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200206030900.147032-1-leonardo@linux.ibm.com> References: <20200206030900.147032-1-leonardo@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572 definitions=2020-02-05_06:2020-02-04,2020-02-05 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 priorityscore=1501 adultscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 spamscore=0 lowpriorityscore=0 bulkscore=0 clxscore=1015 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002060022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implements an additional feature to track lockless pagetable walks, using a per-cpu counter: lockless_pgtbl_walk_counter. Before a lockless pagetable walk, preemption is disabled and the current cpu's counter is increased. When the lockless pagetable walk finishes, the current cpu counter is decreased and the preemption is enabled. With that, it's possible to know in which cpus are happening lockless pagetable walks, and optimize serialize_against_pte_lookup(). Implementation notes: - Every counter can be changed only by it's CPU - It makes use of the original memory barrier in the functions - Any counter can be read by any CPU Due to not locking nor using atomic variables, the impact on the lockless pagetable walk is intended to be minimum. Signed-off-by: Leonardo Bras --- arch/powerpc/mm/book3s64/pgtable.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 535613030363..bb138b628f86 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -83,6 +83,7 @@ static void do_nothing(void *unused) } +static DEFINE_PER_CPU(int, lockless_pgtbl_walk_counter); /* * Serialize against find_current_mm_pte which does lock-less * lookup in page tables with local interrupts disabled. For huge pages @@ -120,6 +121,15 @@ unsigned long __begin_lockless_pgtbl_walk(bool disable_irq) if (disable_irq) local_irq_save(irq_mask); + /* + * Counts this instance of lockless pagetable walk for this cpu. + * Disables preempt to make sure there is no cpu change between + * begin/end lockless pagetable walk, so that percpu counting + * works fine. + */ + preempt_disable(); + (*this_cpu_ptr(&lockless_pgtbl_walk_counter))++; + /* * This memory barrier pairs with any code that is either trying to * delete page tables, or split huge pages. Without this barrier, @@ -158,6 +168,14 @@ inline void __end_lockless_pgtbl_walk(unsigned long irq_mask, bool enable_irq) */ smp_mb(); + /* + * Removes this instance of lockless pagetable walk for this cpu. + * Enables preempt only after end lockless pagetable walk, + * so that percpu counting works fine. + */ + (*this_cpu_ptr(&lockless_pgtbl_walk_counter))--; + preempt_enable(); + /* * Interrupts must be disabled during the lockless page table walk. * That's because the deleting or splitting involves flushing TLBs,