From patchwork Tue Sep 11 19:18:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10596201 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4638F14BD for ; Tue, 11 Sep 2018 19:18:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FFF929BD9 for ; Tue, 11 Sep 2018 19:18:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3372029D0D; Tue, 11 Sep 2018 19:18:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8A9629BD9 for ; Tue, 11 Sep 2018 19:18:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4F328E0007; Tue, 11 Sep 2018 15:18:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CD6098E0001; Tue, 11 Sep 2018 15:18:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE9908E0007; Tue, 11 Sep 2018 15:18:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 8C2A38E0001 for ; Tue, 11 Sep 2018 15:18:40 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id n23-v6so21907647qkn.19 for ; Tue, 11 Sep 2018 12:18:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=jWlOFtELmXfxC3ynjpZuAA8uJ4Rm+lS3xewNX5/reIA=; b=o9glMdZFTzuh60EeNfWc4M3DW8oKY8qxEx6RWksbayYYFx98H6gfJf9LQpluAEy/3e H13hMGipOQINhWunFpPoHDO/a9tmOs6MrcDDEc8GTF8RtowXKmxfv8wkgGN3um6PGxp8 XreEn4r/pXddl38utaPLMm0kMi6nSSSYTIZMB9PAFuD1lVM7TnH2TXHhVhDKZj7ZtYfZ lATiZYZqnXzM2sule/e1NbJPMOGA2dSOLrKk6RoDIsm0SgckfGOcqnVIoLL06GAURf68 uKaq1gqABjmZpmRGoW8pKZ2S8rhoSCB6W9Xi2Wj3Ny3qj7kJU0SveLPbZeUOoWZAAAc3 UM7Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APzg51BzkxlS6EkpBARvZ0CpjkiLfsOkiEvx4tU/b5TAn8eVuNMEkT9O HpVr4rYtcDXdpddaXDqNdJYc/FevYly/muEcxGiwhX6+tPBIXM4mWwkuDjpZVY7LriAgpJ6c6Fh ianicVfidM+8OAvGItxzHz6E5Z+GHwE/I7q/Z5pRCkJJBcBVrLizDb4gwFS05TOimJQ== X-Received: by 2002:a0c:d135:: with SMTP id a50-v6mr19352683qvh.112.1536693520306; Tue, 11 Sep 2018 12:18:40 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbNm4zC0mY/qhwxKrf3dNwuFWMhk0UOqzQ1GaHhCP6heJKoBi285MBxpT5eEz1YZb8+Pdf9 X-Received: by 2002:a0c:d135:: with SMTP id a50-v6mr19352644qvh.112.1536693519398; Tue, 11 Sep 2018 12:18:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536693519; cv=none; d=google.com; s=arc-20160816; b=C4ADSy/o1oWp5JTZyxGtMNUUxso4mItfYn/tuENR8X+unHC2k/Dwv+iYX3k+vCx1rt q7uaC1O0CHq5YTgLRDALfnehPu5CGUm/vxG8HI7GJnh5iIk26PXbI4ZYC1yWyQp/aPyk DePK6a5gbOvENxoKEsgOZeS5ITP8LnKj2VRvEqKKtfhhcPr+NsvVEr9mHE1dgp1SH4Zn t3ZGehNg2Z02D34vbqEpXrQUtGwC3gNB2VeWweJXepf8GJKzYdjJa98qOJIOjnGmw05x D/6EWyh3r/UPAAFAhPX/51uIgZ/r/ZwaFdgShwFUh4krynJyfqZkx5umusoKVG0GcnXb wqTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=jWlOFtELmXfxC3ynjpZuAA8uJ4Rm+lS3xewNX5/reIA=; b=JZF4VWJEv0su82SksTmJdHC4aD7tHrd+lmzK+gS/leEDwxbTmIbSfWAtmmP3SRLqlg PpYkb94DkEyJVijiqcE/XTY5lHANoP791S0zkhvoZcR133dWXYueUFcRO8wQwKzb/NTy 0lbm84qLzbuKmjNDQRENrM0t0V9KbwBMSrLvjpJY+fnRORKjV4CcEv2amnpPt5lie6RK SJSE2g6yUazmVw2eltnj+Z8KEVtdpsUJbjRONwgbv7/LfFXATbTuM3HmZSKnTnDqUQjp s4m3UrSBYVY1fzYOf7WaoYp7aur1MVPh7WPk8iQKhQl9c4TEoPgSUULXR6Hiq0L9aLip aGIg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id o7-v6si403246qtb.200.2018.09.11.12.18.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 12:18:39 -0700 (PDT) Received-SPF: pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) client-ip=66.187.233.73; Authentication-Results: mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E1DC440241C7; Tue, 11 Sep 2018 19:18:38 +0000 (UTC) Received: from llong.com (ovpn-123-172.rdu2.redhat.com [10.10.123.172]) by smtp.corp.redhat.com (Postfix) with ESMTP id CA3542156899; Tue, 11 Sep 2018 19:18:37 +0000 (UTC) From: Waiman Long To: Alexander Viro , Jonathan Corbet Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, "Luis R. Rodriguez" , Kees Cook , Linus Torvalds , Jan Kara , "Paul E. McKenney" , Andrew Morton , Ingo Molnar , Miklos Szeredi , Matthew Wilcox , Larry Woodman , James Bottomley , "Wangkai (Kevin C)" , Michal Hocko , Waiman Long Subject: [PATCH v3 4/4] fs/dcache: Eliminate branches in nr_dentry_negative accounting Date: Tue, 11 Sep 2018 15:18:26 -0400 Message-Id: <1536693506-11949-5-git-send-email-longman@redhat.com> In-Reply-To: <1536693506-11949-1-git-send-email-longman@redhat.com> References: <1536693506-11949-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Tue, 11 Sep 2018 19:18:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Tue, 11 Sep 2018 19:18:39 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'longman@redhat.com' RCPT:'' X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Because the accounting of nr_dentry_negative depends on whether a dentry is a negative one or not, branch instructions are introduced to handle the accounting conditionally. That may potentially slow down the task by a noticeable amount if that introduces sizeable amount of additional branch mispredictions. To avoid that, the accounting code is now modified to use conditional move instructions instead, if supported by the architecture. Signed-off-by: Waiman Long --- fs/dcache.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index c1cc956..dfd5628 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -171,6 +171,29 @@ int proc_nr_dentry(struct ctl_table *table, int write, void __user *buffer, dentry_stat.nr_negative = get_nr_dentry_negative(); return proc_doulongvec_minmax(table, write, buffer, lenp, ppos); } + +/* + * Increment/Decrement nr_dentry_negative if the condition is true. + * For architectures that support some kind of conditional move, compiler + * should be able generate code to inc/dec negative dentry counter + * without any branch instruction. + */ +static inline void cond_negative_dentry_inc(bool cond) +{ + int val = !!cond; + + this_cpu_add(nr_dentry_negative, val); +} + +static inline void cond_negative_dentry_dec(bool cond) +{ + int val = !!cond; + + this_cpu_sub(nr_dentry_negative, val); +} +#else +static inline void cond_negative_dentry_inc(bool cond) { } +static inline void cond_negative_dentry_dec(bool cond) { } #endif /* @@ -343,8 +366,7 @@ static inline void __d_clear_type_and_inode(struct dentry *dentry) flags &= ~(DCACHE_ENTRY_TYPE | DCACHE_FALLTHRU); WRITE_ONCE(dentry->d_flags, flags); dentry->d_inode = NULL; - if (dentry->d_flags & DCACHE_LRU_LIST) - this_cpu_inc(nr_dentry_negative); + cond_negative_dentry_inc(dentry->d_flags & DCACHE_LRU_LIST); } static void dentry_free(struct dentry *dentry) @@ -412,8 +434,7 @@ static void d_lru_add(struct dentry *dentry) D_FLAG_VERIFY(dentry, 0); dentry->d_flags |= DCACHE_LRU_LIST; this_cpu_inc(nr_dentry_unused); - if (d_is_negative(dentry)) - this_cpu_inc(nr_dentry_negative); + cond_negative_dentry_inc(d_is_negative(dentry)); WARN_ON_ONCE(!list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } @@ -422,8 +443,7 @@ static void d_lru_del(struct dentry *dentry) D_FLAG_VERIFY(dentry, DCACHE_LRU_LIST); dentry->d_flags &= ~DCACHE_LRU_LIST; this_cpu_dec(nr_dentry_unused); - if (d_is_negative(dentry)) - this_cpu_dec(nr_dentry_negative); + cond_negative_dentry_dec(d_is_negative(dentry)); WARN_ON_ONCE(!list_lru_del(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } @@ -454,8 +474,7 @@ static void d_lru_isolate(struct list_lru_one *lru, struct dentry *dentry) D_FLAG_VERIFY(dentry, DCACHE_LRU_LIST); dentry->d_flags &= ~DCACHE_LRU_LIST; this_cpu_dec(nr_dentry_unused); - if (d_is_negative(dentry)) - this_cpu_dec(nr_dentry_negative); + cond_negative_dentry_dec(d_is_negative(dentry)); list_lru_isolate(lru, &dentry->d_lru); } @@ -464,8 +483,7 @@ static void d_lru_shrink_move(struct list_lru_one *lru, struct dentry *dentry, { D_FLAG_VERIFY(dentry, DCACHE_LRU_LIST); dentry->d_flags |= DCACHE_SHRINK_LIST; - if (d_is_negative(dentry)) - this_cpu_dec(nr_dentry_negative); + cond_negative_dentry_dec(d_is_negative(dentry)); list_lru_isolate_move(lru, &dentry->d_lru, list); } @@ -1865,8 +1883,7 @@ static void __d_instantiate(struct dentry *dentry, struct inode *inode) /* * Decrement negative dentry count if it was in the LRU list. */ - if (dentry->d_flags & DCACHE_LRU_LIST) - this_cpu_dec(nr_dentry_negative); + cond_negative_dentry_dec(dentry->d_flags & DCACHE_LRU_LIST); hlist_add_head(&dentry->d_u.d_alias, &inode->i_dentry); raw_write_seqcount_begin(&dentry->d_seq); __d_set_inode_and_type(dentry, inode, add_flags);