From patchwork Sun Mar 12 06:29:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 9619241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 515DF604A9 for ; Sun, 12 Mar 2017 06:32:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35D3A28428 for ; Sun, 12 Mar 2017 06:32:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 17AD12843B; Sun, 12 Mar 2017 06:32:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 881B028428 for ; Sun, 12 Mar 2017 06:32:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755694AbdCLGcM (ORCPT ); Sun, 12 Mar 2017 01:32:12 -0500 Received: from mail-pf0-f194.google.com ([209.85.192.194]:35247 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750899AbdCLGcL (ORCPT ); Sun, 12 Mar 2017 01:32:11 -0500 Received: by mail-pf0-f194.google.com with SMTP id 67so15508170pfg.2; Sat, 11 Mar 2017 22:31:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=sSBeVEIijD1tugo72Txl6S/2gXoktgMVCChDFUhjSx0=; b=fQ6Qn27aL9O+nAHzKfZCPplhTd4MGtiN66HOG8JIpMxOcxAGbnc/ZTqET+452ruW0W GwJjMGLXXfLICNwL6wuZpbtbe88ztBpwUEwoVMxKcZet8G2PXziQ2KhSyNoKA4Dy9LYZ fhBwASYO+apCgM0syRCnlKc72AxySXbYYhNL3NWpfJ/0z/l1HfbPS3/K30/Umig6uRGN 75yj9xjM+yf2lO9hjQe7Dfq7YFAVeI3fGCfS1x57RQBiEZ7VYexCNxG9GMmDbftot5Rc wm/37ur3Ho+44Ev0iCfjQSp6rWHtuOaUMVAHxPk6OjndAMKynz0FfJKxEc9kQmc7i5UA TNAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=sSBeVEIijD1tugo72Txl6S/2gXoktgMVCChDFUhjSx0=; b=owOLeN+yzw+Blu5NVk+Jpd5L00EZHBNbej48fBpguLDy2kQXmF1qQZGO0RxQw2M9/U ZF8FMCZYzemO0ZaDVfDVJkgkH6F69TsoABpMS82VtjOGdjUhT0tyYRZutnvJnr+LfrmB FDk9FlbvzEOH2b702Yysq3dJXN7uHWJAzE1Y8upBimpjUYZc31pMLVYHikJZOks8fA0z LZlUxoXNjYA8M01hP3Odk9iGAm88ZAGEfdA+0Tl0D439o+nBqrFA3KN6kghEjYv9frCQ rFh3umtG5D6pwn33mHAbivX0D6k/G5NntN5TYzkkg5bCVKEdXg5cnGd0cp4USbcW/g9v xzSw== X-Gm-Message-State: AMke39kXnr4i76scAqf548ERV8CZ0jfKx4UcL4Es4i8mR1Uzr32NLRwKIeIrrVcVoAIv4Q== X-Received: by 10.98.86.152 with SMTP id h24mr31022647pfj.184.1489300289740; Sat, 11 Mar 2017 22:31:29 -0800 (PST) Received: from zzz.hsd1.wa.comcast.net (c-73-239-167-150.hsd1.wa.comcast.net. [73.239.167.150]) by smtp.gmail.com with ESMTPSA id x21sm27164864pgf.15.2017.03.11.22.31.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Mar 2017 22:31:28 -0800 (PST) From: Eric Biggers To: linux-fsdevel@vger.kernel.org Cc: Al Viro , David Howells , linux-kernel@vger.kernel.org, Eric Biggers Subject: [PATCH v3] statx: optimize copy of struct statx to userspace Date: Sat, 11 Mar 2017 22:29:41 -0800 Message-Id: <20170312062941.862-1-ebiggers3@gmail.com> X-Mailer: git-send-email 2.12.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers I found that statx() was significantly slower than stat(). As a microbenchmark, I compared 10,000,000 invocations of fstat() on a tmpfs file to the same with statx() passed a NULL path: $ time ./stat_benchmark real 0m1.464s user 0m0.275s sys 0m1.187s $ time ./statx_benchmark real 0m5.530s user 0m0.281s sys 0m5.247s statx is expected to be a little slower than stat because struct statx is larger than struct stat, but not by *that* much. It turns out that most of the overhead was in copying struct statx to userspace, mostly in all the stac/clac instructions that got generated for each __put_user() call. (This was on x86_64, but some other architectures, e.g. arm64, have something similar now too.) stat() instead initializes its struct on the stack and copies it to userspace with a single call to copy_to_user(). This turns out to be much faster, and changing statx to do this makes it almost as fast as stat: $ time ./statx_benchmark real 0m1.624s user 0m0.270s sys 0m1.354s Implementation-wise, for zeroing the reserved fields, I chose to simply start by zeroing the full struct with memset. This makes it clear that every byte copied to userspace is initialized, even implicit padding bytes (though there are none currently). In the scenarios I tested, it also performed the same as a designated initializer. Manually initializing each field was still slightly faster, but would have been more error-prone and less verifiable. Signed-off-by: Eric Biggers --- fs/stat.c | 73 +++++++++++++++++++++++++++------------------------------------ 1 file changed, 31 insertions(+), 42 deletions(-) diff --git a/fs/stat.c b/fs/stat.c index fa0be59340cc..3fbecbfa6975 100644 --- a/fs/stat.c +++ b/fs/stat.c @@ -509,46 +509,36 @@ SYSCALL_DEFINE4(fstatat64, int, dfd, const char __user *, filename, } #endif /* __ARCH_WANT_STAT64 || __ARCH_WANT_COMPAT_STAT64 */ -static inline int __put_timestamp(struct timespec *kts, - struct statx_timestamp __user *uts) +static int cp_statx(const struct kstat *stat, struct statx __user *buffer) { - return (__put_user(kts->tv_sec, &uts->tv_sec ) || - __put_user(kts->tv_nsec, &uts->tv_nsec ) || - __put_user(0, &uts->__reserved )); -} - -/* - * Set the statx results. - */ -static long statx_set_result(struct kstat *stat, struct statx __user *buffer) -{ - uid_t uid = from_kuid_munged(current_user_ns(), stat->uid); - gid_t gid = from_kgid_munged(current_user_ns(), stat->gid); - - if (__put_user(stat->result_mask, &buffer->stx_mask ) || - __put_user(stat->mode, &buffer->stx_mode ) || - __clear_user(&buffer->__spare0, sizeof(buffer->__spare0)) || - __put_user(stat->nlink, &buffer->stx_nlink ) || - __put_user(uid, &buffer->stx_uid ) || - __put_user(gid, &buffer->stx_gid ) || - __put_user(stat->attributes, &buffer->stx_attributes ) || - __put_user(stat->blksize, &buffer->stx_blksize ) || - __put_user(MAJOR(stat->rdev), &buffer->stx_rdev_major ) || - __put_user(MINOR(stat->rdev), &buffer->stx_rdev_minor ) || - __put_user(MAJOR(stat->dev), &buffer->stx_dev_major ) || - __put_user(MINOR(stat->dev), &buffer->stx_dev_minor ) || - __put_timestamp(&stat->atime, &buffer->stx_atime ) || - __put_timestamp(&stat->btime, &buffer->stx_btime ) || - __put_timestamp(&stat->ctime, &buffer->stx_ctime ) || - __put_timestamp(&stat->mtime, &buffer->stx_mtime ) || - __put_user(stat->ino, &buffer->stx_ino ) || - __put_user(stat->size, &buffer->stx_size ) || - __put_user(stat->blocks, &buffer->stx_blocks ) || - __clear_user(&buffer->__spare1, sizeof(buffer->__spare1)) || - __clear_user(&buffer->__spare2, sizeof(buffer->__spare2))) - return -EFAULT; - - return 0; + struct statx tmp; + + memset(&tmp, 0, sizeof(tmp)); + + tmp.stx_mask = stat->result_mask; + tmp.stx_blksize = stat->blksize; + tmp.stx_attributes = stat->attributes; + tmp.stx_nlink = stat->nlink; + tmp.stx_uid = from_kuid_munged(current_user_ns(), stat->uid); + tmp.stx_gid = from_kgid_munged(current_user_ns(), stat->gid); + tmp.stx_mode = stat->mode; + tmp.stx_ino = stat->ino; + tmp.stx_size = stat->size; + tmp.stx_blocks = stat->blocks; + tmp.stx_atime.tv_sec = stat->atime.tv_sec; + tmp.stx_atime.tv_nsec = stat->atime.tv_nsec; + tmp.stx_btime.tv_sec = stat->btime.tv_sec; + tmp.stx_btime.tv_nsec = stat->btime.tv_nsec; + tmp.stx_ctime.tv_sec = stat->ctime.tv_sec; + tmp.stx_ctime.tv_nsec = stat->ctime.tv_nsec; + tmp.stx_mtime.tv_sec = stat->mtime.tv_sec; + tmp.stx_mtime.tv_nsec = stat->mtime.tv_nsec; + tmp.stx_rdev_major = MAJOR(stat->rdev); + tmp.stx_rdev_minor = MINOR(stat->rdev); + tmp.stx_dev_major = MAJOR(stat->dev); + tmp.stx_dev_minor = MINOR(stat->dev); + + return copy_to_user(buffer, &tmp, sizeof(tmp)) ? -EFAULT : 0; } /** @@ -572,8 +562,6 @@ SYSCALL_DEFINE5(statx, if ((flags & AT_STATX_SYNC_TYPE) == AT_STATX_SYNC_TYPE) return -EINVAL; - if (!access_ok(VERIFY_WRITE, buffer, sizeof(*buffer))) - return -EFAULT; if (filename) error = vfs_statx(dfd, filename, flags, &stat, mask); @@ -581,7 +569,8 @@ SYSCALL_DEFINE5(statx, error = vfs_statx_fd(dfd, &stat, mask, flags); if (error) return error; - return statx_set_result(&stat, buffer); + + return cp_statx(&stat, buffer); } /* Caller is here responsible for sufficient locking (ie. inode->i_lock) */