From patchwork Sat Mar 11 21:45:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 9619157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4F8E5604CC for ; Sat, 11 Mar 2017 21:47:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 352E727317 for ; Sat, 11 Mar 2017 21:47:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 26C9528304; Sat, 11 Mar 2017 21:47:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A13BF27317 for ; Sat, 11 Mar 2017 21:47:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932856AbdCKVrQ (ORCPT ); Sat, 11 Mar 2017 16:47:16 -0500 Received: from mail-pf0-f194.google.com ([209.85.192.194]:32910 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932675AbdCKVrP (ORCPT ); Sat, 11 Mar 2017 16:47:15 -0500 Received: by mail-pf0-f194.google.com with SMTP id v190so14754739pfb.0; Sat, 11 Mar 2017 13:47:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=9nvwnWPBcssSfPv2X8NBhqwDk4mlrY7OStVyHUyqzt0=; b=mp9dHyPltzSZKTHVuPo2zKv07ONxO0qgXlFTdnEjldUi2OfiAjaNWLuvfq5f/3Fym0 /qgqLklkkHnWfPFR4wqfAy1LeXuOnx00I9a/D989G1W7lx8q+7N4+V84SBEaW1zgCcAi RtyPi8qQgOuhbSZW6DzEro2xbaUc+0oIlUniB6SwO08mgmDEoFI8tbPXfpWXfh/x5H22 AAmQCw4qOvJB8UsszhqSud/R4mdFpht9nHu2iUx3UP29bX2EAKBRD9Vi23h9i7ZDyI/N NA+Qay+pSstaDJZzJ6jvhfWlgtsuPMWdh811hZkr2+R9/hKm/NZwjqKWtsZYIR/fpmPP raLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=9nvwnWPBcssSfPv2X8NBhqwDk4mlrY7OStVyHUyqzt0=; b=Oq+PTJepOT4gOrENMri7qo3b1Sn3xd2ykzfkMwoh1TaZMdEFhXsoWTJSlip07ccvPO rSlgJ9Fxb0mznzahrWBO93mECR9t2SL/ibEZ4PhiTcyUJQMtzAzb/+bVG9wYvVB1p8Zm f+uFKzw661NrLPV3e57zr1nYv3IQ4v7vgORjWQTTdGDjfPflauscJAp4zZOlX8OiLL6V wrXChjwIk8Po/nOK55xkKdQO1WLjsIfc9foePsE66eL93HB5+gbxVNcjm+DqnkLSdByL ICFWatUigKZoz9Am/usVhsoff5bP4tyrfBKcvoRGk7TJh2sTHtOGhoz/6J7DyvllUA6N vjew== X-Gm-Message-State: AMke39n7RTLGQo1LZ2kl7Q25bpVMhmRwvD3Jb10mzDexiJyjeXQsA0DJqPcVkxtv82Ai3w== X-Received: by 10.84.141.129 with SMTP id 1mr36000972plv.166.1489268834481; Sat, 11 Mar 2017 13:47:14 -0800 (PST) Received: from zzz.hsd1.wa.comcast.net (c-73-239-167-150.hsd1.wa.comcast.net. [73.239.167.150]) by smtp.gmail.com with ESMTPSA id h14sm25982027pgn.41.2017.03.11.13.47.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Mar 2017 13:47:13 -0800 (PST) From: Eric Biggers To: linux-fsdevel@vger.kernel.org Cc: Al Viro , David Howells , linux-kernel@vger.kernel.org, Eric Biggers Subject: [PATCH v2] statx: optimize copy of struct statx to userspace Date: Sat, 11 Mar 2017 13:45:55 -0800 Message-Id: <20170311214555.941-1-ebiggers3@gmail.com> X-Mailer: git-send-email 2.12.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Eric Biggers I found that statx() was significantly slower than stat(). As a microbenchmark, I compared 10,000,000 invocations of fstat() on a tmpfs file to the same with statx() passed a NULL path: $ time ./stat_benchmark real 0m1.464s user 0m0.275s sys 0m1.187s $ time ./statx_benchmark real 0m5.530s user 0m0.281s sys 0m5.247s statx is expected to be a little slower than stat because struct statx is larger than struct stat, but not by *that* much. It turns out that most of the overhead was in copying struct statx to userspace, apparently mostly in all the stac/clac instructions that got generated for each __put_user() call. (This was on x86_64, but some other architectures, e.g. arm64, have something similar now too.) stat() instead initializes its struct on the stack and copies it to userspace with a single call to copy_to_user(). This turns out to be much faster, and changing statx to do this makes it almost as fast as stat: $ time ./statx_benchmark real 0m1.573s user 0m0.229s sys 0m1.344s Signed-off-by: Eric Biggers --- fs/stat.c | 72 +++++++++++++++++++++++++++++---------------------------------- 1 file changed, 33 insertions(+), 39 deletions(-) diff --git a/fs/stat.c b/fs/stat.c index fa0be59340cc..5cc267ec7865 100644 --- a/fs/stat.c +++ b/fs/stat.c @@ -509,46 +509,41 @@ SYSCALL_DEFINE4(fstatat64, int, dfd, const char __user *, filename, } #endif /* __ARCH_WANT_STAT64 || __ARCH_WANT_COMPAT_STAT64 */ -static inline int __put_timestamp(struct timespec *kts, - struct statx_timestamp __user *uts) +static inline void init_statx_timestamp(struct statx_timestamp *uts, + const struct timespec *kts) { - return (__put_user(kts->tv_sec, &uts->tv_sec ) || - __put_user(kts->tv_nsec, &uts->tv_nsec ) || - __put_user(0, &uts->__reserved )); + uts->tv_sec = kts->tv_sec; + uts->tv_nsec = kts->tv_nsec; + uts->__reserved = 0; } -/* - * Set the statx results. - */ -static long statx_set_result(struct kstat *stat, struct statx __user *buffer) +static int cp_statx(const struct kstat *stat, struct statx __user *buffer) { - uid_t uid = from_kuid_munged(current_user_ns(), stat->uid); - gid_t gid = from_kgid_munged(current_user_ns(), stat->gid); - - if (__put_user(stat->result_mask, &buffer->stx_mask ) || - __put_user(stat->mode, &buffer->stx_mode ) || - __clear_user(&buffer->__spare0, sizeof(buffer->__spare0)) || - __put_user(stat->nlink, &buffer->stx_nlink ) || - __put_user(uid, &buffer->stx_uid ) || - __put_user(gid, &buffer->stx_gid ) || - __put_user(stat->attributes, &buffer->stx_attributes ) || - __put_user(stat->blksize, &buffer->stx_blksize ) || - __put_user(MAJOR(stat->rdev), &buffer->stx_rdev_major ) || - __put_user(MINOR(stat->rdev), &buffer->stx_rdev_minor ) || - __put_user(MAJOR(stat->dev), &buffer->stx_dev_major ) || - __put_user(MINOR(stat->dev), &buffer->stx_dev_minor ) || - __put_timestamp(&stat->atime, &buffer->stx_atime ) || - __put_timestamp(&stat->btime, &buffer->stx_btime ) || - __put_timestamp(&stat->ctime, &buffer->stx_ctime ) || - __put_timestamp(&stat->mtime, &buffer->stx_mtime ) || - __put_user(stat->ino, &buffer->stx_ino ) || - __put_user(stat->size, &buffer->stx_size ) || - __put_user(stat->blocks, &buffer->stx_blocks ) || - __clear_user(&buffer->__spare1, sizeof(buffer->__spare1)) || - __clear_user(&buffer->__spare2, sizeof(buffer->__spare2))) - return -EFAULT; - - return 0; + struct statx tmp; + + tmp.stx_mask = stat->result_mask; + tmp.stx_blksize = stat->blksize; + tmp.stx_attributes = stat->attributes; + tmp.stx_nlink = stat->nlink; + tmp.stx_uid = from_kuid_munged(current_user_ns(), stat->uid); + tmp.stx_gid = from_kgid_munged(current_user_ns(), stat->gid); + tmp.stx_mode = stat->mode; + memset(tmp.__spare0, 0, sizeof(tmp.__spare0)); + tmp.stx_ino = stat->ino; + tmp.stx_size = stat->size; + tmp.stx_blocks = stat->blocks; + memset(tmp.__spare1, 0, sizeof(tmp.__spare1)); + init_statx_timestamp(&tmp.stx_atime, &stat->atime); + init_statx_timestamp(&tmp.stx_btime, &stat->btime); + init_statx_timestamp(&tmp.stx_ctime, &stat->ctime); + init_statx_timestamp(&tmp.stx_mtime, &stat->mtime); + tmp.stx_rdev_major = MAJOR(stat->rdev); + tmp.stx_rdev_minor = MINOR(stat->rdev); + tmp.stx_dev_major = MAJOR(stat->dev); + tmp.stx_dev_minor = MINOR(stat->dev); + memset(tmp.__spare2, 0, sizeof(tmp.__spare2)); + + return copy_to_user(buffer, &tmp, sizeof(tmp)) ? -EFAULT : 0; } /** @@ -572,8 +567,6 @@ SYSCALL_DEFINE5(statx, if ((flags & AT_STATX_SYNC_TYPE) == AT_STATX_SYNC_TYPE) return -EINVAL; - if (!access_ok(VERIFY_WRITE, buffer, sizeof(*buffer))) - return -EFAULT; if (filename) error = vfs_statx(dfd, filename, flags, &stat, mask); @@ -581,7 +574,8 @@ SYSCALL_DEFINE5(statx, error = vfs_statx_fd(dfd, &stat, mask, flags); if (error) return error; - return statx_set_result(&stat, buffer); + + return cp_statx(&stat, buffer); } /* Caller is here responsible for sufficient locking (ie. inode->i_lock) */