From patchwork Wed Aug 28 18:15:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Foster X-Patchwork-Id: 13781708 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCE8F1AAE18 for ; Wed, 28 Aug 2024 18:14:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868887; cv=none; b=F5ioPOCEYJXCiuXRyk0OFcMAqf2W4hemVO3vNG+7ujvjFrluWaGqj07olkx054SKVpYp8wTvPsUiN3OfE+ELze6C0uOnDNcZ9IXZwn8pid4ZNZgD/2PtK7t6wlxxXRqajoiMrNCtRRhO0b2xoGJidIsN2U9eBCeZIdEQsNBcxK4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868887; c=relaxed/simple; bh=cHRUz83EMXaqlfUJ9OTVmO+N10HSLJcP+o16mw1uDtM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SEHdrDHdQNsLIJNi4uNNOpupJuLk9eqQuIqVluys3HQ+WN7+IF4PDqtLpnL6KA5yNHkazEgVuQ/nrRmDBkoYYsOdXMJli+WWxBdq5OJ5DwTQcdTP9DmCp8UEIkOHKqjJohiWNFHcvfom9dMQKz/7ZGd7Wx7DTTU7EvCpvGdhuag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=S7VhhFSs; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="S7VhhFSs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724868884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yt9reNIdExQOeIUP0ueGXEa/D+je1P4YQPbZxIQzm0Y=; b=S7VhhFSs/HymbOT2VVkqeFWbl4aIk2h06PgUx4rbK+KT+vXhQLFfJOC7iisGp3nBbJkmgE 0Fjj6r8/Fo4wczZb/yu/ymTfX+edHq98PI2Uka8DCfAB517ElqFJttYBBGXKK45yeXbqbI h13kR/F87puVgW0uy8zk5E6FdzNOt0M= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-690-O_KnlUshPYCdEKDZes_H4g-1; Wed, 28 Aug 2024 14:14:40 -0400 X-MC-Unique: O_KnlUshPYCdEKDZes_H4g-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EF6991955D4B; Wed, 28 Aug 2024 18:14:35 +0000 (UTC) Received: from bfoster.redhat.com (unknown [10.22.16.95]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D798119560A3; Wed, 28 Aug 2024 18:14:34 +0000 (UTC) From: Brian Foster To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org, djwong@kernel.org, josef@toxicpanda.com, david@fromorbit.com Subject: [PATCH v2 1/4] fsx: don't skip file size and buf updates on simulated ops Date: Wed, 28 Aug 2024 14:15:31 -0400 Message-ID: <20240828181534.41054-2-bfoster@redhat.com> In-Reply-To: <20240828181534.41054-1-bfoster@redhat.com> References: <20240828181534.41054-1-bfoster@redhat.com> Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 fsx supports the ability to skip through a certain number of operations of a given command sequence before beginning full operation. The way this works is by tracking the operation count, simulating minimal side effects of skipped operations in-memory, and then finally writing out the in-memory state to the target file when full operation begins. Several fallocate() related operations don't correctly track in-memory state when simulated, however. For example, consider an ops file with the following two operations: zero_range 0x0 0x1000 0x0 read 0x0 0x1000 0x0 ... and an fsx run like so: fsx -d -b 2 --replay-ops= This simulates the zero_range operation, but fails to track the file extension that occurs as a side effect such that the subsequent read doesn't occur as expected: Will begin at operation 2 skipping zero size read The read is skipped in this case because the file size is zero. The proper behavior, and what is consistent with other size changing operations, is to make the appropriate in-core changes before checking whether an operation is simulated so the end result of those changes can be reflected on-disk for eventual non-simulated operations. This results in expected behavior with the same ops file and test command: Will begin at operation 2 2 read 0x0 thru 0xfff (0x1000 bytes) Update zero, copy and clone range to do the file size and EOF change related zeroing before checking against the simulated ops count. Signed-off-by: Brian Foster Reviewed-by: Darrick J. Wong --- ltp/fsx.c | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/ltp/fsx.c b/ltp/fsx.c index 2dc59b06..c5727cff 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -1247,6 +1247,17 @@ do_zero_range(unsigned offset, unsigned length, int keep_size) log4(OP_ZERO_RANGE, offset, length, keep_size ? FL_KEEP_SIZE : FL_NONE); + if (!keep_size && end_offset > file_size) { + /* + * If there's a gap between the old file size and the offset of + * the zero range operation, fill the gap with zeroes. + */ + if (offset > file_size) + memset(good_buf + file_size, '\0', offset - file_size); + + file_size = end_offset; + } + if (testcalls <= simulatedopcount) return; @@ -1263,17 +1274,6 @@ do_zero_range(unsigned offset, unsigned length, int keep_size) } memset(good_buf + offset, '\0', length); - - if (!keep_size && end_offset > file_size) { - /* - * If there's a gap between the old file size and the offset of - * the zero range operation, fill the gap with zeroes. - */ - if (offset > file_size) - memset(good_buf + file_size, '\0', offset - file_size); - - file_size = end_offset; - } } #else @@ -1538,6 +1538,11 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest) log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE); + if (dest > file_size) + memset(good_buf + file_size, '\0', dest - file_size); + if (dest + length > file_size) + file_size = dest + length; + if (testcalls <= simulatedopcount) return; @@ -1556,10 +1561,6 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest) } memcpy(good_buf + dest, good_buf + offset, length); - if (dest > file_size) - memset(good_buf + file_size, '\0', dest - file_size); - if (dest + length > file_size) - file_size = dest + length; } #else @@ -1756,6 +1757,11 @@ do_copy_range(unsigned offset, unsigned length, unsigned dest) log5(OP_COPY_RANGE, offset, length, dest, FL_NONE); + if (dest > file_size) + memset(good_buf + file_size, '\0', dest - file_size); + if (dest + length > file_size) + file_size = dest + length; + if (testcalls <= simulatedopcount) return; @@ -1792,10 +1798,6 @@ do_copy_range(unsigned offset, unsigned length, unsigned dest) } memcpy(good_buf + dest, good_buf + offset, length); - if (dest > file_size) - memset(good_buf + file_size, '\0', dest - file_size); - if (dest + length > file_size) - file_size = dest + length; } #else From patchwork Wed Aug 28 18:15:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Foster X-Patchwork-Id: 13781707 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBCBD1A08C0 for ; Wed, 28 Aug 2024 18:14:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868885; cv=none; b=JFs5qHJc0ApF2ZS5PG7revHLxnAtdFAkin4P3Go1sE1J/nvfmATH90Bf9YoMAffWFnHytlhD9zcWOtZKo7k/DcvLmvQMAqSv6Ccqh+NUCjatCtNUMpQnuw+tGsCNoh0ak1vhvaovucf3EBE/Kj34IBKig8yj6xrJirgN6AC02WE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868885; c=relaxed/simple; bh=EltExVNBVtwXO0doVvNv0FiKr0r4Lz2Ykpk1zb9lm4Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JxlbaPWHkf24vWlacoLryU/7JjxGxAVpzA6PlgtFffyY6ZmSPeVo/pSz6ET0wXUoNknZ2aT9vSwP2LL3U2G05akPhU7AGwEkyOHd+amuyG7wuoYFW5UBMfCu1PRgEMMxP7uZ2BqllqXLZe+gvRZAdYH2mIq4qtKWBFUB/9DAWD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=IhfnJBPw; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IhfnJBPw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724868881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y08YxeU7UkCu8KXIjOZTX6xc9D1+311RxroFnSp6Kjc=; b=IhfnJBPwlWw7aK9xJxjmis/BS4IElW4nKtkApQmlVJ7ELKMGXRAHX/e/xv39nRSr0z3LRD 2+t+oBVPCxwuODXh3g+zX3m08y5QwIN70FX5cAq0a8PekxxObwtmwq5tBw043uFuq6kIi7 xKWfOSYql/CieXoKrCgZs5JClsGJ7lw= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-OOUI5mS1P_mQuL46k4M3jA-1; Wed, 28 Aug 2024 14:14:38 -0400 X-MC-Unique: OOUI5mS1P_mQuL46k4M3jA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 56D651955D45; Wed, 28 Aug 2024 18:14:37 +0000 (UTC) Received: from bfoster.redhat.com (unknown [10.22.16.95]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4065319560A3; Wed, 28 Aug 2024 18:14:36 +0000 (UTC) From: Brian Foster To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org, djwong@kernel.org, josef@toxicpanda.com, david@fromorbit.com Subject: [PATCH v2 2/4] fsx: factor out a file size update helper Date: Wed, 28 Aug 2024 14:15:32 -0400 Message-ID: <20240828181534.41054-3-bfoster@redhat.com> In-Reply-To: <20240828181534.41054-1-bfoster@redhat.com> References: <20240828181534.41054-1-bfoster@redhat.com> Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 In preparation for support for eof page pollution, factor out a file size update helper. This updates the internally tracked file size based on the upcoming operation and zeroes the appropriate range in the good buffer for extending operations. Signed-off-by: Brian Foster Reviewed-by: Darrick J. Wong --- ltp/fsx.c | 49 +++++++++++++++++++++---------------------------- 1 file changed, 21 insertions(+), 28 deletions(-) diff --git a/ltp/fsx.c b/ltp/fsx.c index c5727cff..4a9be0e1 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -983,6 +983,17 @@ gendata(char *original_buf, char *good_buf, unsigned offset, unsigned size) } } +/* + * Helper to update the tracked file size. If the offset begins beyond current + * EOF, zero the range from EOF to offset in the good buffer. + */ +void +update_file_size(unsigned offset, unsigned size) +{ + if (offset > file_size) + memset(good_buf + file_size, '\0', offset - file_size); + file_size = offset + size; +} void dowrite(unsigned offset, unsigned size) @@ -1003,10 +1014,8 @@ dowrite(unsigned offset, unsigned size) log4(OP_WRITE, offset, size, FL_NONE); gendata(original_buf, good_buf, offset, size); - if (file_size < offset + size) { - if (file_size < offset) - memset(good_buf + file_size, '\0', offset - file_size); - file_size = offset + size; + if (offset + size > file_size) { + update_file_size(offset, size); if (lite) { warn("Lite file size bug in fsx!"); report_failure(149); @@ -1070,10 +1079,8 @@ domapwrite(unsigned offset, unsigned size) log4(OP_MAPWRITE, offset, size, FL_NONE); gendata(original_buf, good_buf, offset, size); - if (file_size < offset + size) { - if (file_size < offset) - memset(good_buf + file_size, '\0', offset - file_size); - file_size = offset + size; + if (offset + size > file_size) { + update_file_size(offset, size); if (lite) { warn("Lite file size bug in fsx!"); report_failure(200); @@ -1136,9 +1143,7 @@ dotruncate(unsigned size) log4(OP_TRUNCATE, 0, size, FL_NONE); - if (size > file_size) - memset(good_buf + file_size, '\0', size - file_size); - file_size = size; + update_file_size(size, 0); if (testcalls <= simulatedopcount) return; @@ -1247,16 +1252,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size) log4(OP_ZERO_RANGE, offset, length, keep_size ? FL_KEEP_SIZE : FL_NONE); - if (!keep_size && end_offset > file_size) { - /* - * If there's a gap between the old file size and the offset of - * the zero range operation, fill the gap with zeroes. - */ - if (offset > file_size) - memset(good_buf + file_size, '\0', offset - file_size); - - file_size = end_offset; - } + if (!keep_size && end_offset > file_size) + update_file_size(offset, length); if (testcalls <= simulatedopcount) return; @@ -1538,10 +1535,8 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest) log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE); - if (dest > file_size) - memset(good_buf + file_size, '\0', dest - file_size); if (dest + length > file_size) - file_size = dest + length; + update_file_size(dest, length); if (testcalls <= simulatedopcount) return; @@ -1757,10 +1752,8 @@ do_copy_range(unsigned offset, unsigned length, unsigned dest) log5(OP_COPY_RANGE, offset, length, dest, FL_NONE); - if (dest > file_size) - memset(good_buf + file_size, '\0', dest - file_size); if (dest + length > file_size) - file_size = dest + length; + update_file_size(dest, length); if (testcalls <= simulatedopcount) return; @@ -1848,7 +1841,7 @@ do_preallocate(unsigned offset, unsigned length, int keep_size) if (end_offset > file_size) { memset(good_buf + file_size, '\0', end_offset - file_size); - file_size = end_offset; + update_file_size(offset, length); } if (testcalls <= simulatedopcount) From patchwork Wed Aug 28 18:15:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Foster X-Patchwork-Id: 13781710 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 301F31A7AFD for ; Wed, 28 Aug 2024 18:14:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868890; cv=none; b=VJv/8bJX7eFvoswnSfk6yvvxLISy6qB7S234wekavz8hPzxg6Gf2M7nX1gqzAa4TmpfH8tudbQUNwQKPCh0EPNo8wBO5xzkbYYNTDXktZ4JC93vl+4gYEQIK7FVaT9hv+iV+I1KX1ItIL0ix+CHqjYhwnu+8kN/VuId+LDOFfh4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868890; c=relaxed/simple; bh=nn1Nku1we92KHSuoMPkwcwR4bVw07hvl2MP/SYJ4FgY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VX8CzpIGCoPVpzj02CNE4ePRyAOUlDHGa/7V1lRi9Dw/JsFQJGsLu2AVvf16V8M70ucDv991m/M7/cTbNx917wMdA0FUinYBnXG4Oxjj/sQtQ2+LGv03NmEkz8xausAX8KgXlU/ZgBfWyJwCbWVZwJbXkKipOzses/fTt020Tz8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=E0xAOeY5; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="E0xAOeY5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724868887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O0/QF5Ihv3oBscR20RgPJnxLiEgKG1otN46gLtedga0=; b=E0xAOeY5f33JiKLe4RYFaX/9vtOT71aDpKSvvAGzx1Sfu9bqYAIsO2YUw41oKdzgDaSzN3 M2bQ2o6UnTk24Jp9Eji6JMl7saFL81swP+wX0pinRFU3T+bNyI0rNVqc8zvCgg/uYiGBvP VnI3DdSKqLiseL0TwBYnLK9Zz5lBfLw= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-198-iMWsnC-fPxaBcnKhR7ue2w-1; Wed, 28 Aug 2024 14:14:41 -0400 X-MC-Unique: iMWsnC-fPxaBcnKhR7ue2w-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C955D1955D4D; Wed, 28 Aug 2024 18:14:38 +0000 (UTC) Received: from bfoster.redhat.com (unknown [10.22.16.95]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9322C19560A3; Wed, 28 Aug 2024 18:14:37 +0000 (UTC) From: Brian Foster To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org, djwong@kernel.org, josef@toxicpanda.com, david@fromorbit.com Subject: [PATCH v2 3/4] fsx: support eof page pollution for eof zeroing test coverage Date: Wed, 28 Aug 2024 14:15:33 -0400 Message-ID: <20240828181534.41054-4-bfoster@redhat.com> In-Reply-To: <20240828181534.41054-1-bfoster@redhat.com> References: <20240828181534.41054-1-bfoster@redhat.com> Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 File ranges that are newly exposed via size changing operations are expected to return zeroes until written to. This behavior tends to be difficult to regression test as failures can be racy and transient. fsx is probably the best tool for this type of test coverage, but uncovering issues can require running for a significantly longer period of time than is typically invoked through fstests tests. As a result, these types of regressions tend to go unnoticed for an unfortunate amount of time. To facilitate uncovering these problems more quickly, implement an eof pollution mode in fsx that opportunistically injects post-eof data prior to operations that change file size. Since data injection occurs immediately before the size changing operation, it can be used to detect problems in partial eof page/block zeroing associated with each relevant operation. The implementation takes advantage of the fact that mapped writes can place data beyond eof so long as the page starts within eof. The main reason for the isolated per-operation approach (vs. something like allowing mapped writes to write beyond eof, for example) is to accommodate the fact that writeback zeroes post-eof data on the eof page. The current approach is therefore not necessarily guaranteed to detect all problems, but provides more generic and broad test coverage than the alternative of testing explicit command sequences and doesn't require significant changes to how fsx works. If this proves useful long term, further enhancements can be considered that might facilitate the presence of post-eof data across operations. Enable the feature with the -e command line option. It is disabled by default because zeroing behavior is inconsistent across filesystems. This can also be revisited in the future if zeroing behavior is refined for the major filesystems that rely on fstests for regression testing. Signed-off-by: Brian Foster Reviewed-by: Darrick J. Wong --- ltp/fsx.c | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 77 insertions(+), 2 deletions(-) diff --git a/ltp/fsx.c b/ltp/fsx.c index 4a9be0e1..1ba1bf65 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -178,6 +178,7 @@ int dedupe_range_calls = 1; /* -B flag disables */ int copy_range_calls = 1; /* -E flag disables */ int exchange_range_calls = 1; /* -0 flag disables */ int integrity = 0; /* -i flag */ +int pollute_eof = 0; /* -e flag */ int fsxgoodfd = 0; int o_direct; /* -Z */ int aio = 0; @@ -983,6 +984,63 @@ gendata(char *original_buf, char *good_buf, unsigned offset, unsigned size) } } +/* + * Pollute the EOF page with data beyond EOF prior to size change operations. + * This provides additional test coverage for partial EOF block/page zeroing. + * If the upcoming operation does not correctly zero, incorrect file data will + * be detected. + */ +void +pollute_eofpage(unsigned int maxoff) +{ + unsigned offset = file_size; + unsigned pg_offset; + unsigned write_size; + char *p; + + /* + * Nothing to do if pollution disabled or we're simulating. Simulation + * only tracks file size updates and skips syscalls, so we don't want to + * inject file data that won't be zeroed. + */ + if (!pollute_eof || testcalls <= simulatedopcount) + return; + + /* write up to specified max or the end of the eof page */ + pg_offset = offset & mmap_mask; + write_size = MIN(PAGE_SIZE - pg_offset, maxoff - offset); + + if (!pg_offset) + return; + + if (!quiet && + ((progressinterval && testcalls % progressinterval == 0) || + (debug && + (monitorstart == -1 || + (offset + write_size > monitorstart && + (monitorend == -1 || offset <= monitorend)))))) { + prt("%lld pollute_eof\t0x%x thru\t0x%x\t(0x%x bytes)\n", + testcalls, offset, offset + write_size - 1, write_size); + } + + if ((p = (char *)mmap(0, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FILE | MAP_SHARED, fd, + (off_t)(offset - pg_offset))) == MAP_FAILED) { + prterr("pollute_eofpage: mmap"); + return; + } + + /* + * Write to a range just past EOF of the test file. Do not update the + * good buffer because the upcoming operation is expected to zero this + * range of the file. + */ + gendata(original_buf, p, pg_offset, write_size); + + if (munmap(p, PAGE_SIZE) != 0) + prterr("pollute_eofpage: munmap"); +} + /* * Helper to update the tracked file size. If the offset begins beyond current * EOF, zero the range from EOF to offset in the good buffer. @@ -990,8 +1048,10 @@ gendata(char *original_buf, char *good_buf, unsigned offset, unsigned size) void update_file_size(unsigned offset, unsigned size) { - if (offset > file_size) + if (offset > file_size) { + pollute_eofpage(offset + size); memset(good_buf + file_size, '\0', offset - file_size); + } file_size = offset + size; } @@ -1143,6 +1203,9 @@ dotruncate(unsigned size) log4(OP_TRUNCATE, 0, size, FL_NONE); + /* pollute the current EOF before a truncate down */ + if (size < file_size) + pollute_eofpage(maxfilelen); update_file_size(size, 0); if (testcalls <= simulatedopcount) @@ -1305,6 +1368,9 @@ do_collapse_range(unsigned offset, unsigned length) log4(OP_COLLAPSE_RANGE, offset, length, FL_NONE); + /* pollute current eof before collapse truncates down */ + pollute_eofpage(maxfilelen); + if (testcalls <= simulatedopcount) return; @@ -1356,6 +1422,9 @@ do_insert_range(unsigned offset, unsigned length) log4(OP_INSERT_RANGE, offset, length, FL_NONE); + /* pollute current eof before insert truncates up */ + pollute_eofpage(maxfilelen); + if (testcalls <= simulatedopcount) return; @@ -2385,6 +2454,7 @@ usage(void) -b opnum: beginning operation number (default 1)\n\ -c P: 1 in P chance of file close+open at each op (default infinity)\n\ -d: debug output for all operations\n\ + -e: pollute post-eof on size changes (default 0)\n\ -f: flush and invalidate cache after I/O\n\ -g X: write character X instead of random generated data\n\ -i logdev: do integrity testing, logdev is the dm log writes device\n\ @@ -2783,7 +2853,7 @@ main(int argc, char **argv) setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ while ((ch = getopt_long(argc, argv, - "0b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", + "0b:c:de:fg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", longopts, NULL)) != EOF) switch (ch) { case 'b': @@ -2805,6 +2875,11 @@ main(int argc, char **argv) case 'd': debug = 1; break; + case 'e': + pollute_eof = getnum(optarg, &endp); + if (pollute_eof < 0 || pollute_eof > 1) + usage(); + break; case 'f': flush = 1; break; From patchwork Wed Aug 28 18:15:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Foster X-Patchwork-Id: 13781709 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28BAF1AAE1B for ; Wed, 28 Aug 2024 18:14:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868889; cv=none; b=ATUWwB4WJeYMyNoqB5FYCIgJybdRs0N3Ujp4d1BRmncmVjH0kQyCtlhlqqYMCH5zvNt6nPCj54mjq2z7koSeglk29o/LX2WjDF6TAp5EwDeGEOFCgrkIu2qH9FCzQQKkqBOUjv11qOvgS2WyvogEKiuM+YsZEV7TNtm5BDhVSTM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724868889; c=relaxed/simple; bh=R3hqRfCMpX7OqFN6uEnn9Yf+wKQefpGk6wmFFsLIcTI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WJxqBWOJwvw0eP2nQAJXfZDh6wIvKuN4bbiX9mYFyzxV5+Qc+S7mGP+KeAu223bg3B1iUXLPYQSArSZ43FsamC9FzskIiYYGrRzDrr3LIRKRNgz/ctTIxsIUYAjgF7J3lj37FG/ARGYsLH0wUoSr/Lt3uWLjzQxIXJHmYlkskmc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=D5xhVzh8; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="D5xhVzh8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724868887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZJ2WVQQS48DTokQsHCpdP25vC+/ADYzTlUK9GifS8go=; b=D5xhVzh8OtWW1m/z8Ri4wXhzkxicmHWBgGuofVI3O/sKeU6qFijDI57rypxRZQQ2K0Xz3e xTitk/20YgYHotuNLwOEvG/HIWv9/d5/mDY7iJyQcHmQa8jJVi4DCfk6uzyZ6m8QkiGAzA keX4Y4SV+Rifa5baovWrNOsN4heq6N0= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-638-_l8IlOuBPlmE8VlH19CA2g-1; Wed, 28 Aug 2024 14:14:41 -0400 X-MC-Unique: _l8IlOuBPlmE8VlH19CA2g-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1201E1955D53; Wed, 28 Aug 2024 18:14:40 +0000 (UTC) Received: from bfoster.redhat.com (unknown [10.22.16.95]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EC40319560AD; Wed, 28 Aug 2024 18:14:38 +0000 (UTC) From: Brian Foster To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org, djwong@kernel.org, josef@toxicpanda.com, david@fromorbit.com Subject: [PATCH v2 4/4] generic: test to run fsx eof pollution Date: Wed, 28 Aug 2024 14:15:34 -0400 Message-ID: <20240828181534.41054-5-bfoster@redhat.com> In-Reply-To: <20240828181534.41054-1-bfoster@redhat.com> References: <20240828181534.41054-1-bfoster@redhat.com> Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Filesystem regressions related to partial page zeroing can go unnoticed for a decent amount of time. A recent example is the issue of iomap zero range not handling dirty pagecache over unwritten extents, which leads to wrong behavior on certain file extending operations (i.e. truncate, write extension, etc.). fsx does occasionally uncover these sorts of problems, but failures can be rare and/or require longer running tests outside what is typically run via full fstests regression runs. fsx now supports a mode that injects post-eof data in order to explicitly test partial eof zeroing behavior. This uncovers certain problems more quickly and applies coverage more broadly across size changing operations. Add a new test that runs an fsx instance (modeled after generic/127) with eof pollution mode enabled. While the test is generic, it is currently limited to XFS as that is currently the only known major fs that does enough zeroing to satisfy the strict semantics expected by fsx. The long term goal is to uncover and fix issues so more filesystems can enable this test. Signed-off-by: Brian Foster Reviewed-by: Darrick J. Wong --- tests/generic/363 | 23 +++++++++++++++++++++++ tests/generic/363.out | 2 ++ 2 files changed, 25 insertions(+) create mode 100755 tests/generic/363 create mode 100644 tests/generic/363.out diff --git a/tests/generic/363 b/tests/generic/363 new file mode 100755 index 00000000..477c111c --- /dev/null +++ b/tests/generic/363 @@ -0,0 +1,23 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Red Hat, Inc. All Rights Reserved. +# +# FSQA Test No. 363 +# +# Run fsx with EOF pollution enabled. This provides test coverage for partial +# EOF page/block zeroing for operations that change file size. +# + +. ./common/preamble +_begin_fstest rw auto + +_require_test + +# currently only xfs performs enough zeroing to satisfy fsx +_supported_fs xfs + +# on failure, replace -q with -d to see post-eof writes in the dump output +run_fsx "-q -S 0 -e 1 -N 100000" + +status=0 +exit diff --git a/tests/generic/363.out b/tests/generic/363.out new file mode 100644 index 00000000..3d219cd0 --- /dev/null +++ b/tests/generic/363.out @@ -0,0 +1,2 @@ +QA output created by 363 +fsx -q -S 0 -e 1 -N 100000