From patchwork Mon Oct 24 16:28:14 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 9392529 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 87FFB60762 for ; Mon, 24 Oct 2016 16:29:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 777B22919A for ; Mon, 24 Oct 2016 16:29:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6C0A92919C; Mon, 24 Oct 2016 16:29:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09FCD2919A for ; Mon, 24 Oct 2016 16:29:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965416AbcJXQ2b (ORCPT ); Mon, 24 Oct 2016 12:28:31 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:57310 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S965167AbcJXQ20 (ORCPT ); Mon, 24 Oct 2016 12:28:26 -0400 Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id u9OGQxsk004911; Mon, 24 Oct 2016 09:28:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=subject : to : references : cc : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=facebook; bh=2nJGiuqMCuG74RC+1dvPlMkn4BlzJMha6lk3TZDTvQs=; b=krI2XhtwbS8l0f8+osmGA72I4FtKbo7ddsKDwRi1kar5zJsTX/gtQc44hr2iHlWtBcVz ZtxTFctE/w8cxWGOC8ehCgTSuGsT7rNGPa7FC2R/rs4CIV+90IB0t+n/B97mDFBQjkv7 qr80aW9kOeCj53oSBLdqGEQ6Y0R629ncpOk= Received: from mail.thefacebook.com ([199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 2683x3jj9v-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 24 Oct 2016 09:28:23 -0700 Received: from NAM02-BL2-obe.outbound.protection.outlook.com (192.168.54.28) by o365-in.thefacebook.com (192.168.16.16) with Microsoft SMTP Server (TLS) id 14.3.294.0; Mon, 24 Oct 2016 09:28:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=WpZ1753ar87M/9ezDk5h8poYRE4GdDsM0N8EJHjj0ag=; b=JHB3WbGbGOBY9VOB9lzX6cG9NT70SiwSKVOBJXKY7ig9ct0sTX2HzFPsL/5A1f2koQjenQuSp5MS44G8Y8ivA9yPVYVZndaZ8zhlyGj10GYM5DCheuyGS79I/+O+tFRAspDuK/b21+45Z2b4GZ5in1AxKxhBo4CoYDLzmqjXfoU= Received: from [192.168.1.153] (216.160.245.98) by BN6PR15MB1187.namprd15.prod.outlook.com (10.172.205.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.679.12; Mon, 24 Oct 2016 16:28:18 +0000 Subject: Re: Test generic/299 stalling forever To: "Theodore Ts'o" References: <20161013231923.j2fidfbtzdp66x3t@thunk.org> <20161018180107.fscbfm66yidwhey4@thunk.org> <7856791a-0795-9183-6057-6ce8fd0e3d58@fb.com> <30fef8cd-67cc-da49-77d9-9d1a833f8a48@fb.com> <20161019203233.mbbmskpn5ekgl7og@thunk.org> <1fb60e7c-a558-80df-09da-d3c36863a461@fb.com> <20161021221551.sdv4hgw33zjxnkvu@thunk.org> <53fe5a98-6ff9-4fa1-e84c-8a3e16cc0f50@fb.com> <20161023193320.rlzlaxdi4vbyu7of@thunk.org> <20161023212408.cjqmnzw3547ujzil@thunk.org> <20161024033852.quinlee4a24mb2e2@thunk.org> CC: Dave Chinner , , , From: Jens Axboe Message-ID: <773e0780-6641-ec85-5e78-d04e5a82d6b1@fb.com> Date: Mon, 24 Oct 2016 10:28:14 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <20161024033852.quinlee4a24mb2e2@thunk.org> X-Originating-IP: [216.160.245.98] X-ClientProxiedBy: CO1PR16CA0037.namprd16.prod.outlook.com (10.166.27.47) To BN6PR15MB1187.namprd15.prod.outlook.com (10.172.205.141) X-MS-Office365-Filtering-Correlation-Id: 9c96aa71-ce0b-4d78-ebc1-08d3fc2ac3e4 X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1187; 2:0p5WPjAVCw/vzRuKbv+dU0hsc5gQbYbFUbsGh/iz1GykOO64L+C6vyJJ1d2EFypfl/dgVxj2B+C/Ww3rmnVU9oEBSCFtEkK7fIOcmUlYnL/6RKdfg/K1VKmNpic/aP0mMl2zucc9aLCrW846wlC/W8CElPbsV70vNZ6leNjS2UvkpZK5j/Hd2tkE7XlO7QS9y3TuuihPEkkZWABN7hnR/g==; 3:Er6h8qIF78aHaVRgJpTNrDUcKPW0s9+3Ss9uH8zM3vQW7bPR6MjKvT4kKzDoHCwLfaOPu54IL7+szMfnaneypEpEYCmSjgHnS3YL5ywA62th3e8g3w1L6EARGIz9oJTNVZv1mKOO58RliRCDnUjqWQ==; 25:pUwVgrnzRnD7uPrb9TRSRbTzTPtOe0wjINL4VwkO02ORR/6pjDiX4xNx1rytUfvXWj2CC11yoOcFZAU10f2ktAAaXkITvJo6OGEFhaZaPMsmi3NH0EcAoDBQMxYz1/UKnKIZLIpvKu1TnLeGm5eennnlR2OAAt3TfCPGtgsNixfhSuUZ2iro8I8vLDR0hBtU4ZqORAbobsaFfnb1GeTJ0m5TSjxnI/VbLhrJ2CsOo7IEs5yBI515Z9iTc0xMcXg6xRABRWdzpsTVbkB8W8nj23JXrvRob3mo2/X+WUuFT+JHJKgA69iXmzdMcuDH6ZNF8HbUENyKGqigjIsrDd6YxfkZo0K0Yg93ezycABJa5ptDtKk1455mf7yI/PWvzpoLO6qL0ClxPSVuVb9/PC5N3nyVUW++FjT7MkKt6AmEKmgWz+UI6BiLudxAjNzkagKS X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1187; X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1187; 31:eeFId2PWg7oa8mxNBlOOhsFz+ad1Bn3AhvMyb2BtDZVyosbbjo2iXhMKgcItuQfEc6DR0oaIxAH51SdEKoJ9XwdhuHRXZ4j55KAaSXajWsBHpczEf3dj8mAgocmF9YUvMzCgfQ+yK1WnEHVWjuB7Mw5oE7gdmg2z8BDnCDmNicb04aJa/SEHrL5pmlaSqk0sS16Wc+dWN2SMbXJPNT2rM4zhYsj/gOJx478X6CRD9fum24wann4XrshjG8gUf/3e; 20:fsscNubBXTFF602MPFEch/n/gFAEnGAuor2If9J3oXaN5Wdqwo07hN+WvoiPfFR3hQoTuxiek+zIh6YJv/dwYWCX2+1vk3F6m18kZxxYJWwNELpMy3Jbhb9B9k8DMXtA0iVCl2/bcTsz1CEQKTT4z9NKMt2EEz9YNwDgchZgxFI=; 4:r//8QmlxTESYo6w3+dgtkLHkYKZOMmhpfg9iwJaR/mdRQaDeF+Xavci8512eNk0N8f1g5K6LTYD0KIga+bvGXfXE4Rz9SvwQpvSjpBwsfBLyAcgDGi84c6mVJSZQeCQhrUZOaQGHvb0fkJvjdeXhIK7YQsQ3406+tERHnmY/BfmcENnpXF1OHU8roy6pQ8EvTu0Z8PtJq53PoBLc+rcV69A5Iv3a6/kZmokidLwpGlfmFOjvqSuAA9RqyeiNwArMQZ6kSikOC+P90997/T6IWvmWU5ERb/jtHFhi8vFCIrzZtq9oLkMv/G+ApS6OVumZ+o6n5FR+kOrdtKS9rJnGggVSyXN/cGSwkgv1dSIoyXn6btI0z9RFcMUY/VssZLcahFI6sDdrgab1B17R7m4/pMwq6p75Sgx4xNh5abqlOjOliJel0AqQpm6RUWip8xaB X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001); SRVR:BN6PR15MB1187; BCL:0; PCL:0; RULEID:; SRVR:BN6PR15MB1187; X-Forefront-PRVS: 0105DAA385 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6049001)(6009001)(7916002)(24454002)(377454003)(199003)(189002)(81166006)(6666003)(47776003)(5890100001)(189998001)(6916009)(81156014)(66066001)(65806001)(65956001)(64126003)(19580395003)(110136003)(77096005)(117156001)(83506001)(68736007)(19580405001)(8676002)(7736002)(305945005)(586003)(2171001)(86362001)(5660300001)(3846002)(6116002)(31696002)(2906002)(93886004)(33646002)(7846002)(4326007)(92566002)(101416001)(31686004)(65826007)(76176999)(23746002)(54356999)(36756003)(50986999)(97736004)(4001350100001)(230700001)(42186005)(105586002)(106356001)(2950100002)(50466002)(41533002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN6PR15MB1187; H:[192.168.1.153]; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: fb.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; BN6PR15MB1187; 23:TS9EDXhwzLVTO0k0dLCJUN6egoc3g8ila/kdS?= =?Windows-1252?Q?tv62p3aD52CZR/TJzpiR+/h2OdpxkQaqL7Oa5LeW9UxknM/uRtB6OQ/h?= =?Windows-1252?Q?et5NPULbeomx7Kj50V7XCjaEP/pSR9oelo2KD3vRMfMOUDJw34hqRt9G?= =?Windows-1252?Q?nXB/t6KgKiU0rtRCZKWXnUVfSk4gbyFIdG3IxglxJHnROWtqQlhcUJje?= =?Windows-1252?Q?3fHLIl7aBnTNGw2zjfdtcJPervCh6y0a2CDpVPEFy9erw/hbZQk2S0Je?= =?Windows-1252?Q?ugGxT1SfcSqKg6J15V9sM9wpcjAyWoY1nR1spDipkdmB6WxfGxSKxJ2t?= =?Windows-1252?Q?RxY4o1bCAx30EhR3ghr3H86hsojk+Zed8DrbodiSQdYu0cxOwaLUkVg8?= =?Windows-1252?Q?p6ZZuRv5ChLSxvF80woMsTshMBFFhPch1lOcDCG5K2udDtiDoNYcvBNH?= =?Windows-1252?Q?7hpggvrAHyQmR2g5ynN8W2QUbAEnGBmbPt1/XOUVPTns9NBhxM3VXITl?= =?Windows-1252?Q?gpBntnp7Rr9odsrNvENXWgrjDEoFNcTRcJj8sGwczn3IfX5wjTUStZEd?= =?Windows-1252?Q?kJQH0F4hGHwJpnemZbDHOzQ3dMIdXWkccODDyt5gDc5308MM4pzuBupX?= =?Windows-1252?Q?CHd76L0RQRaMUhmwOv2SvlgkwDQMj6Nw8JDeVDusDyBI9PfsdHHuyWVP?= =?Windows-1252?Q?xJ70KbBaa0hNHZVTpPm6YBX738HGg0tgcDBoUdZIQQhHfWyLsgmhjKhj?= =?Windows-1252?Q?LYCAiEJ3e4TxY5z1ui7zAJsB3YUxEe53TRWEudQ5UOOlUIBn6/yKh2J5?= =?Windows-1252?Q?Ws+kNt7KLfUs1bro+XlGtnRiG7hzmgV5mk+hAp36P6TZNc5Oq1HC3NDa?= =?Windows-1252?Q?EjB9AM3fC9yXnBgEF6hSY7p0TQHkQGy/m5NWEyUKdMshBvHH2AvLVj48?= =?Windows-1252?Q?WLKQTu+fkVRS2UYVPd3Rop+MS15XSV/y7hGNyrTqc1Gm+yuHNoiXf2d/?= =?Windows-1252?Q?+saoSRxTgQkRmPZXVZpirBI9qjhP6LnWpnM/n0TqiJWFsttydTFJoYld?= =?Windows-1252?Q?COb2Dm1qsfuDM8UEuIzYbp8IEPl3mDceQ5WY9LhY6BZpsrcyAO3BLcid?= =?Windows-1252?Q?HNKtWTURoAvdZSgij9i4Elqtvd/oUqzaIuIdi56+wwovhrQ0USFXSF7n?= =?Windows-1252?Q?eQfCQESAF0hT6ET1v4FmjxtghXE0+eIlf8QtTbz1z85yMDxs80IUFyuk?= =?Windows-1252?Q?kxt1PJZlOVfEvLFpSKk1hI7l1y7ys3cA2oWnJBZVuU4PkEDLsveNHhJL?= =?Windows-1252?Q?ZIIkFuAnwOTRO/Vu6zHwflxnorjyXSI7DrYKdK61exiAEhuG7JRj4VSU?= =?Windows-1252?Q?tdTZIwbx52Ob8V2XhqDcLHFWoMrqgfnPIskOG1SdG1wogEmhGOPuCe2b?= =?Windows-1252?Q?mF6Ikn2L/57g3pnUzdCg4OHDC1Fqyij626rKBqj67VmdbUrEb14UpWaz?= =?Windows-1252?Q?lgRj23snUCH+Ik12fpXvV9XFqig6lXAPL+hElwudezzoEYh0zELbaHWB?= =?Windows-1252?Q?M1e9ykc5q3AFeI=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1187; 6:QwtPsBcLOYGPgRcNBQ0IF3rEoau8BEGobXNXYvR2dUp6WF3V1n6mSftBIEvNwaPCO+INBHPEU65FjMxN3yyXs6m3AVN4u/KKDHhgacmPkK50x5O7sz5JZEazVEsNVcbX05Mv3J0psI000cRQM1uo42SHt/hfpA5fZ+Kz0Jtu/s2rh9rV6p8sHviwG1Yx7gwpIvUX2tITmDXih+OOOC7pl3wSPp8zhLwnm9dSe6v2tYvFvoRobnH8vIvnqcBr/I54ah3CqtQLgFAGw0uXgvDhkJC8wKarCK8yi2EWPfCrRYy3O+TPbYGv1/i70eIhOwIg; 5:sXfB9ay1HJpyi87IZS4cQM/pAW4s/AMfWNhryXcPU8meg8yl0MVXtq7YvI64psQC+Wdejk4HiJCDT1efFG5As/MQnOrmSLo0pQyUakW7Uv1wK1zkz6sdVwEOf2eo/7gYvg7/ISRL+VQtIhClx0TGqQ==; 24:N5aGA8qcUxNkPNnVAAHiASI3B6qU584PLHj9pU+3smAPltetnjRUG7IJ/fuUXd85sQ1RXg/Et8qlwAJeVgkw8K66V/sbzWSfxW7dOorLM1g=; 7:yHIyTRp8H/6EfZWeun/Z0ZrXyAknly93FTIYy9djdKl41ob6p0fdqpeA7MfRncXf90UYhgL8VW8I4dutbAePAEiF7lMYGbJrNPN5lcPu6yj779v4Ql35Ub4llUxTiScuE6JWgJ5/Z8Vf1jKDW3vLL5lubOY9QaRCiPTaRX0tWYFlLpqlJHTt40y8Lq8PtgVrJ+84R7T5s2MefrY5H1C6ATTZkr3CIg/FtRdXKZaT/y+DmDkydeY3VOIaMXYpndTUpgEPkvd6SwytNXgeQHyVxFrqWJ8YVteZUlKFJfQPSrM8tihcI97BRWo/BYYTHUV7+nwv+8y/+SF63y9V8jphl3+20dKB WaAXZQS1svcbAQQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1187; 20:nnCUPzRK4A1J0gL0vaTmd8Tp9mmbBhiQ+DMnhFPANiOIa5I64V7zFct8YBFpYzAhrjRHX7bcW/B0gjIUUA4Gfjev58ejwiJT9boCxhIYLGLoHjM+xm2NqWxw0gLEldQjakRF71mlotAR4wKKn+IRf3kLPcyGOpBTDlMqSFqgbOc= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2016 16:28:18.6710 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1187 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-10-24_10:, , signatures=0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 10/23/2016 09:38 PM, Theodore Ts'o wrote: > I enabled some more debugging and it's become more clear what's going > on. (See attached for the full log). > > The main issue seems to be that once one of fio is done, it kills off > the other threads (actually, we're using processes): > > process 31848 terminate group_id=0 > process 31848 setting terminate on direct_aio/31846 > process 31848 setting terminate on direct_aio/31848 > process 31848 setting terminate on direct_aio/31849 > process 31848 setting terminate on direct_aio/31851 > process 31848 setting terminate on aio-dio-verifier/31852 > process 31848 setting terminate on buffered-aio-verifier/31854 > process 31851 pid=31851: runstate RUNNING -> FINISHING > process 31851 terminate group_id=0 > process 31851 setting terminate on direct_aio/31846 > process 31851 setting terminate on direct_aio/31848 > process 31851 setting terminate on direct_aio/31849 > process 31851 setting terminate on direct_aio/31851 > process 31851 setting terminate on aio-dio-verifier/31852 > process 31851 setting terminate on buffered-aio-verifier/31854 > process 31852 pid=31852: runstate RUNNING -> FINISHING > process 31846 pid=31846: runstate RUNNING -> FINISHING > ... > > but one or more of the threads doesn't exit within 60 seconds: > > fio: job 'direct_aio' (state=5) hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. > process 31794 pid=31849: runstate RUNNING -> REAPED > fio: job 'buffered-aio-verifier' (state=5) hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. > process 31794 pid=31854: runstate RUNNING -> REAPED > process 31794 terminate group_id=-1 > > The main thread then prints all of the statistics, and calls stat_exit(): > > stat_exit called by tid: 31794 <---- debugging message which prints gettid() > > Unfortunately, this process(es) aren't actually, killed, they are > marked as reap, but they are still in the process listing: > > root@xfstests:~# ps augxww | grep fio > root 1585 0.0 0.0 0 0 ? S< 18:45 0:00 [dm_bufio_cache] > root 7191 0.0 0.0 12732 2200 pts/1 S+ 23:05 0:00 grep fio > root 31849 1.5 0.2 407208 18876 ? Ss 22:36 0:26 /root/xfstests/bin/fio /tmp/31503.fio > root 31854 1.2 0.1 398480 10240 ? Ssl 22:36 0:22 /root/xfstests/bin/fio /tmp/31503.fio > > And if you attach to them with a gdb, they are spinning trying to grab > the stat_mutex(), which they can't get because the main thread has > already called stat_exit() and then has exited. So these two threads > did eventually return, but some time after 60 seconds had passed, and > then they hung waiting for stat_mutex(), which they will never get > because the main thread has already called stat_exit(). > > This probably also explains why you had trouble reproducing it. It > requires a disk whose performance is variable enougoh that under heavy > load, it might take more than 60 seconds for the direct_aio or > buffered-aio-verifier thread to close itself out. Good catch! Yes, that could certainly explain why we are stuck on that stat_mutex and why the main thread just gave up on it and ended up in stat_exit() with a thread (or more) still running. > And I suspect once the main thread exited, it probably also closed out > the debugging channel so the deadlock detector did probably trip, but > somehow we just didn't see the output. > > So I can imagine some possible fixes. We could make the thread > timeout configurable, and/or increase it from 60 seconds to something like > 300 seconds. We could make stat_exit() a no-op --- after all, if the > main thread is exiting, there's no real point to down and then destroy > the stat_mutex. And/or we could change the forced reap to send a kill > -9 to the thread, and instead of maring it as reaped. We have to clean up - for normal runs, it's not a big deal, but if fio is run as a client/server setup, the backend will persist across runs. If we leak, then that could be a concern. How about the below? Bump the timeout to 5 min, 1 min is a little on the short side, we want normal error handling to be out of the way before that happens. And additionally, break out if we have been marked as reaped/exited, so we avoid grabbing the stat mutex again. diff --git a/backend.c b/backend.c index 093b6a3a290e..f0927abfccb0 100644 --- a/backend.c +++ b/backend.c @@ -1723,6 +1723,14 @@ static void *thread_main(void *data) } } + /* + * If we took too long to shut down, the main thread could + * already consider us reaped/exited. If that happens, break + * out and clean up. + */ + if (td->runstate >= TD_EXITED) + break; + clear_state = 1; /* diff --git a/fio.h b/fio.h index 080842aef4f8..74c1b306af26 100644 --- a/fio.h +++ b/fio.h @@ -588,7 +588,7 @@ extern const char *runstate_to_name(int runstate); * Allow 60 seconds for a job to quit on its own, otherwise reap with * a vengeance. */ -#define FIO_REAP_TIMEOUT 60 +#define FIO_REAP_TIMEOUT 300 #define TERMINATE_ALL (-1U) extern void fio_terminate_threads(unsigned int);