From patchwork Fri Jul 1 09:04:54 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sergey Fedorov X-Patchwork-Id: 9209507 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 168B7607D8 for ; Fri, 1 Jul 2016 09:06:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF25D2869A for ; Fri, 1 Jul 2016 09:06:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE63A28515; Fri, 1 Jul 2016 09:06:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 345A028515 for ; Fri, 1 Jul 2016 09:06:09 +0000 (UTC) Received: from localhost ([::1]:60413 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bIuOm-0000bC-AH for patchwork-qemu-devel@patchwork.kernel.org; Fri, 01 Jul 2016 05:06:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43453) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bIuNj-0007le-JI for qemu-devel@nongnu.org; Fri, 01 Jul 2016 05:05:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bIuNe-0007Ut-3y for qemu-devel@nongnu.org; Fri, 01 Jul 2016 05:05:02 -0400 Received: from mail-lf0-x244.google.com ([2a00:1450:4010:c07::244]:34329) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bIuNd-0007UW-Np for qemu-devel@nongnu.org; Fri, 01 Jul 2016 05:04:58 -0400 Received: by mail-lf0-x244.google.com with SMTP id l184so10744556lfl.1 for ; Fri, 01 Jul 2016 02:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=6t5xtEx4SYRjasykvwllmn9DL8NRCBpvIQLmOsZVHfM=; b=vC3Dz8Vq+JkWtNrPuv3TtBVCj0sJEAJxWgA8jA66AxTRg30xaN+q5w/G6oUUXwD1tO ZJaguGn06agSHpcq8jILCLu94PhnAhktAMzVyqLp1cXSINJI6Ocy6eCqA8GFO0i3pfQP i08FKa4jlcHUtcXrkG3dbrPAK8ucP8nmSlw1ZzmfsUMpuRqBldl2Op1jF3lvuNW3SKc+ mI7aJfE2V1wo4UGPQfuSXDrX1gBaTNA9bxQ35xButNwwOImHuQsU1eWVpMTm3tlMlAtz 5YTFlEw+4CXOj9G0nOkK1fT4R81N36V2tJ7v1Za2qwwrd16JQzfhFT4JzshJ/5XnKoe8 ZGJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=6t5xtEx4SYRjasykvwllmn9DL8NRCBpvIQLmOsZVHfM=; b=PCGc/2f67szL3AO0xJH2QUIImFTI7/J3v8twAMUJIKA+ykB+zLMnuQGjMdHc5gx8pD xYOUqeVTZ/jni0GY5Jhwlhie4x7/WoiEClQNg5m9J1fjbwgY8NXZR9DkgU2UZqA0JPES 6JhZiUDs2EuuS/u4JvpMbkuox9z12mbjr/fjRb4EQ8gS9BzZdDmq65/bbFN7XNJfsPz9 XTe7TnJ3rDkjWdkYPdpyBVpXbwvCwy+a5pVbEplOr88J1DtPKlR9pJWT4P0B9sG2kHRE fwAbydStjH1P2a10vxL8k1XbpbZIg1fj+F5cp5Ig+yf0Uz8hM0abVZn/YR47h3x4Cmq4 +14g== X-Gm-Message-State: ALyK8tLLKZMT0bVQ0rKnN7xvpylTFM0yaQnol7U8UPxoBiQjbpd9t09UyILO5B4uujgbrQ== X-Received: by 10.25.149.146 with SMTP id x140mr5352169lfd.131.1467363896702; Fri, 01 Jul 2016 02:04:56 -0700 (PDT) Received: from [192.168.0.65] (broadband-46-188-121-154.2com.net. [46.188.121.154]) by smtp.gmail.com with ESMTPSA id 193sm2181443ljj.49.2016.07.01.02.04.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Jul 2016 02:04:55 -0700 (PDT) To: =?UTF-8?Q?Alex_Benn=c3=a9e?= References: <1466375313-7562-1-git-send-email-sergey.fedorov@linaro.org> <1466375313-7562-7-git-send-email-sergey.fedorov@linaro.org> <87lh1o0y1k.fsf@linaro.org> <5774E8C2.1050506@gmail.com> <87furvq85v.fsf@linaro.org> <5774F60C.3010707@gmail.com> <57763049.2090800@gmail.com> From: Sergey Fedorov Message-ID: <57763236.8000203@gmail.com> Date: Fri, 1 Jul 2016 12:04:54 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 In-Reply-To: <57763049.2090800@gmail.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:4010:c07::244 Subject: Re: [Qemu-devel] [RFC 6/8] linux-user: Support CPU work queue X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Crosthwaite , Sergey Fedorov , patches@linaro.org, Riku Voipio , qemu-devel@nongnu.org, Paolo Bonzini , Richard Henderson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP On 01/07/16 11:56, Sergey Fedorov wrote: > On 30/06/16 13:35, Sergey Fedorov wrote: >> On 30/06/16 13:32, Alex Bennée wrote: >>> Sergey Fedorov writes: >>> >>>> On 29/06/16 19:17, Alex Bennée wrote: >>>>> So I think there is a deadlock we can get with the async work: >>>>> >>>>> (gdb) thread apply all bt >>>>> >>>>> Thread 11 (Thread 0x7ffefeca7700 (LWP 2912)): >>>>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 >>>>> #1 0x00005555555cb777 in wait_cpu_work () at /home/alex/lsrc/qemu/qemu.git/linux-user/main.c:155 >>>>> #2 0x00005555555a0cee in wait_safe_cpu_work () at /home/alex/lsrc/qemu/qemu.git/cpu-exec-common.c:87 >>>>> #3 0x00005555555cb8fe in cpu_exec_end (cpu=0x555555bb67e0) at /home/alex/lsrc/qemu/qemu.git/linux-user/main.c:222 >>>>> #4 0x00005555555cc7a7 in cpu_loop (env=0x555555bbea58) at /home/alex/lsrc/qemu/qemu.git/linux-user/main.c:749 >>>>> #5 0x00005555555db0b2 in clone_func (arg=0x7fffffffc9c0) at /home/alex/lsrc/qemu/qemu.git/linux-user/syscall.c:5424 >>>>> #6 0x00007ffff6bed6fa in start_thread (arg=0x7ffefeca7700) at pthread_create.c:333 >>>>> #7 0x00007ffff6923b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 >>>>> >>>>> >>>>> >>>>> Thread 3 (Thread 0x7ffff7f38700 (LWP 2904)): >>>>> #0 0x00005555555faf5d in safe_syscall_base () >>>>> #1 0x00005555555cfeaf in safe_futex (uaddr=0x7ffff528a0a4, op=128, val=1, timeout=0x0, uaddr2=0x0, val3=-162668384) >>>>> at /home/alex/lsrc/qemu/qemu.git/linux-user/syscall.c:706 >>>>> #2 0x00005555555dd7cc in do_futex (uaddr=4132298916, op=128, val=1, timeout=0, uaddr2=0, val3=-162668384) >>>>> at /home/alex/lsrc/qemu/qemu.git/linux-user/syscall.c:6246 >>>>> #3 0x00005555555e8cdb in do_syscall (cpu_env=0x555555a81118, num=240, arg1=-162668380, arg2=128, arg3=1, arg4=0, arg5=0, arg6=-162668384, >>>>> arg7=0, arg8=0) at /home/alex/lsrc/qemu/qemu.git/linux-user/syscall.c:10642 >>>>> #4 0x00005555555cd20e in cpu_loop (env=0x555555a81118) at /home/alex/lsrc/qemu/qemu.git/linux-user/main.c:883 >>>>> #5 0x00005555555db0b2 in clone_func (arg=0x7fffffffc9c0) at /home/alex/lsrc/qemu/qemu.git/linux-user/syscall.c:5424 >>>>> #6 0x00007ffff6bed6fa in start_thread (arg=0x7ffff7f38700) at pthread_create.c:333 >>>>> #7 0x00007ffff6923b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 >>>>> >>>>> So everything is stalled awaiting this thread waking up and draining >>>>> its queue. So for linux-user I think we need some mechanism to kick >>>>> these syscalls which I assume means throwing a signal at it. >>>> Nice catch! How did you get it? >>> Running pigz (armhf, debian) to compress stuff. >>> >>>> We always go through cpu_exec_end() >>>> before serving a guest syscall and always go through cpu_exec_start() >>>> before entering the guest code execution loop. If we always schedule >>>> safe work on the current thread's queue then I think there's a way to >>>> make it safe and avoid kicking syscalls. >>> Not let the signals complete until safe work is done? >> I'm thinking of waiting for completion of safe works in cpu_exec_start() >> as well as in cpu_exec_end(). > I found a mistake in my code which causes deadlocks in my run of pigz. > Could you also try running it after applying the following patch? > > diff --git a/linux-user/main.c b/linux-user/main.c > index 6da3bb32186b..1dca55145c56 100644 > --- a/linux-user/main.c > +++ b/linux-user/main.c > @@ -214,7 +214,7 @@ static inline void cpu_exec_end(CPUState *cpu) > cpu->running = false; > tcg_pending_cpus--; > if (!tcg_pending_cpus) { > - pthread_cond_broadcast(&exclusive_cond); > + signal_cpu_work(); > } > exclusive_idle(); > flush_queued_work(cpu); > > Or better this one: /* Make sure everything is in a consistent state for calling fork(). */ @@ -140,7 +139,6 @@ void fork_end(int child) pthread_mutex_init(&cpu_list_mutex, NULL); pthread_cond_init(&exclusive_cond, NULL); pthread_cond_init(&exclusive_resume, NULL); - pthread_cond_init(&work_cond, NULL); qemu_mutex_init(&tcg_ctx.tb_ctx.tb_lock); gdbserver_fork(thread_cpu); } else { @@ -151,12 +149,12 @@ void fork_end(int child) void wait_cpu_work(void) { - pthread_cond_wait(&work_cond, &exclusive_lock); + pthread_cond_wait(&exclusive_cond, &exclusive_lock); } void signal_cpu_work(void) { - pthread_cond_broadcast(&work_cond); + pthread_cond_broadcast(&exclusive_cond); } /* Wait for pending exclusive operations to complete. The exclusive lock @@ -214,7 +212,7 @@ static inline void cpu_exec_end(CPUState *cpu) cpu->running = false; tcg_pending_cpus--; if (!tcg_pending_cpus) { - pthread_cond_broadcast(&exclusive_cond); + signal_cpu_work(); } exclusive_idle(); flush_queued_work(cpu); Thanks, Sergey diff --git a/linux-user/main.c b/linux-user/main.c index 6da3bb32186b..aba550b69611 100644 --- a/linux-user/main.c +++ b/linux-user/main.c @@ -111,7 +111,6 @@ static pthread_mutex_t cpu_list_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_mutex_t exclusive_lock = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t exclusive_cond = PTHREAD_COND_INITIALIZER; static pthread_cond_t exclusive_resume = PTHREAD_COND_INITIALIZER; -static pthread_cond_t work_cond = PTHREAD_COND_INITIALIZER; static bool exclusive_pending;