From patchwork Wed Feb 3 15:29:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 12064591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FAA5C433E0 for ; Wed, 3 Feb 2021 15:29:32 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BD1AD64E4B for ; Wed, 3 Feb 2021 15:29:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD1AD64E4B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 900C56EB16; Wed, 3 Feb 2021 15:29:30 +0000 (UTC) Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A11C6EB16 for ; Wed, 3 Feb 2021 15:29:29 +0000 (UTC) Received: by mail-wr1-x429.google.com with SMTP id v15so24870544wrx.4 for ; Wed, 03 Feb 2021 07:29:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dX5VWhzdyCSXKOkzbFh/rppPL8SS0wtBsuXFORopB0E=; b=XopLrHKZbh8IvKLagLigdfsTrY8sxR+m6Xie6uCiR53EextjydiFKe5V6Zo+ctkjqU TM2/sYWjHi7tScNkKM4zbwFzfqrGn+XBtdHx+y27ZOUIXFJvowhu2Yw3RsI7uMPl/j+z tw0mjJGsjsICgamEykClEFgk+oHPA/vn3vJv4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dX5VWhzdyCSXKOkzbFh/rppPL8SS0wtBsuXFORopB0E=; b=r1qb5h3dLPwmWM55g301GpE7DhCbIT6L5LptYB1o/1NDqNUlGH1rbIM9ykkCOshq5v +lHpriMLNxawdEljfDMbkbjm7oEB/Zr3VoW/qgy/85SVnq8ihlrHtYffZ22lUie0/GVq qy1BTL62fqZNRuPNRLrbdiBgjt4P5rL7iiby8K70C0j4mkacWwQtGhD1I1i/Sc4X1L8Q NGxchTa8xydXERkDjy8YtdkjvQlKVZVujfLoOAYtEJ6jD6kI4agu5wJAjlwVECz8bJ2r bpqoVdFZSczlSkupmUfrEXV2DplVPvWdVK65tke0ROD6rGu1RHN4moPxXfLNY8qZBB1X Rkbg== X-Gm-Message-State: AOAM531xo+p/9BpcfWOjDmzczdZui+A/tQEtpTIrwAr9hfytHB8R9IJm MGnZxK0lrfVJyN1JTdBMxMF54eKhX1uVZtiX X-Google-Smtp-Source: ABdhPJzVLJmtYENS2YgeQZySm5SoeXYyzpaLIlSonXEHkElg5fxVGgKG2F6eexgXJaLvEh8wld5viA== X-Received: by 2002:adf:83a6:: with SMTP id 35mr4126060wre.274.1612366167884; Wed, 03 Feb 2021 07:29:27 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id l1sm2779528wmq.17.2021.02.03.07.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:29:27 -0800 (PST) From: Daniel Vetter To: DRI Development Subject: [PATCH] dma-fence: Document recoverable page fault implications Date: Wed, 3 Feb 2021 16:29:21 +0100 Message-Id: <20210203152921.2429937-1-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Vetter , Felix Kuehling , linaro-mm-sig@lists.linaro.org, Jerome Glisse , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Daniel Vetter , =?utf-8?q?Christian_K=C3=B6nig?= , linux-media@vger.kernel.org Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Recently there was a fairly long thread about recoreable hardware page faults, how they can deadlock, and what to do about that. While the discussion is still fresh I figured good time to try and document the conclusions a bit. This documentation section explains what's the potential problem, and the remedies we've discussed, roughly ordered from best to worst. v2: Linus -> Linux typoe (Dave) v3: - Make it clear drivers only need to implement one option (Christian) - Make it clearer that implicit sync is out the window with exclusive fences (Christian) - Add the fairly theoretical option of segementing the memory (either statically or through dynamic checks at runtime for which piece of memory is managed how) and explain why it's not a great idea (Felix) References: https://lore.kernel.org/dri-devel/20210107030127.20393-1-Felix.Kuehling@amd.com/ Cc: Dave Airlie Cc: Maarten Lankhorst Cc: Thomas Hellström Cc: "Christian König" Cc: Jerome Glisse Cc: Felix Kuehling Signed-off-by: Daniel Vetter Cc: Sumit Semwal Cc: linux-media@vger.kernel.org Cc: linaro-mm-sig@lists.linaro.org Reviewed-by: Christian König Acked-by: Thomas Hellström --- Documentation/driver-api/dma-buf.rst | 76 ++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst index a2133d69872c..7f37ec30d9fd 100644 --- a/Documentation/driver-api/dma-buf.rst +++ b/Documentation/driver-api/dma-buf.rst @@ -257,3 +257,79 @@ fences in the kernel. This means: userspace is allowed to use userspace fencing or long running compute workloads. This also means no implicit fencing for shared buffers in these cases. + +Recoverable Hardware Page Faults Implications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Modern hardware supports recoverable page faults, which has a lot of +implications for DMA fences. + +First, a pending page fault obviously holds up the work that's running on the +accelerator and a memory allocation is usually required to resolve the fault. +But memory allocations are not allowed to gate completion of DMA fences, which +means any workload using recoverable page faults cannot use DMA fences for +synchronization. Synchronization fences controlled by userspace must be used +instead. + +On GPUs this poses a problem, because current desktop compositor protocols on +Linux rely on DMA fences, which means without an entirely new userspace stack +built on top of userspace fences, they cannot benefit from recoverable page +faults. Specifically this means implicit synchronization will not be possible. +The exception is when page faults are only used as migration hints and never to +on-demand fill a memory request. For now this means recoverable page +faults on GPUs are limited to pure compute workloads. + +Furthermore GPUs usually have shared resources between the 3D rendering and +compute side, like compute units or command submission engines. If both a 3D +job with a DMA fence and a compute workload using recoverable page faults are +pending they could deadlock: + +- The 3D workload might need to wait for the compute job to finish and release + hardware resources first. + +- The compute workload might be stuck in a page fault, because the memory + allocation is waiting for the DMA fence of the 3D workload to complete. + +There are a few options to prevent this problem, one of which drivers need to +ensure: + +- Compute workloads can always be preempted, even when a page fault is pending + and not yet repaired. Not all hardware supports this. + +- DMA fence workloads and workloads which need page fault handling have + independent hardware resources to guarantee forward progress. This could be + achieved through e.g. through dedicated engines and minimal compute unit + reservations for DMA fence workloads. + +- The reservation approach could be further refined by only reserving the + hardware resources for DMA fence workloads when they are in-flight. This must + cover the time from when the DMA fence is visible to other threads up to + moment when fence is completed through dma_fence_signal(). + +- As a last resort, if the hardware provides no useful reservation mechanics, + all workloads must be flushed from the GPU when switching between jobs + requiring DMA fences or jobs requiring page fault handling: This means all DMA + fences must complete before a compute job with page fault handling can be + inserted into the scheduler queue. And vice versa, before a DMA fence can be + made visible anywhere in the system, all compute workloads must be preempted + to guarantee all pending GPU page faults are flushed. + +- Only a fairly theoretical option would be to untangle these dependencies when + allocating memory to repair hardware page faults, either through separate + memory blocks or runtime tracking of the full dependency graph of all DMA + fences. This results very wide impact on the kernel, since resolving the page + on the CPU side can itself involve a page fault. It is much more feasible and + robust to limit the impact of handling hardware page faults to the specific + driver. + +Note that workloads that run on independent hardware like copy engines or other +GPUs do not have any impact. This allows us to keep using DMA fences internally +in the kernel even for resolving hardware page faults, e.g. by using copy +engines to clear or copy memory needed to resolve the page fault. + +In some ways this page fault problem is a special case of the `Infinite DMA +Fences` discussions: Infinite fences from compute workloads are allowed to +depend on DMA fences, but not the other way around. And not even the page fault +problem is new, because some other CPU thread in userspace might +hit a page fault which holds up a userspace fence - supporting page faults on +GPUs doesn't anything fundamentally new.