From patchwork Mon Jun 25 13:31:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Michal Hocko X-Patchwork-Id: 10486151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0C648603B5 for ; Mon, 25 Jun 2018 13:32:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F20FD27F92 for ; Mon, 25 Jun 2018 13:32:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E494727FA9; Mon, 25 Jun 2018 13:32:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59A5C27F92 for ; Mon, 25 Jun 2018 13:32:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E18606B0005; Mon, 25 Jun 2018 09:32:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DC7086B0006; Mon, 25 Jun 2018 09:32:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C68066B0007; Mon, 25 Jun 2018 09:32:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 5FB646B0005 for ; Mon, 25 Jun 2018 09:32:01 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id i19-v6so2332080eds.20 for ; Mon, 25 Jun 2018 06:32:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=OekeeTrAh7WlJlQ02voRgPNcnyzaJNmS6YGOs95FY7c=; b=eKzUpquEaGr/Ro/CJD5w0Fn79FUNicKgPB7rqoCthFYywM4JgkJnGzoxKcRvnKLN60 92CB+1iIOdXNk+ySuJ8vGj8PGVrYWqdva2s/iRD9RyhQjmWTawyyyl8fS8dJ8T8quKgR ukDYb/Yv0O8gUFlR8PzoVN7FLdz4rrAvc86sH5WSCtODix+HAwTOr0sZzsBzkkZywQtw sMy3aOblCRRuQwOoldimkan2SsHkd3NYk0eX+pLmDS49Nw6jmw5m29xfLUDJGZodCq/V dsBdm30RScNHisnwZr7Gnc3Oq8SFiuIGJksvpb6Y1x/N6PeHcg0R9yMz2R8iqt+4NjyN vBSw== X-Original-Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning mhocko@kernel.org does not designate 195.135.220.15 as permitted sender) smtp.mailfrom=mhocko@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Gm-Message-State: APt69E3t9tjTWQ0u9AlbGXz3xDOB3hfflvjR1Wmj8u4li01Vsanbn+XB /FJWeD/Yt9EWQWBYCasiDDndi9/T2ljTBQgXamAda/Rgafo3OyDEiC1uSwpBAbIhIzvkszZ1wQB +7MHirgAfQRMvIPLaEteOhF8iteFjeBmYkVHAQWSYxwE0UMynVL3WRzuBgcy8EGc= X-Received: by 2002:a50:a106:: with SMTP id 6-v6mr11686587edj.12.1529933520798; Mon, 25 Jun 2018 06:32:00 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJydSjhCUePhl/1S4Lcj3c3eUkr/YFZDuazMG96AQretuchaBinqCGs2bkfXRyMJ/5K7HFx X-Received: by 2002:a50:a106:: with SMTP id 6-v6mr11686532edj.12.1529933519783; Mon, 25 Jun 2018 06:31:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529933519; cv=none; d=google.com; s=arc-20160816; b=ZkzfQQ//3K4p4iZ4bs70LuZ2dop25coSj6BPVAhvvbxQYss/D1KmIr3ls8nElV75Kj xavtYHncbW9YUVMEy12nqHv0FAYALUzZ8mPQbFXv/5CaE5WY3EFrEcCv238bbXlUdecf 4vDDbL+lUKDjbycoALRXLBJ0BSw5aOprb6W+Ct7UdYqKKARt2aGv8VcOoTb9iS9yGpwS 4Dfzrc31NuJzcQStXOpO/g0oWQFsbKzl3S87SEVy1avpa1auC9IpYLj3HqWKMaD9FI/E QMSpZ/m/pfHADEs6FfNtCuzcxR7HaCmidw4J9CBYQQd86fVq421BNWidYdO3kmAPg+qW euVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:arc-authentication-results; bh=OekeeTrAh7WlJlQ02voRgPNcnyzaJNmS6YGOs95FY7c=; b=hjRrrstYOi2to5GiyRrohjnUGhChFNdOmUR7pWm/MrqHRMSnz6qjaIKnnCfDwrV2YZ zuC+7/GSjeLgB+z1TqLVCueShHRCZ0Sa0PEkqNb1vdwtTfNrnMltbfTduwGv+oOd2E+B R5etOVlyLmYfUKPfITyfwxhShv8FHwl4kZmBRfoucLi/pMRn0DeNTTSHgHcOhnXRH4bz j+7/dtUsEmmEbKchdMk6A2Oc9iFTEU5AkLX9YSutn4bddz190aK3bsZfHi1HlgQ6tavD uwz1YuU8eOQ6Rdr4BYZ9TQ2q3uOmWUQ/Q6BNvdB1F37vwjy7CLVauOMNTNwnJbwFC33E KI6g== ARC-Authentication-Results: i=1; mx.google.com; spf=softfail (google.com: domain of transitioning mhocko@kernel.org does not designate 195.135.220.15 as permitted sender) smtp.mailfrom=mhocko@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id a24-v6si7059569edc.227.2018.06.25.06.31.59 for (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 25 Jun 2018 06:31:59 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning mhocko@kernel.org does not designate 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning mhocko@kernel.org does not designate 195.135.220.15 as permitted sender) smtp.mailfrom=mhocko@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext-too.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 0AC1AAEE3; Mon, 25 Jun 2018 13:31:59 +0000 (UTC) Date: Mon, 25 Jun 2018 15:31:57 +0200 From: Michal Hocko To: Felix Kuehling Cc: Christian =?iso-8859-1?Q?K=F6nig?= , LKML , "David (ChunMing) Zhou" , Paolo Bonzini , Radim =?utf-8?B?S3LEjW3DocWZ?= , Alex Deucher , David Airlie , Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , Doug Ledford , Jason Gunthorpe , Mike Marciniszyn , Dennis Dalessandro , Sudeep Dutt , Ashutosh Dixit , Dimitri Sivanich , Boris Ostrovsky , Juergen Gross , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrea Arcangeli , kvm@vger.kernel.org, amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-rdma@vger.kernel.org, xen-devel@lists.xenproject.org, linux-mm@kvack.org, David Rientjes Subject: Re: [RFC PATCH] mm, oom: distinguish blockable mode for mmu notifiers Message-ID: <20180625133157.GL28965@dhcp22.suse.cz> References: <20180622150242.16558-1-mhocko@kernel.org> <0aa9f695-5702-6704-9462-7779cbfdb3fd@amd.com> <20180622152444.GC10465@dhcp22.suse.cz> <20180625080103.GB28965@dhcp22.suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180625080103.GB28965@dhcp22.suse.cz> User-Agent: Mutt/1.9.5 (2018-04-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP On Mon 25-06-18 10:01:03, Michal Hocko wrote: > On Fri 22-06-18 16:09:06, Felix Kuehling wrote: > > On 2018-06-22 11:24 AM, Michal Hocko wrote: > > > On Fri 22-06-18 17:13:02, Christian König wrote: > > >> Hi Michal, > > >> > > >> [Adding Felix as well] > > >> > > >> Well first of all you have a misconception why at least the AMD graphics > > >> driver need to be able to sleep in an MMU notifier: We need to sleep because > > >> we need to wait for hardware operations to finish and *NOT* because we need > > >> to wait for locks. > > >> > > >> I'm not sure if your flag now means that you generally can't sleep in MMU > > >> notifiers any more, but if that's the case at least AMD hardware will break > > >> badly. In our case the approach of waiting for a short time for the process > > >> to be reaped and then select another victim actually sounds like the right > > >> thing to do. > > > Well, I do not need to make the notifier code non blocking all the time. > > > All I need is to ensure that it won't sleep if the flag says so and > > > return -EAGAIN instead. > > > > > > So here is what I do for amdgpu: > > > > In the case of KFD we also need to take the DQM lock: > > > > amdgpu_mn_invalidate_range_start_hsa -> amdgpu_amdkfd_evict_userptr -> > > kgd2kfd_quiesce_mm -> kfd_process_evict_queues -> evict_process_queues_cpsch > > > > So we'd need to pass the blockable parameter all the way through that > > call chain. > > Thanks, I have missed that part. So I guess I will start with something > similar to intel-gfx and back off when the current range needs some > treatment. So this on top. Does it look correct? > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c > index d138a526feff..e2d422b3eb0b 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c > @@ -266,6 +266,11 @@ static int amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn, > struct amdgpu_mn_node *node; > struct amdgpu_bo *bo; > > + if (!blockable) { > + amdgpu_mn_read_unlock(); > + return -EAGAIN; > + } > + > node = container_of(it, struct amdgpu_mn_node, it); > it = interval_tree_iter_next(it, start, end); Ble, just noticed that half of the change didn't get to git index... This is what I have commit c4701b36ac2802b903db3d05cf77c030fccce3a8 Author: Michal Hocko Date: Mon Jun 25 15:24:03 2018 +0200 fold me - amd gpu notifiers can sleep deeper in the callchain (evict_process_queues_cpsch on a lock and amdgpu_mn_invalidate_node on unbound timeout) make sure we bail out when we have an intersecting range for starter diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c index d138a526feff..3399a4a927fb 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c @@ -225,6 +225,11 @@ static int amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn, while (it) { struct amdgpu_mn_node *node; + if (!blockable) { + amdgpu_mn_read_unlock(rmn); + return -EAGAIN; + } + node = container_of(it, struct amdgpu_mn_node, it); it = interval_tree_iter_next(it, start, end); @@ -266,6 +271,11 @@ static int amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn, struct amdgpu_mn_node *node; struct amdgpu_bo *bo; + if (!blockable) { + amdgpu_mn_read_unlock(rmn); + return -EAGAIN; + } + node = container_of(it, struct amdgpu_mn_node, it); it = interval_tree_iter_next(it, start, end);