From patchwork Fri May 24 08:12:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2DCAD76 for ; Fri, 24 May 2019 08:12:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F6C32884F for ; Fri, 24 May 2019 08:12:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 132132876C; Fri, 24 May 2019 08:12:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4C7D285F9 for ; Fri, 24 May 2019 08:12:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4DCA6B0006; Fri, 24 May 2019 04:12:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BD7616B0007; Fri, 24 May 2019 04:12:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9F416B0008; Fri, 24 May 2019 04:12:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by kanga.kvack.org (Postfix) with ESMTP id 7FEF76B0006 for ; Fri, 24 May 2019 04:12:44 -0400 (EDT) Received: by mail-ot1-f70.google.com with SMTP id a22so4118027otr.21 for ; Fri, 24 May 2019 01:12:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=f5DG36R8dorxM7QpzbvNwRCVr1cC7ERIoXVIrITIVeo=; b=IENBJWnwMS2LEOO7HFvBJUDl5Pe/6zaHahpCCGATyhELEAhN1Xqzi/+svQ1tXKQS97 rGCAJiWq0LrPvR046wZahTKvsCoByV5iBrHM7MW/gB8b3YnT/VBjKcqo4SUGsxf+CBav vr2AizJGnbpVBjDyaoNrRFnkVS+cwWxKzEREQcx+F999uLTP62vP49Yqr0W8e1InXuFu GdDN8lrDKNyj/1mfo5YMQrqrFtTMJyi62JYRquNEKW8W7R4LMB+hGSAPPNhdg/gQ8Vfz qlccoS+tvRQmd/I9p1IbdaRgVjRkg6vkVudYQOI7AY2VuPEUmQCmnhGnnsqFiYkpdXtK LMfQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAUsHzkjJOmudYF9YglCK9yA77ffI+hTUD/BzntfTwv6c3SYobbZ OxeLwK343WYnj2vnU7QyzYW+Hfpz9nIIzhfuox4J3vOuRGrhsccebLDNMBg6mEdudDcZkU5z8vJ EONGRWVrc6ATNMZHmkmH5YPCsBZlMxuj6owWb4EKElvXIi6DmvxovRpZDJc83c80alQ== X-Received: by 2002:a05:6830:148e:: with SMTP id s14mr14870186otq.54.1558685564231; Fri, 24 May 2019 01:12:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqy1iXvjxg2awNdnGqmoI0znCzFK81mbPoRVwjOc+5M41vW1w9gjuNrbxTDjqGdh84cDT5Se X-Received: by 2002:a05:6830:148e:: with SMTP id s14mr14870159otq.54.1558685563393; Fri, 24 May 2019 01:12:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685563; cv=none; d=google.com; s=arc-20160816; b=gpg8bm4ICrjM7A2RrZqf1srlJljCOeTHjb0hQGb+XTfXFGb1h5nglz7zAlfpENTMcJ HKGGHe7559HzfCMe+3OFsuF+KsFVNItGkzL8RUuaQJNUh7CIbuHAbqy6j3+qf7tVordK uGVW6vzpfxhWXJ297UIwbiOjVb+mCTVjzTzbPUplBiHV3srjgb6CI9f7hmBe7Idmd3W9 LjJwMQAu7SStFYs8VNB1u3pWJYkmN1yJSgbRdGPLx6aJDzCEL++0MRosgUAOH4KiVyg4 d0mhZQyllYuoBDV88MY8Jfo3XtdGJZ4jFt60gFR8Ee+ZJqjT7wHhsQe4Th/eZOo8Vamx 2EOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=f5DG36R8dorxM7QpzbvNwRCVr1cC7ERIoXVIrITIVeo=; b=MTG59LrAJUmqPAA4olks1uFB/XSb+iX3f6vbsOjJofn48GSVHuItHj8f+HRW9TfY6d z6Esj7ZISwC9sdD3jQ+KzDwA2tFBwK9+uPqWOHz7wZBkv1YEOsaacXLBaxx2cr9nRKnf fNYmlvzFOJryndaJafhAXQRisUtiE0fordBWQ8vTWBJ1nydZqooQkzZvLASEebVwmdbU zc15NdfHs2bRPqnPh8Btcka1EfROU9wqwxuOFRtlPLU5w643Dra94HgIlds/LQv2RCYL WPAdmJeyYS+7iINYgT3I2WXMdp8VbbALGHsVC5oZA3RoyXJUWtMLjQaMeqcWhqMV8h4h avzg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id c3si1055512otp.40.2019.05.24.01.12.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:12:43 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9DCAC3092654; Fri, 24 May 2019 08:12:37 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id D650119C4F; Fri, 24 May 2019 08:12:31 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [PATCH net-next 1/6] vhost: generalize adding used elem Date: Fri, 24 May 2019 04:12:13 -0400 Message-Id: <20190524081218.2502-2-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Fri, 24 May 2019 08:12:42 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use one generic vhost_copy_to_user() instead of two dedicated accessor. This will simplify the conversion to fine grain accessors. About 2% improvement of PPS were seen during vitio-user txonly test. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 1e3ed41ae1f3..47fb3a297c29 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2255,16 +2255,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (count == 1) { - if (vhost_put_user(vq, heads[0].id, &used->id)) { - vq_err(vq, "Failed to write used id"); - return -EFAULT; - } - if (vhost_put_user(vq, heads[0].len, &used->len)) { - vq_err(vq, "Failed to write used len"); - return -EFAULT; - } - } else if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { vq_err(vq, "Failed to write used"); return -EFAULT; } From patchwork Fri May 24 08:12:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B01A114B6 for ; Fri, 24 May 2019 08:12:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E77528754 for ; Fri, 24 May 2019 08:12:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 921CC28820; Fri, 24 May 2019 08:12:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DABC62882F for ; Fri, 24 May 2019 08:12:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B63A86B0007; Fri, 24 May 2019 04:12:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AEC896B0008; Fri, 24 May 2019 04:12:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B5E86B000A; Fri, 24 May 2019 04:12:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by kanga.kvack.org (Postfix) with ESMTP id 6FED46B0007 for ; Fri, 24 May 2019 04:12:48 -0400 (EDT) Received: by mail-ot1-f70.google.com with SMTP id x23so4134756otp.5 for ; Fri, 24 May 2019 01:12:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=qb96nu4CUkjJTYQbuYkg+ri4uz9s8zY59/jg+fy8rlY=; b=Qmh7qkEjpI6jQXbw3kQA2JREnbS9gbmVZmVsK/ctb/+vuk+zIv9kK3iHg5uqNMhU4B lyWSOkggV1+vix+eZ/5j66t5Ba1lSlr0eQXeb0FU9AkEY2pfSAdUDotMJlepszfZozBZ 1mEwAJmaudUuiZqmYkNZ3wcd3aFAuMk0Q0/SLVEiAwe4FWJaOd5DWyti7CWm8Q7q3rhX r/Kvc3rbw9VY3VporSd4/65dSnhGAiswd2vUMg7HQP7qj9e11F0cZReqoE6DHmiWl/n5 kzuBc1IMI0SPGyYBKkOyj7AokVKD5soBRCMOzU1N/vDvRvuo3Xh4DN5axmB2DVgMZJnb Xp8Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAVmEqkUhCbVTwavfY0sBlV8RyedFZnMjUTheY+V6nUp3t1JIRUq kpQwXypjllrCuc6vRwwiG4LO03l0L5n/YJvoN/nS6WJc/mU+f+W/44lcM9OYxMb3rpF0MrS1VyL /fYBerjTEj7t6UBt228cWVhvXLp9qcGPihKgVZmGAP+5G8/zhycau/qrpmzBeTylatw== X-Received: by 2002:a9d:629a:: with SMTP id x26mr38890091otk.7.1558685568158; Fri, 24 May 2019 01:12:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqw86PhFNc2PZpuSFolU/fP/QoD0k2lDdpO6cf8fK9b21m9gSpY/8aaKpd4eJ5OoZMFXZJZ2 X-Received: by 2002:a9d:629a:: with SMTP id x26mr38890044otk.7.1558685566995; Fri, 24 May 2019 01:12:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685566; cv=none; d=google.com; s=arc-20160816; b=eu7f1zeoZxx/D/rIrjT7fAud0DU6s/w+DOgb1Y+InCscrI5jiUN+Go235hXKatmQ2R ST8TVix+NXmwuCPz2xC62H1m2mj2bJNFdJuw71YN55rbSSbuJsUmc3hgVsCVP3whWvIJ Mg3BW49ZGWkyxYxgkjTvHiHZQ1zERUztUIeOyN+mWy+V2cnOcGwbARrtYglcYpLaBz7Q tPJh1PXeeRk0AcGsXMsjAE2qjprNNCWnG1Q9EWuwrG5mpN0T5prNz9ZqZ6WEPGfmBGrD 6hXKLl8rt71ALBct/VVkgHGH7vuyK2TEj2SmJrGvMQ/wIm+MArJgparNkrwv1wFL9O02 zVVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=qb96nu4CUkjJTYQbuYkg+ri4uz9s8zY59/jg+fy8rlY=; b=dyFWs2HHnV3cbNQH8CK4LYntpEAcOTyGhz0xQk6SaPTqhzx6rXFLCnnIrBa+B8wFlC iUix3aHhL4rmKbjC2JmOKsPSUK+CuN01mzUSMbpgT5oyWpqTECtAiOc2HSXIhHvkJjPT J12s89IZ2ol2ytgWsSuMLjTOK4W/lxHYfnG6KKadrb0RE6TBX1pFsPN0gE5a9uySU7mi 7gglJU5fr9duV0RVOFsFrY9p4d9GdeMXZEDQ0j23MIttZQyqcI4vBaYFIc9Mw8K9rssU IjmtZ0v3hPYuoAMXi0uhbflanw+9TBukE+pq90gOvFh1yaCxWRM1qiZpjqxVvqaLbVSu ja8w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id d2si1136246oig.46.2019.05.24.01.12.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:12:46 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D9DDB8110A; Fri, 24 May 2019 08:12:44 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 263D419C58; Fri, 24 May 2019 08:12:37 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [PATCH net-next 2/6] vhost: fine grain userspace memory accessors Date: Fri, 24 May 2019 04:12:14 -0400 Message-Id: <20190524081218.2502-3-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 24 May 2019 08:12:46 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is used to hide the metadata address from virtqueue helpers. This will allow to implement a vmap based fast accessing to metadata. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 94 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 17 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 47fb3a297c29..e78c195448f0 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -869,6 +869,34 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, ret; \ }) +static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), + vhost_avail_event(vq)); +} + +static inline int vhost_put_used(struct vhost_virtqueue *vq, + struct vring_used_elem *head, int idx, + int count) +{ + return vhost_copy_to_user(vq, vq->used->ring + idx, head, + count * sizeof(*head)); +} + +static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), + &vq->used->flags); +} + +static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), + &vq->used->idx); +} + #define vhost_get_user(vq, x, ptr, type) \ ({ \ int ret; \ @@ -907,6 +935,43 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) mutex_unlock(&d->vqs[i]->mutex); } +static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_avail(vq, *idx, &vq->avail->idx); +} + +static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, + __virtio16 *head, int idx) +{ + return vhost_get_avail(vq, *head, + &vq->avail->ring[idx & (vq->num - 1)]); +} + +static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, + __virtio16 *flags) +{ + return vhost_get_avail(vq, *flags, &vq->avail->flags); +} + +static inline int vhost_get_used_event(struct vhost_virtqueue *vq, + __virtio16 *event) +{ + return vhost_get_avail(vq, *event, vhost_used_event(vq)); +} + +static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_used(vq, *idx, &vq->used->idx); +} + +static inline int vhost_get_desc(struct vhost_virtqueue *vq, + struct vring_desc *desc, int idx) +{ + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); +} + static int vhost_new_umem_range(struct vhost_umem *umem, u64 start, u64 size, u64 end, u64 userspace_addr, int perm) @@ -1844,8 +1909,7 @@ EXPORT_SYMBOL_GPL(vhost_log_write); static int vhost_update_used_flags(struct vhost_virtqueue *vq) { void __user *used; - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), - &vq->used->flags) < 0) + if (vhost_put_used_flags(vq)) return -EFAULT; if (unlikely(vq->log_used)) { /* Make sure the flag is seen before log. */ @@ -1862,8 +1926,7 @@ static int vhost_update_used_flags(struct vhost_virtqueue *vq) static int vhost_update_avail_event(struct vhost_virtqueue *vq, u16 avail_event) { - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), - vhost_avail_event(vq))) + if (vhost_put_avail_event(vq)) return -EFAULT; if (unlikely(vq->log_used)) { void __user *used; @@ -1899,7 +1962,7 @@ int vhost_vq_init_access(struct vhost_virtqueue *vq) r = -EFAULT; goto err; } - r = vhost_get_used(vq, last_used_idx, &vq->used->idx); + r = vhost_get_used_idx(vq, &last_used_idx); if (r) { vq_err(vq, "Can't access used idx at %p\n", &vq->used->idx); @@ -2098,7 +2161,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, last_avail_idx = vq->last_avail_idx; if (vq->avail_idx == vq->last_avail_idx) { - if (unlikely(vhost_get_avail(vq, avail_idx, &vq->avail->idx))) { + if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { vq_err(vq, "Failed to access avail idx at %p\n", &vq->avail->idx); return -EFAULT; @@ -2125,8 +2188,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, /* Grab the next descriptor number they're advertising, and increment * the index we've seen. */ - if (unlikely(vhost_get_avail(vq, ring_head, - &vq->avail->ring[last_avail_idx & (vq->num - 1)]))) { + if (unlikely(vhost_get_avail_head(vq, &ring_head, last_avail_idx))) { vq_err(vq, "Failed to read head: idx %d address %p\n", last_avail_idx, &vq->avail->ring[last_avail_idx % vq->num]); @@ -2161,8 +2223,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, i, vq->num, head); return -EINVAL; } - ret = vhost_copy_from_user(vq, &desc, vq->desc + i, - sizeof desc); + ret = vhost_get_desc(vq, &desc, i); if (unlikely(ret)) { vq_err(vq, "Failed to get descriptor: idx %d addr %p\n", i, vq->desc + i); @@ -2255,7 +2316,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_put_used(vq, heads, start, count)) { vq_err(vq, "Failed to write used"); return -EFAULT; } @@ -2297,8 +2358,7 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, /* Make sure buffer is written before we update index. */ smp_wmb(); - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), - &vq->used->idx)) { + if (vhost_put_used_idx(vq)) { vq_err(vq, "Failed to increment used idx"); return -EFAULT; } @@ -2331,7 +2391,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { __virtio16 flags; - if (vhost_get_avail(vq, flags, &vq->avail->flags)) { + if (vhost_get_avail_flags(vq, &flags)) { vq_err(vq, "Failed to get flags"); return true; } @@ -2345,7 +2405,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (unlikely(!v)) return true; - if (vhost_get_avail(vq, event, vhost_used_event(vq))) { + if (vhost_get_used_event(vq, &event)) { vq_err(vq, "Failed to get used event idx"); return true; } @@ -2390,7 +2450,7 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (vq->avail_idx != vq->last_avail_idx) return false; - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (unlikely(r)) return false; vq->avail_idx = vhost16_to_cpu(vq, avail_idx); @@ -2426,7 +2486,7 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) /* They could have slipped one in as we were doing that: make * sure it's written, then check again. */ smp_mb(); - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (r) { vq_err(vq, "Failed to check avail idx at %p: %d\n", &vq->avail->idx, r); From patchwork Fri May 24 08:12:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42D0E76 for ; Fri, 24 May 2019 08:13:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3303428751 for ; Fri, 24 May 2019 08:13:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2730A2882F; Fri, 24 May 2019 08:13:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A83FC2884F for ; Fri, 24 May 2019 08:12:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C26FB6B0008; Fri, 24 May 2019 04:12:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BD6EF6B000A; Fri, 24 May 2019 04:12:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9F136B000C; Fri, 24 May 2019 04:12:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by kanga.kvack.org (Postfix) with ESMTP id 7FE8A6B0008 for ; Fri, 24 May 2019 04:12:58 -0400 (EDT) Received: by mail-ot1-f69.google.com with SMTP id v16so4124176otp.17 for ; Fri, 24 May 2019 01:12:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=4Rl5lDKbaY3RNqz1r+a26UzztQrfNaFSiQ/pob2EKMg=; b=sbphttNeDx8P2aPT/kE0083AnkMFuFe2SIPAJAy+4+wB/Dcxd7LLRUf2Agke7pcfjY G3DEiIWjkxhiUs6nFGmyKIRtUpz+/XNChwaQ5GC/zWLIJZAi7o2c9AHbnfgXr8mASql+ qxVZ0LdNsQPwkltZNpI/b0sZ/68rjGtbIWwFGyo5WeQZ9XSg1/qauNjFx+Bmm/mt/M53 ilNYZ/B1sfAfFkuQ8+UaXn4UHhaTl4QrDz10ynDIa049MjILJMrNFGX5ckRhoSFsYogK bNXS/SmJZyPCStJiUE3gxWPt2s5wCl6Q5ttajtGq/GaV4ktlf6lLamJSrbTpMZMzBVTi tHSQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAVjLr06bT2WLrA0zvWmzByi1p18eTx90oqx27pUXrmR+MMQjsB+ trFaRbENJMHuBsN1pY5Zrb/TlRieTPDYSKbu7KeG6r75E8hNwByzN/MYzP5/ShTumbZZF9PQbT+ 1oOTYZH2txZ/vphzpcARrztQO8sUMI/pidvkOZTHi7+d7RNlb/a0Qc5rtW7s0MezcNg== X-Received: by 2002:aca:bb07:: with SMTP id l7mr421756oif.124.1558685578139; Fri, 24 May 2019 01:12:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqxvRRSI2fWe6uB3e+s57I29u9RU1BWej6dZfPaU18c3p8rD/0lsZ1ERYANOLIPevQ+EL3wV X-Received: by 2002:aca:bb07:: with SMTP id l7mr421729oif.124.1558685577387; Fri, 24 May 2019 01:12:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685577; cv=none; d=google.com; s=arc-20160816; b=nnHs+hzU8KPPyRuKJml03kkRdmLiFtPRH7KRF45m8GZJQ7NIX8pAaxAocR3sHXHA7w XBa0t9b6UdKMVRfNZnDerW+2chDpu1hoHaWuR4lZf5PqKZXWZYCLEY6gufFDb9hEQAAi KchNwyAb3T1iHfzEhb2Nf0S1xq8rVDKtT/dfCFZYh+C4q3Mt8zgYAAXNeuwsaWKw9ZLa JiLBDJTQnDEPH1Z5OLvlN6P0D30QEf/DBpEWPAbd9SAJn36HVz6PWfw9hHzTvV68B7sx 57wBZoNUqB14FMA2P7hxVO+a5ouPv3QTEpsfvWHnefN41C4+0+CAUMIVMQ8Cj7quzg74 U3vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=4Rl5lDKbaY3RNqz1r+a26UzztQrfNaFSiQ/pob2EKMg=; b=pxnbAHZushofghR/2rAPv0azQI4ARF0u+V/Inpg4LpytHioXV3NYgh0pUvLA6hvuhw nowKG4jMPKRK3SA5wE8mctrhiJzaMUysjBgzKEG1DGcSL2JrpHCU1givmgwje6rtQ57r g4VpCoY3CH/gb5OH210m1LM0pP6tBW/fUKAdgeg18tziO8gO8QWdiEwl1nyyTat+Fumg nMZ2rWdLeyxFiQw/jJbya//58mFp85FhUAXDgD9UXYdu05KhaojAqe+dw0AHBnN1s/qc jxdVN2IqJ5XMHvGKKOtYumCUP4BdmSrNhdoj9xk9WakUoPY/ZcTmUITrPXGW6M/+GiTf QTzA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id s8si1001575otp.321.2019.05.24.01.12.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:12:57 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A98B03082B6B; Fri, 24 May 2019 08:12:56 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8589E19C4F; Fri, 24 May 2019 08:12:45 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [PATCH net-next 3/6] vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() Date: Fri, 24 May 2019 04:12:15 -0400 Message-Id: <20190524081218.2502-4-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 24 May 2019 08:12:56 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Rename the function to be more accurate since it actually tries to prefetch vq metadata address in IOTLB. And this will be used by following patch to prefetch metadata virtual addresses. Signed-off-by: Jason Wang --- drivers/vhost/net.c | 4 ++-- drivers/vhost/vhost.c | 4 ++-- drivers/vhost/vhost.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index df51a35cf537..bf55f995ebae 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -971,7 +971,7 @@ static void handle_tx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); @@ -1140,7 +1140,7 @@ static void handle_rx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index e78c195448f0..b353a00094aa 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -1313,7 +1313,7 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, return true; } -int vq_iotlb_prefetch(struct vhost_virtqueue *vq) +int vq_meta_prefetch(struct vhost_virtqueue *vq) { size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; @@ -1332,7 +1332,7 @@ int vq_iotlb_prefetch(struct vhost_virtqueue *vq) num * sizeof(*vq->used->ring) + s, VHOST_ADDR_USED); } -EXPORT_SYMBOL_GPL(vq_iotlb_prefetch); +EXPORT_SYMBOL_GPL(vq_meta_prefetch); /* Can we log writes? */ /* Caller should have device mutex but not vq mutex */ diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 9490e7ddb340..7a7fc001265f 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -209,7 +209,7 @@ bool vhost_enable_notify(struct vhost_dev *, struct vhost_virtqueue *); int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, unsigned int log_num, u64 len, struct iovec *iov, int count); -int vq_iotlb_prefetch(struct vhost_virtqueue *vq); +int vq_meta_prefetch(struct vhost_virtqueue *vq); struct vhost_msg_node *vhost_new_msg(struct vhost_virtqueue *vq, int type); void vhost_enqueue_msg(struct vhost_dev *dev, From patchwork Fri May 24 08:12:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959299 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB58014B6 for ; Fri, 24 May 2019 08:13:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BBC1A1FE84 for ; Fri, 24 May 2019 08:13:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AFA7828751; Fri, 24 May 2019 08:13:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31A6228846 for ; Fri, 24 May 2019 08:13:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 485BC6B000A; Fri, 24 May 2019 04:13:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 410746B000C; Fri, 24 May 2019 04:13:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8296B000D; Fri, 24 May 2019 04:13:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by kanga.kvack.org (Postfix) with ESMTP id F085A6B000A for ; Fri, 24 May 2019 04:13:03 -0400 (EDT) Received: by mail-ot1-f72.google.com with SMTP id t17so4115410otp.19 for ; Fri, 24 May 2019 01:13:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=m6tK+A2i9kvOrNp8SLo2LyEmYDirSbo50rDKkd7XZTw=; b=JJo3CFelQPORhooV2LWjlf0DLeV47lD5ER5jncpwrD4kXexqpeuIOkdDihG/TbMCiQ OOWG/9392ArwqBGJeU238D4IysL+OrfNCzMdYNamGEQfokK7Aqvo8cuLcgoAns3OaaoN FQtut/4dtSbpHPxjyYikUpTx075n2J9gRQMH0iyFIR0lsQFwwiyhgC7+HW392X6T6thn O0Gh0ucJAMKI5VAKu/JBn6E2CR51Fv6QpeBLuSOy/cTFwuptL2Cw0y7JnkvOtbWRH46z GTJmepwzArehBCSb+tR2BnuZuTVswaZqQOJG/3jcrHZdK9OZA0VX+BdoDnB8vphyB2QT nC4w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAVHGjwB2tKXldd7nOSGJTnbLUJn4lE8/b1OQmwgWJ0dokktqCN4 20IRo0vuTx/8Axx5d8Fl11UhGUR0XI+FA2nwQ0sjJCMjmg3K7AiPhpJsgXKV1xNEXYeHysoAl/q D2IP3JjdKR+28vO8PB+ih6YE2bSI6R02+GjJE1gAf/N4RGHGN6V5YTC0SzBu1AFjAhA== X-Received: by 2002:aca:5350:: with SMTP id h77mr5775389oib.57.1558685583657; Fri, 24 May 2019 01:13:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqwIJ796SqJ6kGlk1g2Lb2ZIkv8l9DkCYAqhFaRcyqTX9f88ffNiRTyYI5RIJz9jKZ7bRTt9 X-Received: by 2002:aca:5350:: with SMTP id h77mr5775354oib.57.1558685582806; Fri, 24 May 2019 01:13:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685582; cv=none; d=google.com; s=arc-20160816; b=gioxYs0sDdsYieV8u5qmI6RNvAG6eypWU546SY1wuU9iJs577sPLWnwW9XhwnrRKlA aVkyTYhE/MvMsD83mpLB7q7oJjYFnlBVpBZIlTdrSjnuyfCRZr7RcrM8kuf6npHdrmw1 HpXAT1C8xa9FMTQjwW1eoHtS1e8Zkvl6kJ0s0sCbVKZL0VUtjQW4496nHOm3zIZJTiur qJtXoGJM9aZkd3psquvBmSWTzc+uoKqYGS2pA29BI2Tglg3AKKHGgAd+roEIWGLu5GYZ 3gHsHvizUHiq3BiKFmoxGwg5dsa72ycg1KckbpfysoPRYVivvkwF5IigGHo+TYhvaHZW 7aig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=m6tK+A2i9kvOrNp8SLo2LyEmYDirSbo50rDKkd7XZTw=; b=ZIg6vx222IXPjI4+SpFxeIQiauHSxqCWIWw2SCtc+WsMbalxpyUY1JX88sLbq7ticS InUE4w78AvNMmMEJYFiDm4vQILddSzYAys4UDFueUfkfDfpXDN3cSzmRwrtgCdNtBW6U Q6sAxH4tXJJjCvsuzFWHR1Rz8c2DmDTyxoaSyiPqN4YVnGpR3sYP5OFqwonB/xVGFpqe MNXUqUwt3adtX10hZpl23y1x1eiqaJS2OcdZcaSiFxYFKYH/WRETng3iLUP2lGo/i6rX Zs/il8LzWa01n8vcQZrQCqXz/RthBoShNxJ2brLsoJZ3aMRBbP2D5ZhjkZubloQrrm/m SMxA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id t21si1217266oih.211.2019.05.24.01.13.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:13:02 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id ED9AC9B424; Fri, 24 May 2019 08:13:01 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30D2719C4F; Fri, 24 May 2019 08:12:56 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [PATCH net-next 4/6] vhost: introduce helpers to get the size of metadata area Date: Fri, 24 May 2019 04:12:16 -0400 Message-Id: <20190524081218.2502-5-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Fri, 24 May 2019 08:13:02 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To avoid code duplication since it will be used by kernel VA prefetching. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 51 ++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index b353a00094aa..8605e44a7001 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -413,6 +413,32 @@ static void vhost_dev_free_iovecs(struct vhost_dev *dev) vhost_vq_free_iovecs(dev->vqs[i]); } +static size_t vhost_get_avail_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + size_t event __maybe_unused = + vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->avail) + + sizeof(*vq->avail->ring) * num + event; +} + +static size_t vhost_get_used_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + size_t event __maybe_unused = + vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->used) + + sizeof(*vq->used->ring) * num + event; +} + +static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + return sizeof(*vq->desc) * num; +} + void vhost_dev_init(struct vhost_dev *dev, struct vhost_virtqueue **vqs, int nvqs, int iov_limit) { @@ -1257,13 +1283,9 @@ static bool vq_access_ok(struct vhost_virtqueue *vq, unsigned int num, struct vring_used __user *used) { - size_t s __maybe_unused = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - - return access_ok(desc, num * sizeof *desc) && - access_ok(avail, - sizeof *avail + num * sizeof *avail->ring + s) && - access_ok(used, - sizeof *used + num * sizeof *used->ring + s); + return access_ok(desc, vhost_get_desc_size(vq, num)) && + access_ok(avail, vhost_get_avail_size(vq, num)) && + access_ok(used, vhost_get_used_size(vq, num)); } static void vhost_vq_meta_update(struct vhost_virtqueue *vq, @@ -1315,22 +1337,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, int vq_meta_prefetch(struct vhost_virtqueue *vq) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; if (!vq->iotlb) return 1; return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, - num * sizeof(*vq->desc), VHOST_ADDR_DESC) && + vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->avail, - sizeof *vq->avail + - num * sizeof(*vq->avail->ring) + s, + vhost_get_avail_size(vq, num), VHOST_ADDR_AVAIL) && iotlb_access_ok(vq, VHOST_ACCESS_WO, (u64)(uintptr_t)vq->used, - sizeof *vq->used + - num * sizeof(*vq->used->ring) + s, - VHOST_ADDR_USED); + vhost_get_used_size(vq, num), VHOST_ADDR_USED); } EXPORT_SYMBOL_GPL(vq_meta_prefetch); @@ -1347,13 +1365,10 @@ EXPORT_SYMBOL_GPL(vhost_log_access_ok); static bool vq_log_access_ok(struct vhost_virtqueue *vq, void __user *log_base) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - return vq_memory_access_ok(log_base, vq->umem, vhost_has_feature(vq, VHOST_F_LOG_ALL)) && (!vq->log_used || log_access_ok(log_base, vq->log_addr, - sizeof *vq->used + - vq->num * sizeof *vq->used->ring + s)); + vhost_get_used_size(vq, vq->num))); } /* Can we start vq? */ From patchwork Fri May 24 08:12:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D299714B6 for ; Fri, 24 May 2019 08:13:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C24A7285F9 for ; Fri, 24 May 2019 08:13:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B69E828820; Fri, 24 May 2019 08:13:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0C2B2885E for ; Fri, 24 May 2019 08:13:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C01926B000C; Fri, 24 May 2019 04:13:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BB0CE6B000D; Fri, 24 May 2019 04:13:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A50C56B000E; Fri, 24 May 2019 04:13:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by kanga.kvack.org (Postfix) with ESMTP id 7AD776B000C for ; Fri, 24 May 2019 04:13:15 -0400 (EDT) Received: by mail-ot1-f71.google.com with SMTP id x27so4133753ote.6 for ; Fri, 24 May 2019 01:13:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=n9srHPDVBP4VTzg82Kr5crrs8DkwIQG+abHb+DSGfoA=; b=uVMw6AI5HmPjq3ej6xHRq6KpmQCtZaVWXMDgFLKCq6GgelZG99WoMOQ8X57nKTj0vc 3VwS1jsKrCVSfKTItwI90qNprapJNP+qgfg1L9Jprxn2bE/HodspBSYzIS3pWol/jeB+ JxvxmHnTJEdRtQBW9JMdIXNhvjQlOoD+aQx635hmTq8HAaME3ShvjmujQNFVoqcn5Z4P hmrwo1TN/beNaOQyGDe1JQvX4xXnvpzjhMyqNNARDOu4xXzCbXO70ZHOgM1QgtVOJ+c2 vC4NMrk+rnHvRO3QeoZpnEQZiA0cED3FvViKN3tKIDqxSvZh9x4QRx//X+DzIXhwEChz nB1Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAWBvwMvOXYKzV52wxVTpkPglNrkg1qC08cDFd88W5scUy1qc33u BoGMNaqR+FmJifLEqV/oImsc1HSVAyTQtxwrLJFl0OVQijZC2UlvmcyZt7bU6YxoVz05ebNYDKE LrhzBpX4ClM7dwNTbKl54xHzuLfaFzH5PItEcfOQficvtph+MngTfnI6UI90tO+Kygg== X-Received: by 2002:aca:4202:: with SMTP id p2mr5534582oia.85.1558685595033; Fri, 24 May 2019 01:13:15 -0700 (PDT) X-Google-Smtp-Source: APXvYqy1PNjThOiSSXPEwV2zvgsJHimX+qvgF5QksomE56j6vvChg6KNcgv344HpUmAeVwd0Mlju X-Received: by 2002:aca:4202:: with SMTP id p2mr5534544oia.85.1558685593730; Fri, 24 May 2019 01:13:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685593; cv=none; d=google.com; s=arc-20160816; b=zSvheKizbU+owQrYsp+ZhfQgr/qHOBbt0ZAspacb2YTCmSvHfpvtqOdbMCbVogtMn7 hbd1uS5GZuE1tvjAGHhU872/czkUMQOX7ir6ctK951GVw966A3v2oeAZ9gg8hzmFRo1u mDerAA1xSsLiZ8yM/z/Xx93kYBmkb6FUwkQD+Fpf6yVWmw/1PviZo5yddrP+zsTSmQll vqoW6KPnav0jTU3PsojJOm43t1ekUTWElNVl1U5jP5NZQNDLEITuUr+11bAya/PUx72B RJqqsnuWGrOFCld8lVPmLhL+lKZajtQTt3oe4MF+59378/IqbjJ+Mzik+nrQlr60VbnW Z6ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=n9srHPDVBP4VTzg82Kr5crrs8DkwIQG+abHb+DSGfoA=; b=FEPjS4YRoBPDc3x64Zlk0hbKHNoJ0mmdwkwu4oui72oqKPWe2qI4WaXVGMHqmdS/mE DFIzViT6MHhNRXwpWYjuotXTpESdjOnnSkwkSa7OSfVYr92yzYtJQ4MWynVA0kye5gf7 N5L24wG/b3/YAP0LUEyvp670833n/YMOy1tAw2qWlvEZZ2Cvp0Ca9vqFvl+878ZOPBG1 /UmIA0gxw/aW0P3Zog1f5GZh/lAOrjq+RnhYCc6JHXSROl+zuc+2Hb1oQKoQvkFrXqdT dRnxwrwcIkJjuLdaQxmNuqLD0GXDqbEI+edzBXrVT4anv39Y1wRlmhHnC2dt+XroEdDA Ni3w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id w185si1274481oib.117.2019.05.24.01.13.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:13:13 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B85B23082263; Fri, 24 May 2019 08:13:08 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC31219724; Fri, 24 May 2019 08:13:02 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [PATCH net-next 5/6] vhost: factor out setting vring addr and num Date: Fri, 24 May 2019 04:12:17 -0400 Message-Id: <20190524081218.2502-6-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 24 May 2019 08:13:09 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Factoring vring address and num setting which needs special care for accelerating vq metadata accessing. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 177 ++++++++++++++++++++++++------------------ 1 file changed, 103 insertions(+), 74 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 8605e44a7001..8bbda1777c61 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -1468,6 +1468,104 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m) return -EFAULT; } +static long vhost_vring_set_num(struct vhost_dev *d, + struct vhost_virtqueue *vq, + void __user *argp) +{ + struct vhost_vring_state s; + + /* Resizing ring with an active backend? + * You don't want to do that. */ + if (vq->private_data) + return -EBUSY; + + if (copy_from_user(&s, argp, sizeof s)) + return -EFAULT; + + if (!s.num || s.num > 0xffff || (s.num & (s.num - 1))) + return -EINVAL; + vq->num = s.num; + + return 0; +} + +static long vhost_vring_set_addr(struct vhost_dev *d, + struct vhost_virtqueue *vq, + void __user *argp) +{ + struct vhost_vring_addr a; + + if (copy_from_user(&a, argp, sizeof a)) + return -EFAULT; + if (a.flags & ~(0x1 << VHOST_VRING_F_LOG)) + return -EOPNOTSUPP; + + /* For 32bit, verify that the top 32bits of the user + data are set to zero. */ + if ((u64)(unsigned long)a.desc_user_addr != a.desc_user_addr || + (u64)(unsigned long)a.used_user_addr != a.used_user_addr || + (u64)(unsigned long)a.avail_user_addr != a.avail_user_addr) + return -EFAULT; + + /* Make sure it's safe to cast pointers to vring types. */ + BUILD_BUG_ON(__alignof__ *vq->avail > VRING_AVAIL_ALIGN_SIZE); + BUILD_BUG_ON(__alignof__ *vq->used > VRING_USED_ALIGN_SIZE); + if ((a.avail_user_addr & (VRING_AVAIL_ALIGN_SIZE - 1)) || + (a.used_user_addr & (VRING_USED_ALIGN_SIZE - 1)) || + (a.log_guest_addr & (VRING_USED_ALIGN_SIZE - 1))) + return -EINVAL; + + /* We only verify access here if backend is configured. + * If it is not, we don't as size might not have been setup. + * We will verify when backend is configured. */ + if (vq->private_data) { + if (!vq_access_ok(vq, vq->num, + (void __user *)(unsigned long)a.desc_user_addr, + (void __user *)(unsigned long)a.avail_user_addr, + (void __user *)(unsigned long)a.used_user_addr)) + return -EINVAL; + + /* Also validate log access for used ring if enabled. */ + if ((a.flags & (0x1 << VHOST_VRING_F_LOG)) && + !log_access_ok(vq->log_base, a.log_guest_addr, + sizeof *vq->used + + vq->num * sizeof *vq->used->ring)) + return -EINVAL; + } + + vq->log_used = !!(a.flags & (0x1 << VHOST_VRING_F_LOG)); + vq->desc = (void __user *)(unsigned long)a.desc_user_addr; + vq->avail = (void __user *)(unsigned long)a.avail_user_addr; + vq->log_addr = a.log_guest_addr; + vq->used = (void __user *)(unsigned long)a.used_user_addr; + + return 0; +} + +static long vhost_vring_set_num_addr(struct vhost_dev *d, + struct vhost_virtqueue *vq, + unsigned int ioctl, + void __user *argp) +{ + long r; + + mutex_lock(&vq->mutex); + + switch (ioctl) { + case VHOST_SET_VRING_NUM: + r = vhost_vring_set_num(d, vq, argp); + break; + case VHOST_SET_VRING_ADDR: + r = vhost_vring_set_addr(d, vq, argp); + break; + default: + BUG(); + } + + mutex_unlock(&vq->mutex); + + return r; +} long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *argp) { struct file *eventfp, *filep = NULL; @@ -1477,7 +1575,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg struct vhost_virtqueue *vq; struct vhost_vring_state s; struct vhost_vring_file f; - struct vhost_vring_addr a; u32 idx; long r; @@ -1490,26 +1587,14 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg idx = array_index_nospec(idx, d->nvqs); vq = d->vqs[idx]; + if (ioctl == VHOST_SET_VRING_NUM || + ioctl == VHOST_SET_VRING_ADDR) { + return vhost_vring_set_num_addr(d, vq, ioctl, argp); + } + mutex_lock(&vq->mutex); switch (ioctl) { - case VHOST_SET_VRING_NUM: - /* Resizing ring with an active backend? - * You don't want to do that. */ - if (vq->private_data) { - r = -EBUSY; - break; - } - if (copy_from_user(&s, argp, sizeof s)) { - r = -EFAULT; - break; - } - if (!s.num || s.num > 0xffff || (s.num & (s.num - 1))) { - r = -EINVAL; - break; - } - vq->num = s.num; - break; case VHOST_SET_VRING_BASE: /* Moving base with an active backend? * You don't want to do that. */ @@ -1535,62 +1620,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg if (copy_to_user(argp, &s, sizeof s)) r = -EFAULT; break; - case VHOST_SET_VRING_ADDR: - if (copy_from_user(&a, argp, sizeof a)) { - r = -EFAULT; - break; - } - if (a.flags & ~(0x1 << VHOST_VRING_F_LOG)) { - r = -EOPNOTSUPP; - break; - } - /* For 32bit, verify that the top 32bits of the user - data are set to zero. */ - if ((u64)(unsigned long)a.desc_user_addr != a.desc_user_addr || - (u64)(unsigned long)a.used_user_addr != a.used_user_addr || - (u64)(unsigned long)a.avail_user_addr != a.avail_user_addr) { - r = -EFAULT; - break; - } - - /* Make sure it's safe to cast pointers to vring types. */ - BUILD_BUG_ON(__alignof__ *vq->avail > VRING_AVAIL_ALIGN_SIZE); - BUILD_BUG_ON(__alignof__ *vq->used > VRING_USED_ALIGN_SIZE); - if ((a.avail_user_addr & (VRING_AVAIL_ALIGN_SIZE - 1)) || - (a.used_user_addr & (VRING_USED_ALIGN_SIZE - 1)) || - (a.log_guest_addr & (VRING_USED_ALIGN_SIZE - 1))) { - r = -EINVAL; - break; - } - - /* We only verify access here if backend is configured. - * If it is not, we don't as size might not have been setup. - * We will verify when backend is configured. */ - if (vq->private_data) { - if (!vq_access_ok(vq, vq->num, - (void __user *)(unsigned long)a.desc_user_addr, - (void __user *)(unsigned long)a.avail_user_addr, - (void __user *)(unsigned long)a.used_user_addr)) { - r = -EINVAL; - break; - } - - /* Also validate log access for used ring if enabled. */ - if ((a.flags & (0x1 << VHOST_VRING_F_LOG)) && - !log_access_ok(vq->log_base, a.log_guest_addr, - sizeof *vq->used + - vq->num * sizeof *vq->used->ring)) { - r = -EINVAL; - break; - } - } - - vq->log_used = !!(a.flags & (0x1 << VHOST_VRING_F_LOG)); - vq->desc = (void __user *)(unsigned long)a.desc_user_addr; - vq->avail = (void __user *)(unsigned long)a.avail_user_addr; - vq->log_addr = a.log_guest_addr; - vq->used = (void __user *)(unsigned long)a.used_user_addr; - break; case VHOST_SET_VRING_KICK: if (copy_from_user(&f, argp, sizeof f)) { r = -EFAULT; From patchwork Fri May 24 08:12:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10959323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66BD21708 for ; Fri, 24 May 2019 08:13:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 579FF28852 for ; Fri, 24 May 2019 08:13:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B70928846; Fri, 24 May 2019 08:13:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD82D2884B for ; Fri, 24 May 2019 08:13:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB0E96B000E; Fri, 24 May 2019 04:13:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B3BAC6B0010; Fri, 24 May 2019 04:13:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B4B96B0266; Fri, 24 May 2019 04:13:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by kanga.kvack.org (Postfix) with ESMTP id 5F5056B000E for ; Fri, 24 May 2019 04:13:23 -0400 (EDT) Received: by mail-oi1-f200.google.com with SMTP id l12so3306600oii.10 for ; Fri, 24 May 2019 01:13:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=iAF7wUzieA7v4TExc3b/fzfpim+fLDDOKOz7EvkcgEk=; b=qhMhYghAHGdHzOZWjeos483VT5+k01khQDLztlSHro7bZbMb4F6Ouc32bxU6yPFj40 JTHUr6mrnx11kvli9DRMgzAQKFrgVxrtef3uW4CbEBOvjGALmQiDK/BU5xaBssbirnT1 zixi3cYY1NylUhxy3mm2Dw4gAXANIgXFsBPPdYmJ2zrViDOfux03GOsI3V3ae9QwuMv1 o1rWcYe6nVdt4lLvAU6fsAxbWipSrTlSoy9sA1tgVURbManOiTVkGSj3ULejpFJImaAx AnkUszJM5FTYmA7n43SMVg/i1FHwZUXYptijv69VILQJfFCWO/fSBK9xrCwwfAQF3mmO XkVg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAXIxElONbX4Pvvbn2YiLwmoF55whjQptfkMmkDyfL54ge6ftC5T gDVrWPBz4HsDZl03OJd0c87Shm/j85tb6bHL8tt8QYbeyAxNzfDDCaJ5PcXy037XUxrv6VmwLks orhcs6K5jjjJl3B9FbwARpuxId6x8ZqIOLPS/ycxD/sNqdzv+2HBxSSWKH4V5wuLZmA== X-Received: by 2002:a9d:7f83:: with SMTP id t3mr14603804otp.299.1558685603028; Fri, 24 May 2019 01:13:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqzl0pMnkldWVa9CPIReYWr6h24aDWLqrhGjs8tkOBn+xlS54+xlAQdt40YtHRIKTmMWYmH/ X-Received: by 2002:a9d:7f83:: with SMTP id t3mr14603764otp.299.1558685602025; Fri, 24 May 2019 01:13:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558685602; cv=none; d=google.com; s=arc-20160816; b=IczMirsdJZ86ZEM0b6mC5xYbfU7NudZ6gMAZBep4Fagw6iNFs3n4vMQHZDPKQcG2kn jQnPsXB5twhAjW/NLhBcwojuk4m0YRK8BdNCZ/zezciIbZiqAzXb9joxm4bFr2kvjRVB 03hyx4WDW+4SdOu1yb3R31WNqAJOqZ20Cxi8eL4SfgfIlUWbo+cvg1nssZOHofRO/dia kHPkb98KismG3lzJN9KSD1abBfRfO3xAXcoHwXeGxaNrzI09LfqL8AuWjBvTN58HFNC4 2+cp6ScjCHm/wnkU7FfTPvA/IOPKuDTMPOFc6507Sfy2GGNkqtL25BacLPnITglXEAxe Agnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=iAF7wUzieA7v4TExc3b/fzfpim+fLDDOKOz7EvkcgEk=; b=THQC8bVZs4r6ptZ5fsjhF0apgPZur1BJBRxwyIiVToB2gENMmFp9kV9zbEQBaAkSNV GkrHOMVMjwadkxueu3XCECtDr0yvA4rB5Fc2G2OKs8wQT44yJWtezqNz5thO0naDW/EF aQjKkzOP7mZTes+xo1JJHm9+alNmHzSnekimrzE+g04medKVZjDzo9SRhtVI1XE3pbEe Ko8R1RAEiyiJLv0nWxRkxDq8miMEXn00CmWeiqibtcf7dNignQ9KKKgIvq5HMMJbRgfI IR2P6mBP/jUB0+gvq+kpT3nTbnJPzLGU7ynCSpHOOmo9q7Mk1Vq5myr8dICfFJDghTTY GWOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id h10si919289otk.231.2019.05.24.01.13.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 May 2019 01:13:22 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 36F0A3082B5F; Fri, 24 May 2019 08:13:21 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F76F19724; Fri, 24 May 2019 08:13:08 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org, peterx@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com, Andrea Arcangeli Subject: [PATCH net-next 6/6] vhost: access vq metadata through kernel virtual address Date: Fri, 24 May 2019 04:12:18 -0400 Message-Id: <20190524081218.2502-7-jasowang@redhat.com> In-Reply-To: <20190524081218.2502-1-jasowang@redhat.com> References: <20190524081218.2502-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 24 May 2019 08:13:21 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It was noticed that the copy_to/from_user() friends that was used to access virtqueue metdata tends to be very expensive for dataplane implementation like vhost since it involves lots of software checks, speculation barriers, hardware feature toggling (e.g SMAP). The extra cost will be more obvious when transferring small packets since the time spent on metadata accessing become more significant. This patch tries to eliminate those overheads by accessing them through direct mapping of those pages. Invalidation callbacks is implemented for co-operation with general VM management (swap, KSM, THP or NUMA balancing). We will try to get the direct mapping of vq metadata before each round of packet processing if it doesn't exist. If we fail, we will simplely fallback to copy_to/from_user() friends. This invalidation and direct mapping access are synchronized through spinlock and RCU. All matedata accessing through direct map is protected by RCU, and the setup or invalidation are done under spinlock. This method might does not work for high mem page which requires temporary mapping so we just fallback to normal copy_to/from_user() and may not for arch that has virtual tagged cache since extra cache flushing is needed to eliminate the alias. This will result complex logic and bad performance. For those archs, this patch simply go for copy_to/from_user() friends. This is done by ruling out kernel mapping codes through ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE. Note that this is only done when device IOTLB is not enabled. We could use similar method to optimize IOTLB in the future. Tests shows at most about 23% improvement on TX PPS when using virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell: SMAP on | SMAP off Before: 5.2Mpps | 7.1Mpps After: 6.4Mpps | 8.2Mpps Cc: Andrea Arcangeli Cc: James Bottomley Cc: Christoph Hellwig Cc: David Miller Cc: Jerome Glisse Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-parisc@vger.kernel.org Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 515 +++++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 36 +++ 2 files changed, 548 insertions(+), 3 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 8bbda1777c61..fcc2ffd3e12a 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -299,6 +299,160 @@ static void vhost_vq_meta_reset(struct vhost_dev *d) __vhost_vq_meta_reset(d->vqs[i]); } +#if VHOST_ARCH_CAN_ACCEL_UACCESS +static void vhost_map_unprefetch(struct vhost_map *map) +{ + kfree(map->pages); + map->pages = NULL; + map->npages = 0; + map->addr = NULL; +} + +static void vhost_uninit_vq_maps(struct vhost_virtqueue *vq) +{ + struct vhost_map *map[VHOST_NUM_ADDRS]; + int i; + + spin_lock(&vq->mmu_lock); + for (i = 0; i < VHOST_NUM_ADDRS; i++) { + map[i] = rcu_dereference_protected(vq->maps[i], + lockdep_is_held(&vq->mmu_lock)); + if (map[i]) + rcu_assign_pointer(vq->maps[i], NULL); + } + spin_unlock(&vq->mmu_lock); + + synchronize_rcu(); + + for (i = 0; i < VHOST_NUM_ADDRS; i++) + if (map[i]) + vhost_map_unprefetch(map[i]); + +} + +static void vhost_reset_vq_maps(struct vhost_virtqueue *vq) +{ + int i; + + vhost_uninit_vq_maps(vq); + for (i = 0; i < VHOST_NUM_ADDRS; i++) + vq->uaddrs[i].size = 0; +} + +static bool vhost_map_range_overlap(struct vhost_uaddr *uaddr, + unsigned long start, + unsigned long end) +{ + if (unlikely(!uaddr->size)) + return false; + + return !(end < uaddr->uaddr || start > uaddr->uaddr - 1 + uaddr->size); +} + +static void vhost_invalidate_vq_start(struct vhost_virtqueue *vq, + int index, + unsigned long start, + unsigned long end) +{ + struct vhost_uaddr *uaddr = &vq->uaddrs[index]; + struct vhost_map *map; + int i; + + if (!vhost_map_range_overlap(uaddr, start, end)) + return; + + spin_lock(&vq->mmu_lock); + ++vq->invalidate_count; + + map = rcu_dereference_protected(vq->maps[index], + lockdep_is_held(&vq->mmu_lock)); + if (map) { + if (uaddr->write) { + for (i = 0; i < map->npages; i++) + set_page_dirty(map->pages[i]); + } + rcu_assign_pointer(vq->maps[index], NULL); + } + spin_unlock(&vq->mmu_lock); + + if (map) { + synchronize_rcu(); + vhost_map_unprefetch(map); + } +} + +static void vhost_invalidate_vq_end(struct vhost_virtqueue *vq, + int index, + unsigned long start, + unsigned long end) +{ + if (!vhost_map_range_overlap(&vq->uaddrs[index], start, end)) + return; + + spin_lock(&vq->mmu_lock); + --vq->invalidate_count; + spin_unlock(&vq->mmu_lock); +} + +static int vhost_invalidate_range_start(struct mmu_notifier *mn, + const struct mmu_notifier_range *range) +{ + struct vhost_dev *dev = container_of(mn, struct vhost_dev, + mmu_notifier); + int i, j; + + if (!mmu_notifier_range_blockable(range)) + return -EAGAIN; + + for (i = 0; i < dev->nvqs; i++) { + struct vhost_virtqueue *vq = dev->vqs[i]; + + for (j = 0; j < VHOST_NUM_ADDRS; j++) + vhost_invalidate_vq_start(vq, j, + range->start, + range->end); + } + + return 0; +} + +static void vhost_invalidate_range_end(struct mmu_notifier *mn, + const struct mmu_notifier_range *range) +{ + struct vhost_dev *dev = container_of(mn, struct vhost_dev, + mmu_notifier); + int i, j; + + for (i = 0; i < dev->nvqs; i++) { + struct vhost_virtqueue *vq = dev->vqs[i]; + + for (j = 0; j < VHOST_NUM_ADDRS; j++) + vhost_invalidate_vq_end(vq, j, + range->start, + range->end); + } +} + +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { + .invalidate_range_start = vhost_invalidate_range_start, + .invalidate_range_end = vhost_invalidate_range_end, +}; + +static void vhost_init_maps(struct vhost_dev *dev) +{ + struct vhost_virtqueue *vq; + int i, j; + + dev->mmu_notifier.ops = &vhost_mmu_notifier_ops; + + for (i = 0; i < dev->nvqs; ++i) { + vq = dev->vqs[i]; + for (j = 0; j < VHOST_NUM_ADDRS; j++) + RCU_INIT_POINTER(vq->maps[j], NULL); + } +} +#endif + static void vhost_vq_reset(struct vhost_dev *dev, struct vhost_virtqueue *vq) { @@ -327,7 +481,11 @@ static void vhost_vq_reset(struct vhost_dev *dev, vq->busyloop_timeout = 0; vq->umem = NULL; vq->iotlb = NULL; + vq->invalidate_count = 0; __vhost_vq_meta_reset(vq); +#if VHOST_ARCH_CAN_ACCEL_UACCESS + vhost_reset_vq_maps(vq); +#endif } static int vhost_worker(void *data) @@ -459,7 +617,9 @@ void vhost_dev_init(struct vhost_dev *dev, INIT_LIST_HEAD(&dev->read_list); INIT_LIST_HEAD(&dev->pending_list); spin_lock_init(&dev->iotlb_lock); - +#if VHOST_ARCH_CAN_ACCEL_UACCESS + vhost_init_maps(dev); +#endif for (i = 0; i < dev->nvqs; ++i) { vq = dev->vqs[i]; @@ -468,6 +628,7 @@ void vhost_dev_init(struct vhost_dev *dev, vq->heads = NULL; vq->dev = dev; mutex_init(&vq->mutex); + spin_lock_init(&vq->mmu_lock); vhost_vq_reset(dev, vq); if (vq->handle_kick) vhost_poll_init(&vq->poll, vq->handle_kick, @@ -547,7 +708,18 @@ long vhost_dev_set_owner(struct vhost_dev *dev) if (err) goto err_cgroup; +#if VHOST_ARCH_CAN_ACCEL_UACCESS + err = mmu_notifier_register(&dev->mmu_notifier, dev->mm); + if (err) + goto err_mmu_notifier; +#endif + return 0; + +#if VHOST_ARCH_CAN_ACCEL_UACCESS +err_mmu_notifier: + vhost_dev_free_iovecs(dev); +#endif err_cgroup: kthread_stop(worker); dev->worker = NULL; @@ -638,6 +810,107 @@ static void vhost_clear_msg(struct vhost_dev *dev) spin_unlock(&dev->iotlb_lock); } +#if VHOST_ARCH_CAN_ACCEL_UACCESS +static void vhost_setup_uaddr(struct vhost_virtqueue *vq, + int index, unsigned long uaddr, + size_t size, bool write) +{ + struct vhost_uaddr *addr = &vq->uaddrs[index]; + + addr->uaddr = uaddr; + addr->size = size; + addr->write = write; +} + +static void vhost_setup_vq_uaddr(struct vhost_virtqueue *vq) +{ + vhost_setup_uaddr(vq, VHOST_ADDR_DESC, + (unsigned long)vq->desc, + vhost_get_desc_size(vq, vq->num), + false); + vhost_setup_uaddr(vq, VHOST_ADDR_AVAIL, + (unsigned long)vq->avail, + vhost_get_avail_size(vq, vq->num), + false); + vhost_setup_uaddr(vq, VHOST_ADDR_USED, + (unsigned long)vq->used, + vhost_get_used_size(vq, vq->num), + true); +} + +static int vhost_map_prefetch(struct vhost_virtqueue *vq, + int index) +{ + struct vhost_map *map; + struct vhost_uaddr *uaddr = &vq->uaddrs[index]; + struct page **pages; + int npages = DIV_ROUND_UP(uaddr->size, PAGE_SIZE); + int npinned; + void *vaddr, *v; + int err; + int i; + + spin_lock(&vq->mmu_lock); + + err = -EFAULT; + if (vq->invalidate_count) + goto err; + + err = -ENOMEM; + map = kmalloc(sizeof(*map), GFP_ATOMIC); + if (!map) + goto err; + + pages = kmalloc_array(npages, sizeof(struct page *), GFP_ATOMIC); + if (!pages) + goto err_pages; + + err = EFAULT; + npinned = __get_user_pages_fast(uaddr->uaddr, npages, + uaddr->write, pages); + if (npinned > 0) + release_pages(pages, npinned); + if (npinned != npages) + goto err_gup; + + for (i = 0; i < npinned; i++) + if (PageHighMem(pages[i])) + goto err_gup; + + vaddr = v = page_address(pages[0]); + + /* For simplicity, fallback to userspace address if VA is not + * contigious. + */ + for (i = 1; i < npinned; i++) { + v += PAGE_SIZE; + if (v != page_address(pages[i])) + goto err_gup; + } + + map->addr = vaddr + (uaddr->uaddr & (PAGE_SIZE - 1)); + map->npages = npages; + map->pages = pages; + + rcu_assign_pointer(vq->maps[index], map); + /* No need for a synchronize_rcu(). This function should be + * called by dev->worker so we are serialized with all + * readers. + */ + spin_unlock(&vq->mmu_lock); + + return 0; + +err_gup: + kfree(pages); +err_pages: + kfree(map); +err: + spin_unlock(&vq->mmu_lock); + return err; +} +#endif + void vhost_dev_cleanup(struct vhost_dev *dev) { int i; @@ -667,8 +940,16 @@ void vhost_dev_cleanup(struct vhost_dev *dev) kthread_stop(dev->worker); dev->worker = NULL; } - if (dev->mm) + if (dev->mm) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + mmu_notifier_unregister(&dev->mmu_notifier, dev->mm); +#endif mmput(dev->mm); + } +#if VHOST_ARCH_CAN_ACCEL_UACCESS + for (i = 0; i < dev->nvqs; i++) + vhost_uninit_vq_maps(dev->vqs[i]); +#endif dev->mm = NULL; } EXPORT_SYMBOL_GPL(vhost_dev_cleanup); @@ -897,6 +1178,26 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + *((__virtio16 *)&used->ring[vq->num]) = + cpu_to_vhost16(vq, vq->avail_idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), vhost_avail_event(vq)); } @@ -905,6 +1206,27 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, struct vring_used_elem *head, int idx, int count) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_used *used; + size_t size; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + size = count * sizeof(*head); + memcpy(used->ring + idx, head, size); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_copy_to_user(vq, vq->used->ring + idx, head, count * sizeof(*head)); } @@ -912,6 +1234,25 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + used->flags = cpu_to_vhost16(vq, vq->used_flags); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), &vq->used->flags); } @@ -919,6 +1260,25 @@ static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + used->idx = cpu_to_vhost16(vq, vq->last_used_idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), &vq->used->idx); } @@ -964,12 +1324,50 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *idx = avail->idx; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *idx, &vq->avail->idx); } static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, __virtio16 *head, int idx) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *head = avail->ring[idx & (vq->num - 1)]; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *head, &vq->avail->ring[idx & (vq->num - 1)]); } @@ -977,24 +1375,98 @@ static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, __virtio16 *flags) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *flags = avail->flags; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *flags, &vq->avail->flags); } static inline int vhost_get_used_event(struct vhost_virtqueue *vq, __virtio16 *event) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *event = (__virtio16)avail->ring[vq->num]; + rcu_read_unlock(); + return 0; + } + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *event, vhost_used_event(vq)); } static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + *idx = used->idx; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_used(vq, *idx, &vq->used->idx); } static inline int vhost_get_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, int idx) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + struct vhost_map *map; + struct vring_desc *d; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_DESC]); + if (likely(map)) { + d = map->addr; + *desc = *(d + idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); } @@ -1335,12 +1807,32 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, return true; } +#if VHOST_ARCH_CAN_ACCEL_UACCESS +static void vhost_vq_map_prefetch(struct vhost_virtqueue *vq) +{ + struct vhost_map __rcu *map; + int i; + + for (i = 0; i < VHOST_NUM_ADDRS; i++) { + rcu_read_lock(); + map = rcu_dereference(vq->maps[i]); + rcu_read_unlock(); + if (unlikely(!map)) + vhost_map_prefetch(vq, i); + } +} +#endif + int vq_meta_prefetch(struct vhost_virtqueue *vq) { unsigned int num = vq->num; - if (!vq->iotlb) + if (!vq->iotlb) { +#if VHOST_ARCH_CAN_ACCEL_UACCESS + vhost_vq_map_prefetch(vq); +#endif return 1; + } return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && @@ -1551,6 +2043,16 @@ static long vhost_vring_set_num_addr(struct vhost_dev *d, mutex_lock(&vq->mutex); +#if VHOST_ARCH_CAN_ACCEL_UACCESS + /* Unregister MMU notifer to allow invalidation callback + * can access vq->uaddrs[] without holding a lock. + */ + if (d->mm) + mmu_notifier_unregister(&d->mmu_notifier, d->mm); + + vhost_uninit_vq_maps(vq); +#endif + switch (ioctl) { case VHOST_SET_VRING_NUM: r = vhost_vring_set_num(d, vq, argp); @@ -1562,6 +2064,13 @@ static long vhost_vring_set_num_addr(struct vhost_dev *d, BUG(); } +#if VHOST_ARCH_CAN_ACCEL_UACCESS + vhost_setup_vq_uaddr(vq); + + if (d->mm) + mmu_notifier_register(&d->mmu_notifier, d->mm); +#endif + mutex_unlock(&vq->mutex); return r; diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 7a7fc001265f..4a24ffee0bdb 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -12,6 +12,9 @@ #include #include #include +#include +#include +#include struct vhost_work; typedef void (*vhost_work_fn_t)(struct vhost_work *work); @@ -80,6 +83,21 @@ enum vhost_uaddr_type { VHOST_NUM_ADDRS = 3, }; +struct vhost_map { + int npages; + void *addr; + struct page **pages; +}; + +struct vhost_uaddr { + unsigned long uaddr; + size_t size; + bool write; +}; + +#define VHOST_ARCH_CAN_ACCEL_UACCESS defined(CONFIG_MMU_NOTIFIER) && \ + ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + /* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue { struct vhost_dev *dev; @@ -90,7 +108,22 @@ struct vhost_virtqueue { struct vring_desc __user *desc; struct vring_avail __user *avail; struct vring_used __user *used; + +#if VHOST_ARCH_CAN_ACCEL_UACCESS + /* Read by memory accessors, modified by meta data + * prefetching, MMU notifier and vring ioctl(). + * Synchonrized through mmu_lock (writers) and RCU (writers + * and readers). + */ + struct vhost_map __rcu *maps[VHOST_NUM_ADDRS]; + /* Read by MMU notifier, modified by vring ioctl(), + * synchronized through MMU notifier + * registering/unregistering. + */ + struct vhost_uaddr uaddrs[VHOST_NUM_ADDRS]; +#endif const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS]; + struct file *kick; struct eventfd_ctx *call_ctx; struct eventfd_ctx *error_ctx; @@ -145,6 +178,8 @@ struct vhost_virtqueue { bool user_be; #endif u32 busyloop_timeout; + spinlock_t mmu_lock; + int invalidate_count; }; struct vhost_msg_node { @@ -158,6 +193,7 @@ struct vhost_msg_node { struct vhost_dev { struct mm_struct *mm; + struct mmu_notifier mmu_notifier; struct mutex mutex; struct vhost_virtqueue **vqs; int nvqs;