From patchwork Wed May 11 03:54:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845779 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EF39C433EF for ; Wed, 11 May 2022 03:55:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239329AbiEKDzd (ORCPT ); Tue, 10 May 2022 23:55:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238010AbiEKDzb (ORCPT ); Tue, 10 May 2022 23:55:31 -0400 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E0C12108A5 for ; Tue, 10 May 2022 20:55:29 -0700 (PDT) Received: by mail-pf1-x434.google.com with SMTP id v11so891343pff.6 for ; Tue, 10 May 2022 20:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2p2fPjfsIsJ6E9/ORkfpw1BwG9ijb5KY/aOnRzBglX8=; b=pUfw03v2yYPpGXgiiO1AI1VBqjOOMzwAHj2d2Yks2pr5NwxeXTDwKx+aDG4hnBLUPJ DyRLE8I/8b4bz9D8mjyxi6HdnDigwv22Gs6PYIxPKG7cAtPUMJKXw++F4iyGPgCS7oB/ ySiOS8IQFGWr+Cax/kEOnhxLlG7w4r3dNHAB0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2p2fPjfsIsJ6E9/ORkfpw1BwG9ijb5KY/aOnRzBglX8=; b=v1qBBF2aHa+piTihMY+l7JOZt3MutDdtC4z1jZfsYG4RLNWfKLWQsJNgcudTzm15k2 LIISYAMjgNc6ANo2/sxdOFj+J0nK1sTwIivL4IS9TyDQAnOK5sfUDxLi79NidxaqYE0h 9GRX8oMLumIqXJJKJf2PjPEJVvLbiECJy5jNt86R2akZqVIsjq9BYGq9wNpPNC+qyUZ+ +8qvqTZKn0E0xhe9qGdYao3yUivhunvz3pY3eHzRYJua0//kKLJJiZLnQGb9qaGt+aYP mqGurvMiT45AQYj7jmCO4F/Vs+hCWHF1+LTAiaGwedcm5FuVUaNhagYw8iW3KsbVVlH2 SkYw== X-Gm-Message-State: AOAM533PD2bGdAXj/YwL7t9q05llYv2Lyg9DPMAglwLpYUpv33WLba3u hLKvIpxt2qCiFTSw1Lwr7v6wN831rpNejm2TvTBMlQZkuzHTJ9blsNdpjkfMc5Ka5Bm9/ztZiLX jmBLJcPitQvUBHZJPmUdT/9D3+K/yM9FPcDEpDYW7aidvnvhHeMbtC9mh3evv/M2IOxnK X-Google-Smtp-Source: ABdhPJzc/72l8fO9n7RvaaUC+1kyZHe2Tt9jHZADFhzTnhiYXdy4MIHTlfNLC8Izcw43sJcjBnCDXA== X-Received: by 2002:a63:e16:0:b0:3c6:12b1:ce15 with SMTP id d22-20020a630e16000000b003c612b1ce15mr19238632pgl.37.1652241328326; Tue, 10 May 2022 20:55:28 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:27 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Joe Damato Subject: [RFC,net-next,x86 1/6] arch, x86, uaccess: Add nontemporal copy functions Date: Tue, 10 May 2022 20:54:22 -0700 Message-Id: <1652241268-46732-2-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC Add a generic non-temporal wrapper to uaccess which can be overridden by arches that support non-temporal copies. An implementation is added for x86 which wraps an existing non-temporal copy in the kernel. Signed-off-by: Joe Damato --- arch/x86/include/asm/uaccess_64.h | 6 ++++++ include/linux/uaccess.h | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h index 45697e0..ed41dba 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -65,6 +65,12 @@ extern long __copy_user_flushcache(void *dst, const void __user *src, unsigned s extern void memcpy_page_flushcache(char *to, struct page *page, size_t offset, size_t len); +static inline unsigned long +__copy_from_user_nocache(void *dst, const void __user *src, unsigned long size) +{ + return (unsigned long)__copy_user_nocache(dst, src, (unsigned int) size, 0); +} + static inline int __copy_from_user_inatomic_nocache(void *dst, const void __user *src, unsigned size) diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 5461794..d1f57a1 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -234,6 +234,12 @@ static inline bool pagefault_disabled(void) #ifndef ARCH_HAS_NOCACHE_UACCESS static inline __must_check unsigned long +__copy_from_user_nocache(void *to, const void __user *from, unsigned long n) +{ + return __copy_from_user(to, from, n); +} + +static inline __must_check unsigned long __copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n) { From patchwork Wed May 11 03:54:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845780 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04D00C433F5 for ; Wed, 11 May 2022 03:55:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241862AbiEKDzj (ORCPT ); Tue, 10 May 2022 23:55:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239334AbiEKDzd (ORCPT ); Tue, 10 May 2022 23:55:33 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 724D4210B86 for ; Tue, 10 May 2022 20:55:31 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id l20-20020a17090a409400b001dd2a9d555bso966594pjg.0 for ; Tue, 10 May 2022 20:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OGmy4QLI/8fGmA3FerkRtku5TKK3UOkAtDIIlXf/W7s=; b=Lvz2NFU0wxJq7i37Ze4DZNA+uM0xodpQ2/GD0sncaN3Rc26o53fkHgOXTmfUTOGYMe PQ8YhJRQE2joOk55gZTNRlr0yhTfNgbEswFespAwO54TVWi3uXnQh34OkhOecNRke75l vkqlDGGRVgUjE7Hj88oeaMjrx58TpDNV9ffOA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OGmy4QLI/8fGmA3FerkRtku5TKK3UOkAtDIIlXf/W7s=; b=O15Q5jPAYVufsM+WJ6zkZAipHXlmHw4WWO0wKAE6fczEbLBLUm1FfVBSksSLggclEE ax8VZkz12kISDwZMxiDyPlZElbkbYniXYVWoIWVO/RX/fPrNF0W+5P+JBWGpq106iI+a M5SLgxqGWll1pdcbrhyhi16mmSzs7ByMM0QAlCqMjWnGS20kj5LmU+f/kw8AcqUaI71i VAui/SoY+5j7DVROPZ6hTuLUqHftHY+onE+ktI/vTKP8ROWdxsYTpqFJ/4v26tKa26X4 e5SmKLM0Uv2rsKiEcukl3VC1ilY4PBpjcNvZc+RlPyFFtYCeY3FCl+w840QJMAXIdfIZ uXtw== X-Gm-Message-State: AOAM532RRdsDPogPALCDDgB6mPHY3kzTY8rMBed3eVLlAeQIWbAaOFlr Uvl/Erz0zCnkddN13irthI8mNVY+Ol0ktALoLcrZ2Do9YWPSzVMe4jJdqdYXUnCi4YVd02/+/IU 9R7w4xbJyheXD6WN9T+aLjz+E9QT8y+zbqGax+uTV8ac1wxHA4XMXMzlathreIRXfSg85 X-Google-Smtp-Source: ABdhPJzw3DCldBbT9UirZQI+OHb9LG4rGmq8XkuDK/k9AREZfDPPdxjaEuFH1i4LKnAoxp/vAOJzyw== X-Received: by 2002:a17:90b:14ce:b0:1dc:eff5:52b6 with SMTP id jz14-20020a17090b14ce00b001dceff552b6mr3169500pjb.148.1652241330638; Tue, 10 May 2022 20:55:30 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:30 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Alexander Viro Cc: Joe Damato Subject: [RFC,net-next 2/6] iov_iter: Allow custom copyin function Date: Tue, 10 May 2022 20:54:23 -0700 Message-Id: <1652241268-46732-3-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC When calling copy_page_from_iter_iovec, allow callers to specify the copy function they'd like to use. The only caller is updated to pass raw_copy_from_user. Signed-off-by: Joe Damato --- lib/iov_iter.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330..ef22ec1 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -253,7 +253,9 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b } static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i) + struct iov_iter *i, + unsigned long (*_copyin)(void *to, const void __user *from, + unsigned long n)) { size_t skip, copy, left, wanted; const struct iovec *iov; @@ -278,7 +280,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t to = kaddr + offset; /* first chunk, usually the only one */ - left = copyin(to, buf, copy); + left = _copyin(to, buf, copy); copy -= left; skip += copy; to += copy; @@ -288,7 +290,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t iov++; buf = iov->iov_base; copy = min(bytes, iov->iov_len); - left = copyin(to, buf, copy); + left = _copyin(to, buf, copy); copy -= left; skip = copy; to += copy; @@ -307,7 +309,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t kaddr = kmap(page); to = kaddr + offset; - left = copyin(to, buf, copy); + left = _copyin(to, buf, copy); copy -= left; skip += copy; to += copy; @@ -316,7 +318,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t iov++; buf = iov->iov_base; copy = min(bytes, iov->iov_len); - left = copyin(to, buf, copy); + left = _copyin(to, buf, copy); copy -= left; skip = copy; to += copy; @@ -899,7 +901,7 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, if (unlikely(!page_copy_sane(page, offset, bytes))) return 0; if (likely(iter_is_iovec(i))) - return copy_page_from_iter_iovec(page, offset, bytes, i); + return copy_page_from_iter_iovec(page, offset, bytes, i, raw_copy_from_user); if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { void *kaddr = kmap_local_page(page); size_t wanted = _copy_from_iter(kaddr + offset, bytes, i); From patchwork Wed May 11 03:54:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845781 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 791B5C433FE for ; Wed, 11 May 2022 03:55:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241868AbiEKDzm (ORCPT ); Tue, 10 May 2022 23:55:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240025AbiEKDze (ORCPT ); Tue, 10 May 2022 23:55:34 -0400 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 691DD213326 for ; Tue, 10 May 2022 20:55:33 -0700 (PDT) Received: by mail-pg1-x529.google.com with SMTP id a191so710924pge.2 for ; Tue, 10 May 2022 20:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tkAZPhAkhghXKAcrYQ097N+yUjSoD43IH6MO8WVhkW0=; b=XTIXKAsQLSW9h7YC+Hl0jhrCRlhJoL7ZN43rl4HojAKB6QTZDyKFGdsQ3+/4q11Gf9 1yrehn7dms/+YB3orW/WWboVn+fzhaRuXuXvjIXG6v7vC15AgKmlBlD8gilh49cnbXYJ dkZW2WjQK5eGZGAmm374sgwDgQoibYPvNPMlg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tkAZPhAkhghXKAcrYQ097N+yUjSoD43IH6MO8WVhkW0=; b=tMkJUjdqQYh0sT+ucofPJl8BMm/cPq++VxFayaz1nt2Ba2L+ERqNk8brIDuXLdaDKl OCDIx7QsXP78jCRJparZ07bbCKGHxX20mS3hSThdJqHBnoyYXtu4pytgA9afN8JUSCQ7 QGZrPl3OYx5oQmFvgFMmnGChVG6zAteT5+24ZozP9avYm8LKUfOsL28DQzCooDJLYQfR He5foNosAAYeHKejBlnqxtplI5GAFpIKSy2jE7oGRHfoITyAADxgzWrqAIsE72Y4iE2i eoQ35bTniCMPlrU5Z5eV0kXzK0LqpXIK88Zokqp4XlSlBSzWV/JJ/YK0GrjgtyTutUXw Lacg== X-Gm-Message-State: AOAM530rZazrxLejCZlddUBgyP7f9I3Lgb5623xeZ0jM76gpTYQO1XCm JtPMV8Ij6RIloOT4Ulx7sHBG/rPGNgLtEzbP9aVUBpweANvSwWVlqC6LouZlpNiE+7kTLpZ2ene WE37GqYWhz2z96XzfMUxXtbnPHnOt1+GE9D/vShGxldEcBQ6UmvYaSNvn3ZJ41ZKnEOOC X-Google-Smtp-Source: ABdhPJxYgsPGtWhs7POp5A5xnYfBle+QWXILF7HGaYr8jorK2248KV18yTDhWf6EaGfBzvQ3RnPdig== X-Received: by 2002:a05:6a00:10cc:b0:505:ada6:e03e with SMTP id d12-20020a056a0010cc00b00505ada6e03emr23348965pfu.45.1652241332539; Tue, 10 May 2022 20:55:32 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:32 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Alexander Viro Cc: Joe Damato Subject: [RFC,net-next 3/6] iov_iter: Add a nocache copy iov iterator Date: Tue, 10 May 2022 20:54:24 -0700 Message-Id: <1652241268-46732-4-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC Add copy_page_from_iter_nocache, which wraps copy_page_from_iter_iovec and passes in a custom copyin function: __copy_from_user_nocache. This allows callers of copy_page_from_iter_nocache to copy data without disturbing the CPU cache. Signed-off-by: Joe Damato --- include/linux/uio.h | 2 ++ lib/iov_iter.c | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 739285f..58c7946 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -142,6 +142,8 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); +size_t copy_page_from_iter_nocache(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i); size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index ef22ec1..985bf58 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -895,6 +895,26 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, } EXPORT_SYMBOL(copy_page_to_iter); +size_t copy_page_from_iter_nocache(struct page *page, size_t offset, size_t + bytes, struct iov_iter *i) +{ + if (unlikely(!page_copy_sane(page, offset, bytes))) + return 0; + if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { + WARN_ON(1); + return 0; + } + if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { + void *kaddr = kmap_atomic(page); + size_t wanted = _copy_from_iter_nocache(kaddr + offset, bytes, i); + + kunmap_atomic(kaddr); + return wanted; + } else + return copy_page_from_iter_iovec(page, offset, bytes, i, + __copy_from_user_nocache); +} + size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { From patchwork Wed May 11 03:54:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845785 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D67B8C4332F for ; Wed, 11 May 2022 03:56:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241903AbiEKDzy (ORCPT ); Tue, 10 May 2022 23:55:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241197AbiEKDzi (ORCPT ); Tue, 10 May 2022 23:55:38 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9343521719C for ; Tue, 10 May 2022 20:55:35 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id e24so1061625pjt.2 for ; Tue, 10 May 2022 20:55:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aZBrMhde8FpEVWkUnj31KTacbZRR8gMfK1YB4YXQ0jc=; b=h83XAA98G4q5ePsZrIsLvPfnCfnbTjI1gSU8AzJSpPDJntT477r7QuDZCMsCDJKWG9 TqetClC0MszlVLsO1ojgFUTsLAAGc1OgU2jb+FQKxhxP0S81l5KGJ6eWfk7b45R21mEG Eq2ntBM7BWiZhqRxdb4HIcuGcH1vbYaRo55F4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aZBrMhde8FpEVWkUnj31KTacbZRR8gMfK1YB4YXQ0jc=; b=1Ho6oKirBAJyeb1PbGakwR4zHRLdgJdt1RXqSS7ybs4YTQIyOCGEhtx4pghMLrI/qE lCYf7DkkkajFHvAvkBSCoAfIit++oAZ08e6AzGWABftnExZMqr/p1Jz3NOeFBg8sPwOg HJmtE13Mujs4Miree5jtUbiirgO/SQfummc6JIrU8GM6oeYgS7OlWscw60ZTZFAV443c EF7B64JOAFot4OPGxaHOtu/Y5exgbeBz2wafBHu4KfqVuhGElXF4kqERsydzhF914aW7 8AL5aA7fnJawBaquCOmKO/GW7QX2xo/szIPKZFG8UvaHiJAG3WpZZKC4lgrRyH9WJ5ao xETw== X-Gm-Message-State: AOAM531JGfhm9Fx615j4ihtoArgTFa5A/LDc45e58cab2xwnMnqcLH7k A788B9P4ep92UgqibEbllFrMkPHFbtSBR9JHiDAfyIRhgYFvVO/6GBvnGGYMwhaHVPL6N3dHVfb A3Xfxx7LK7rJcfO3/C5UZqaAZ0LBeHuVCgODovPXHHFIaDTG9kv7mPtTMFUrXEYvK8j4V X-Google-Smtp-Source: ABdhPJxhMetGgdMphFhnMhts3MbZSr4+18d+mQckIRBqUPMXsNdPRazRpLHEHvEAaWw9V+TQ0BG9Vg== X-Received: by 2002:a17:902:ecca:b0:15e:8971:4540 with SMTP id a10-20020a170902ecca00b0015e89714540mr23357838plh.43.1652241334487; Tue, 10 May 2022 20:55:34 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:33 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 4/6] net: Add a struct for managing copy functions Date: Tue, 10 May 2022 20:54:25 -0700 Message-Id: <1652241268-46732-5-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC Add struct skb_copier which encapsulates two functions for copying data, and provide a default copier, skb_copier. Separate skb_copy_datagram_from_iter into a a helper function, do_skb_copy_datagram, which takes a struct skb_copier. Signed-off-by: Joe Damato --- net/core/datagram.c | 49 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 17 deletions(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index 50f4fae..a87c41b 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -532,18 +532,19 @@ int skb_copy_datagram_iter(const struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_datagram_iter); -/** - * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. - * @skb: buffer to copy - * @offset: offset in the buffer to start copying to - * @from: the copy source - * @len: amount of data to copy to buffer from iovec - * - * Returns 0 or -EFAULT. - */ -int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, - struct iov_iter *from, - int len) +struct skb_copier { + size_t (*copy_from_iter)(void *addr, size_t bytes, struct iov_iter *i); + size_t (*copy_page_from_iter)(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i); +}; + +struct skb_copier skb_copier = { + .copy_from_iter = copy_from_iter, + .copy_page_from_iter = copy_page_from_iter +}; + +static int do_skb_copy_datagram(struct sk_buff *skb, int offset, + struct iov_iter *from, int len, struct skb_copier copier) { int start = skb_headlen(skb); int i, copy = start - offset; @@ -553,7 +554,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, if (copy > 0) { if (copy > len) copy = len; - if (copy_from_iter(skb->data + offset, copy, from) != copy) + if (copier.copy_from_iter(skb->data + offset, copy, from) != copy) goto fault; if ((len -= copy) == 0) return 0; @@ -573,7 +574,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, if (copy > len) copy = len; - copied = copy_page_from_iter(skb_frag_page(frag), + copied = copier.copy_page_from_iter(skb_frag_page(frag), skb_frag_off(frag) + offset - start, copy, from); if (copied != copy) @@ -595,9 +596,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, if ((copy = end - offset) > 0) { if (copy > len) copy = len; - if (skb_copy_datagram_from_iter(frag_iter, - offset - start, - from, copy)) + if (do_skb_copy_datagram(frag_iter, offset - start, from, copy, copier)) goto fault; if ((len -= copy) == 0) return 0; @@ -611,6 +610,22 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, fault: return -EFAULT; } + +/** + * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. + * @skb: buffer to copy + * @offset: offset in the buffer to start copying to + * @from: the copy source + * @len: amount of data to copy to buffer from iovec + * + * Returns 0 or -EFAULT. + */ +int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, + struct iov_iter *from, + int len) +{ + return do_skb_copy_datagram(skb, offset, from, len, skb_copier); +} EXPORT_SYMBOL(skb_copy_datagram_from_iter); int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, From patchwork Wed May 11 03:54:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845782 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AE72C433F5 for ; Wed, 11 May 2022 03:55:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241877AbiEKDzt (ORCPT ); Tue, 10 May 2022 23:55:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241266AbiEKDzj (ORCPT ); Tue, 10 May 2022 23:55:39 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9818F218FFE for ; Tue, 10 May 2022 20:55:37 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id t11-20020a17090ad50b00b001d95bf21996so3746410pju.2 for ; Tue, 10 May 2022 20:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HVRP118ZOlUZuoG/hDkho2/EllajCBtZWJSv42QBoCc=; b=GEO29e6cFLU3C6rQeRvcHHx1B7Gl6areArEPEnkD1l5qphc906h2sIwiQ9NEQT3nxy hnqzbFtx/vuGJLQlvdprTdtvWNTrHhwoY3C4BKHov0yQYeFQb7vB7W02Zx/MJLmHLzIm 0waUvGXOJQ5G3GNz0yafi51anAAeRtIfzso7A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HVRP118ZOlUZuoG/hDkho2/EllajCBtZWJSv42QBoCc=; b=NKMq72hAsdqKy6J1LD1fYVshJZP1pGB61sUem9/X0Ts+VnaWK5/zSbg+l+Hkx+yB9p 9MHfQEHpXh1CqrHCvCMQ71GJs9f6DXcP20Q7bnuZxGLhTjfG181Hgtsr6/pT51wzkNAM V466yu9eHVdJNMDHsGvwTHrqQ8DFmCJHH+vaB2mnOSoootCpBZjMiVRIknEWj+ZRlLou BW4LlCq247ffCN4IY/b8ssoIOhObcJE5Xy2Hr/Bk1rNHLZgN+LbPYI+/LCGwbj8tdARt Hp939jmFAbUEcI307PAk72iYN4bzT/GYaj4/RhQq35HSU5uH+FLCIgOzVseEzV5trBca Ozjg== X-Gm-Message-State: AOAM5338Eu/EI6MJhu5D2SngGUhreyzRHral+hWICQQ8h4Ysqfp/BFMu fg692t1xf0vFPhT6ZF0MtzGdeh8FAaxQMcGK0WJeYenjylqyA86mTotnVXpqZdF//7rstatr4Lg tLyxWfM0S0bPLhRit6xUzKaPodgm69GUFRUJVSnz2L0b1c/T7VkAulH7l0wOx320sHzCA X-Google-Smtp-Source: ABdhPJx2cBOX8TKr/h/WTeqUC4FZm457izcVFqtGtuZ/tZr6Sm6aO2hN978l1aOE3sJBvl184EerOw== X-Received: by 2002:a17:902:8644:b0:15a:3b4a:538a with SMTP id y4-20020a170902864400b0015a3b4a538amr23511594plt.146.1652241336651; Tue, 10 May 2022 20:55:36 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:35 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 5/6] net: Add a way to copy skbs without affect cache Date: Tue, 10 May 2022 20:54:26 -0700 Message-Id: <1652241268-46732-6-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC Add an skb_copier, skb_nocache_copier, which contains function pointers to nontemporal copy routines. Using skb_nocache_copier and do_skb_copy_datagram implement skb_copy_datagram_from_iter_nocache. This function is intended to be used by callers which would like to copy data into SKBs using nontemporal instructions to avoid the CPU cache. Signed-off-by: Joe Damato --- include/linux/skbuff.h | 2 ++ net/core/datagram.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 97de40b..32c0cba 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3918,6 +3918,8 @@ int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, struct ahash_request *hash); int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, struct iov_iter *from, int len); +int skb_copy_datagram_from_iter_nocache(struct sk_buff *skb, int offset, + struct iov_iter *from, int len); int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm); void skb_free_datagram(struct sock *sk, struct sk_buff *skb); void __skb_free_datagram_locked(struct sock *sk, struct sk_buff *skb, int len); diff --git a/net/core/datagram.c b/net/core/datagram.c index a87c41b..da8557b 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -543,6 +543,11 @@ struct skb_copier skb_copier = { .copy_page_from_iter = copy_page_from_iter }; +struct skb_copier skb_nocache_copier = { + .copy_from_iter = copy_from_iter_nocache, + .copy_page_from_iter = copy_page_from_iter_nocache +}; + static int do_skb_copy_datagram(struct sk_buff *skb, int offset, struct iov_iter *from, int len, struct skb_copier copier) { @@ -611,6 +616,13 @@ static int do_skb_copy_datagram(struct sk_buff *skb, int offset, return -EFAULT; } +int skb_copy_datagram_from_iter_nocache(struct sk_buff *skb, int offset, + struct iov_iter *from, int len) +{ + return do_skb_copy_datagram(skb, offset, from, len, skb_nocache_copier); +} +EXPORT_SYMBOL(skb_copy_datagram_from_iter_nocache); + /** * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. * @skb: buffer to copy From patchwork Wed May 11 03:54:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12845784 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3BFCC433F5 for ; Wed, 11 May 2022 03:56:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241894AbiEKDzv (ORCPT ); Tue, 10 May 2022 23:55:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241887AbiEKDzq (ORCPT ); Tue, 10 May 2022 23:55:46 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75EFD116 for ; Tue, 10 May 2022 20:55:39 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id n18so675216plg.5 for ; Tue, 10 May 2022 20:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8Oj5fYgOyP9r6j4Oo2YFgfJ89J2j62xWeof0Jo9AY3w=; b=v9w+8OSo8NSVmDmBQbc8obXr4Wi1nhD7MIDl5mmowMNt72/kNMxqluqzW8jS3Kp7UB /7VISt6MIJKtVD9o0A7m6LzNO5Lqp9ow+x5B9kBGB7t7PtEZGsvXfcXksfmbNr4z51g2 LTzKHBx8TvH8VwSw5qQxFSvqLW2aYGDGcyTCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8Oj5fYgOyP9r6j4Oo2YFgfJ89J2j62xWeof0Jo9AY3w=; b=1+kBaCJRc8nAW/DCaYP2rGe/CGKjXDzF3heg2UpOB0ZuaqGDwJWLzionC0FnlfmUyo bBu4g3f9Ycfx7rlw8FTCRhvemO3riIyoTudiH9aiopziyMLs7fLZ+bNucCrmT7nUZCsi xypT3BlvWru+ehk5xoCNikEcbQTrXphfcso75JACNfbm4S/F4ePhgGuUOgcHdbDQ1fll PiwY4XskHmsp56J9gt1nBoHSD73wtjVcdJh2Nf5kArcjnEn3XGxAcv1s32r7u3P6+DC9 MnZGZz/M9NsCNS8Q1UPiI55zjh1YCCvm0xT9Ng4uoOjh5dKvTNTY6CqqHJJBvu5O7Cfe +Etg== X-Gm-Message-State: AOAM5323CPoA90WQ78iX/5vq4Eb7RLjyJsJeZZfX9AP5YkZ7+PqjBdIG V4y3wPbEa8cZ6fzpvbYS1Iaudmkw3euZSKsLlaB0X4gghYlvmj4nk8yiasZPYYWds6xv7bWVr4S CPn7+bbswyZaF9usHc47ZOSSvt2Wd2ofHCx/B741cqW9f06A0wtrD94au5NNmc0OEcW6W X-Google-Smtp-Source: ABdhPJzDBQpUZk7W8Mq0NWHrteV9nx9UH/BVPj5fyK4gC7QCMNmi9ReuRK03KcV0ElJJJiZ8qW3FTA== X-Received: by 2002:a17:902:8644:b0:153:9f01:2090 with SMTP id y4-20020a170902864400b001539f012090mr22917397plt.101.1652241338601; Tue, 10 May 2022 20:55:38 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:38 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 6/6] net: unix: Add MSG_NTCOPY Date: Tue, 10 May 2022 20:54:27 -0700 Message-Id: <1652241268-46732-7-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC Add a new sendmsg flag, MSG_NTCOPY, which user programs can use to signal to the kernel that data copied into the kernel during sendmsg should be done so using nontemporal copies, if it is supported by the architecture. Signed-off-by: Joe Damato --- include/linux/socket.h | 1 + net/unix/af_unix.c | 13 +++++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index 12085c9..c9b10aa 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -318,6 +318,7 @@ struct ucred { * plain text and require encryption */ +#define MSG_NTCOPY 0x2000000 /* Use a non-temporal copy */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index e1dd9e9..ccbd643 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1907,7 +1907,11 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg, skb_put(skb, len - data_len); skb->data_len = data_len; skb->len = len; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, len); + if (msg->msg_flags & MSG_NTCOPY) + err = skb_copy_datagram_from_iter_nocache(skb, 0, &msg->msg_iter, len); + else + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, len); + if (err) goto out_free; @@ -2167,7 +2171,12 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, skb_put(skb, size - data_len); skb->data_len = data_len; skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + + if (msg->msg_flags & MSG_NTCOPY) + err = skb_copy_datagram_from_iter_nocache(skb, 0, &msg->msg_iter, size); + else + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { kfree_skb(skb); goto out_err;