From patchwork Fri Jun 25 01:01:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Croce X-Patchwork-Id: 12343411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5271CC49EA6 for ; Fri, 25 Jun 2021 01:02:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA0F56135A for ; Fri, 25 Jun 2021 01:02:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA0F56135A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=15OWW4ykbnkjdlJjtsXfgDCOFC28dGWtVN3Cv+AXDco=; b=cMINN/Wb8vRTrP 0PZ8o2AJR1zoy0utFEBp8DgOz1f+6q46b91yFt+9oXK7RfTYsZrMzTWva61ftzyZ3mXZnb1BQynuO BChbJbBbd830Gpf6AOH83dx9DZQcwX1JboLOkyAOZyCK0nR70S5IRrkHqZYC37bpAK3fCfONSGwvP FOpeJXpbr7TW8vyRDcYVckjJDQ4Vsi/0ca7aCCXMDmc5PxL6h/XRB+ltUqxvgoHSi3Cboi35pB4Je 2+TRsKnjtiQIj/Kbr0p35rbVLX2vF5pXagJIjsMMYzOYQhhZ6LLBaznKkvxi3VH3kzJ71QHo1dW7z JkTK8PUzTyJI9WAOLvSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEp-00GvrA-LM; Fri, 25 Jun 2021 01:02:31 +0000 Received: from mail-ed1-f52.google.com ([209.85.208.52]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEn-00GvpM-6V for linux-riscv@lists.infradead.org; Fri, 25 Jun 2021 01:02:30 +0000 Received: by mail-ed1-f52.google.com with SMTP id h17so11141314edw.11 for ; Thu, 24 Jun 2021 18:02:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZzhxbsS0xblHmbaMS57RZ4S8yZ1GoCabsZyCXQhJYrY=; b=iu+FIq/U8++yCVAQz3s9zkB8B4n1VuLARR0TW8zjHrSSC5rQpKFUvOxmKbKQrN6wSK 2xftz5h3UIosyyzThHT88FP8BFMBpCsbIyuEpvJ7H97oriuer80QUnDeich3GwDOFTOM cpXV2W8mwcmUZJ1TIBBAONEJbDFOjvwqjYgdP7fjQg/HQIGy59XYPy7970NoAIi5W+bs vlyJFmvB3eK10fDMlRZny2Dnyf8nCx1ZaVfj2G+Nr3n/gIOZ/Nn9+h7cxf7x/dcD8v1T bk9em4hChDL505WD/gHcG6FOKERBxRAB3G9T5z9Bx0a0TPz00633GBuwSgbwIjTTa86t jdqw== X-Gm-Message-State: AOAM532RAo1ddIyZN1s+c4KWHd/XkhBYfG5dvUd0PNlMPtG2h+OXCFBv N52I2WKjS5g0PCySmjDqh0k= X-Google-Smtp-Source: ABdhPJzUlanwgvJ9sy/0XoJxNoRpkQdc4Ar2JfL2/NSyHYMU4NhXrSCqoGE3LIUdsSRwDZUL/8ERpA== X-Received: by 2002:a05:6402:35c3:: with SMTP id z3mr10939073edc.346.1624582947742; Thu, 24 Jun 2021 18:02:27 -0700 (PDT) Received: from msft-t490s.home (host-95-251-17-240.retail.telecomitalia.it. [95.251.17.240]) by smtp.gmail.com with ESMTPSA id yc29sm1921909ejb.106.2021.06.24.18.02.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 18:02:26 -0700 (PDT) From: Matteo Croce To: linux-kernel@vger.kernel.org, Nick Kossifidis , Guo Ren , Christoph Hellwig , David Laight , Palmer Dabbelt , Emil Renner Berthing , Drew Fustini Cc: linux-arch@vger.kernel.org, Andrew Morton , Nick Desaulniers , linux-riscv@lists.infradead.org Subject: [PATCH 1/3] lib/string: optimized memcpy Date: Fri, 25 Jun 2021 03:01:58 +0200 Message-Id: <20210625010200.362755-2-mcroce@linux.microsoft.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210625010200.362755-1-mcroce@linux.microsoft.com> References: <20210625010200.362755-1-mcroce@linux.microsoft.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210624_180229_290321_86FDC0DF X-CRM114-Status: GOOD ( 18.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Matteo Croce Rewrite the generic memcpy() to copy a word at time, without generating unaligned accesses. The procedure is made of three steps: First copy data one byte at time until the destination buffer is aligned to a long boundary. Then copy the data one long at time shifting the current and the next long to compose a long at every cycle. Finally, copy the remainder one byte at time. Signed-off-by: Matteo Croce --- lib/string.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 78 insertions(+), 3 deletions(-) diff --git a/lib/string.c b/lib/string.c index 546d59711a12..15e906f97d9e 100644 --- a/lib/string.c +++ b/lib/string.c @@ -33,6 +33,24 @@ #include #include +#define MIN_THRESHOLD (sizeof(long) * 2) + +/* convenience union to avoid cast between different pointer types */ +union types { + u8 *as_u8; + unsigned long *as_ulong; + uintptr_t as_uptr; +}; + +union const_types { + const u8 *as_u8; + const unsigned long *as_ulong; + uintptr_t as_uptr; +}; + +static const unsigned int bytes_long = sizeof(long); +static const unsigned int word_mask = bytes_long - 1; + #ifndef __HAVE_ARCH_STRNCASECMP /** * strncasecmp - Case insensitive, length-limited string comparison @@ -878,16 +896,73 @@ EXPORT_SYMBOL(memset64); * You should not use this function to access IO space, use memcpy_toio() * or memcpy_fromio() instead. */ + +#ifdef __BIG_ENDIAN +#define MERGE_UL(h, l, d) ((h) << ((d) * 8) | (l) >> ((bytes_long - (d)) * 8)) +#else +#define MERGE_UL(h, l, d) ((h) >> ((d) * 8) | (l) << ((bytes_long - (d)) * 8)) +#endif + void *memcpy(void *dest, const void *src, size_t count) { - char *tmp = dest; - const char *s = src; + union const_types s = { .as_u8 = src }; + union types d = { .as_u8 = dest }; + int distance = 0; + + if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (count < MIN_THRESHOLD) + goto copy_remainder; + + /* Copy a byte at time until destination is aligned. */ + for (; d.as_uptr & word_mask; count--) + *d.as_u8++ = *s.as_u8++; + + distance = s.as_uptr & word_mask; + } + if (distance) { + unsigned long last, next; + + /* + * s is distance bytes ahead of d, and d just reached + * the alignment boundary. Move s backward to word align it + * and shift data to compensate for distance, in order to do + * word-by-word copy. + */ + s.as_u8 -= distance; + + next = s.as_ulong[0]; + for (; count >= bytes_long + word_mask; count -= bytes_long) { + last = next; + next = s.as_ulong[1]; + + d.as_ulong[0] = MERGE_UL(last, next, distance); + + d.as_ulong++; + s.as_ulong++; + } + + /* Restore s with the original offset. */ + s.as_u8 += distance; + } else { + /* + * If the source and dest lower bits are the same, do a simple + * 32/64 bit wide copy. + */ + for (; count >= bytes_long; count -= bytes_long) + *d.as_ulong++ = *s.as_ulong++; + } + +copy_remainder: while (count--) - *tmp++ = *s++; + *d.as_u8++ = *s.as_u8++; + return dest; } EXPORT_SYMBOL(memcpy); + +#undef MERGE_UL + #endif #ifndef __HAVE_ARCH_MEMMOVE From patchwork Fri Jun 25 01:01:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Croce X-Patchwork-Id: 12343415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEE7CC49EA7 for ; Fri, 25 Jun 2021 01:02:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C9DD613B9 for ; Fri, 25 Jun 2021 01:02:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C9DD613B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OJNSMLx/eAbsAJHHn8SAzf2E9PqxLcbTJMdj9KfpXY8=; b=AWSk6NZFH07XWN pmEFkqK3fzvXDSe0lxZpnZrL0ZFoHZey8WengRscZMoZaoXVRYx/Wv04Yc3v1ptzapQyTOSMLGe2M tgzcLK/Ji9VIaGh89UigQ7LH4LuEpJReVMAFcB4zOeBoPYuiNJHjhGkBzFrFovVyuvOUyo6qsB1Zq Yft2KnJrfAE0XjYhb0wjvT44hxdKdx8YKsyKJvPVJZnmmfU/64RTrwolSXPs0JhLShkxsYduZ5u3V qGOgdl8tp897me5Sap5geekJVAX/9aD4yPNv75Hkt00OZfZjNepUK6nIs+GevUZoqgFev7PXUriph vgaDOUnouUmcwYlVls5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEr-00GvsB-K9; Fri, 25 Jun 2021 01:02:33 +0000 Received: from mail-ej1-f53.google.com ([209.85.218.53]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEo-00Gvpz-LW for linux-riscv@lists.infradead.org; Fri, 25 Jun 2021 01:02:32 +0000 Received: by mail-ej1-f53.google.com with SMTP id he7so12358219ejc.13 for ; Thu, 24 Jun 2021 18:02:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5d1bwUaSuG/f6mTu6ru/WOtu9a5hLw1cNtYO/dabh8E=; b=Kf5u1Lo3qEoUt7/Kn0Pg9+wvOtM02MSCvCOqHUTAuJz1RNOyHNcwZn4pNkEL8TTd5r kP0TT3eKJ0kZcaDr5wfGM5EAmt8mbLqPDf7MzRH+sNsIRzdXQ3q6jVWYZsmflq2YrkW3 f+M7aNUEQy8lzU3YnLdI0vy92Km+BCOENdUK+6lV6fLp0TvcQk/NeTJjRkhDprW4yRCF 2iBeCCRbAqNMuJhbhhrKgb1lUC43D6gVhEphAWbvS414+U3YFm4lVrWtl1HRfAAe8RH5 M1hTZR/UJauFVRxUi1LYHjCf+6VEf43VP9+2iZJK926jrkkL5rK+ZMTzJnCQ23kaqqiU +FKQ== X-Gm-Message-State: AOAM532yAVjpe2XoPhpXu3syRIsZOOQdZ2EDNy/rbJTEUhYXJ3LiXCw6 kFs4ttRY8EFgy1XVLZAvU9U= X-Google-Smtp-Source: ABdhPJxINwo0+vWL6bmf2rOXNOQTpbhqaRuhL6BUfM1PoIYKKfYvwU+5wDopzTUQ2tk9MFuhlrmHSA== X-Received: by 2002:a17:906:30d0:: with SMTP id b16mr8028751ejb.495.1624582949235; Thu, 24 Jun 2021 18:02:29 -0700 (PDT) Received: from msft-t490s.home (host-95-251-17-240.retail.telecomitalia.it. [95.251.17.240]) by smtp.gmail.com with ESMTPSA id yc29sm1921909ejb.106.2021.06.24.18.02.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 18:02:28 -0700 (PDT) From: Matteo Croce To: linux-kernel@vger.kernel.org, Nick Kossifidis , Guo Ren , Christoph Hellwig , David Laight , Palmer Dabbelt , Emil Renner Berthing , Drew Fustini Cc: linux-arch@vger.kernel.org, Andrew Morton , Nick Desaulniers , linux-riscv@lists.infradead.org Subject: [PATCH 2/3] lib/string: optimized memmove Date: Fri, 25 Jun 2021 03:01:59 +0200 Message-Id: <20210625010200.362755-3-mcroce@linux.microsoft.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210625010200.362755-1-mcroce@linux.microsoft.com> References: <20210625010200.362755-1-mcroce@linux.microsoft.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210624_180230_793526_E9D1B4BE X-CRM114-Status: GOOD ( 12.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Matteo Croce When the destination buffer is before the source one, or when the buffers doesn't overlap, it's safe to use memcpy() instead, which is optimized to use a bigger data size possible. This "optimization" only covers a common case. In future, proper code which does the same thing as memcpy() does but backwards can be done. Signed-off-by: Matteo Croce --- lib/string.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/lib/string.c b/lib/string.c index 15e906f97d9e..69adec252597 100644 --- a/lib/string.c +++ b/lib/string.c @@ -976,19 +976,13 @@ EXPORT_SYMBOL(memcpy); */ void *memmove(void *dest, const void *src, size_t count) { - char *tmp; - const char *s; + if (dest < src || src + count <= dest) + return memcpy(dest, src, count); + + if (dest > src) { + const char *s = src + count; + char *tmp = dest + count; - if (dest <= src) { - tmp = dest; - s = src; - while (count--) - *tmp++ = *s++; - } else { - tmp = dest; - tmp += count; - s = src; - s += count; while (count--) *--tmp = *--s; } From patchwork Fri Jun 25 01:02:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Croce X-Patchwork-Id: 12343413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCD40C49EA5 for ; Fri, 25 Jun 2021 01:02:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C38B6135A for ; Fri, 25 Jun 2021 01:02:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C38B6135A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NMz3RgL50z8N7GLVde3RL1kruVhn+lCjZ/M6E4UyPPc=; b=B/7lZzAXJqX+DH Bud38LWRjouP/OhHmm9S93943PTTAjmN4nUVfgtxtD8d/UCR6EIod02MnTvxoU4yEsGCJ42uD6bZu EK/AwqK9Oand3Ywyopa+DyZCC2+E726QMvz8RFRCLDtR2a5yRxKR44Es4rJ+8+/hCLjdgC1s6Xznz xX3mtGhbNjBmFRUsrIeFcHF3no6enDIZM9x1GxG89mMh5iLg7ZvctvTDtdvuii9rHlHfIRyN58veo xnvXPIi40O1/XejVRlMNa2/+X0RzwkJZAWrmT/d/+bmXLqS3W/0c82hvYlzVeiV3GwiRkl0ZTCWUa ZQ0bja3n5ir1BAvbIjjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEu-00GvtP-4m; Fri, 25 Jun 2021 01:02:36 +0000 Received: from mail-ej1-f45.google.com ([209.85.218.45]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lwaEq-00Gvqr-F3 for linux-riscv@lists.infradead.org; Fri, 25 Jun 2021 01:02:34 +0000 Received: by mail-ej1-f45.google.com with SMTP id nb6so12401222ejc.10 for ; Thu, 24 Jun 2021 18:02:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PTl2a45oK7ererAoqlIPpBJuapGCJZ6anTugsk9TTSQ=; b=pTJ5YNLaQMZVJQNbMIHMZPqGTpuWLabIgCYWzok7f5nwOWqqybv/q1Js5nFYIsf/8Q Pni3M3xnw64FDyHl8E+MK/oxYa0TO1wGoOieKeu1LKPDrvRaMV9nVFhjrkQWhwbrSmbn doTN5T/1R3OL82JcuFRuw+rrA7YZcvOnVUEmseJtpnFGxlikBF8BtIwVNiF54/u5yC/I mBGwrbDmA1PWTgb2sGrVxXFXNc+DFSF2r2LfwGNFtOj2oj6TFRa6fV5iO435DoBQ9nfP M6elpZVgxKxvQbZQGPs7il59krezg37A2KxXBlhCSymsiq6UI/gLT3mNWRur42GkHa2t UAhA== X-Gm-Message-State: AOAM5322zXd6jCLZ7r45ZwT3xyQLEZcVMeUpeC7IpD4vHtaJ3KPqowEz JNtP467FO/kT0g3lb6XS1PA= X-Google-Smtp-Source: ABdhPJxVCiXfkPACT3+y9E9wzhTSnrJLbSJnilSAXh81SgJREFTyL7yrttPSEAYBAPGeBAxl1Ro+MQ== X-Received: by 2002:a17:906:c1d0:: with SMTP id bw16mr8146696ejb.146.1624582950900; Thu, 24 Jun 2021 18:02:30 -0700 (PDT) Received: from msft-t490s.home (host-95-251-17-240.retail.telecomitalia.it. [95.251.17.240]) by smtp.gmail.com with ESMTPSA id yc29sm1921909ejb.106.2021.06.24.18.02.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 18:02:30 -0700 (PDT) From: Matteo Croce To: linux-kernel@vger.kernel.org, Nick Kossifidis , Guo Ren , Christoph Hellwig , David Laight , Palmer Dabbelt , Emil Renner Berthing , Drew Fustini Cc: linux-arch@vger.kernel.org, Andrew Morton , Nick Desaulniers , linux-riscv@lists.infradead.org Subject: [PATCH 3/3] lib/string: optimized memset Date: Fri, 25 Jun 2021 03:02:00 +0200 Message-Id: <20210625010200.362755-4-mcroce@linux.microsoft.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210625010200.362755-1-mcroce@linux.microsoft.com> References: <20210625010200.362755-1-mcroce@linux.microsoft.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210624_180232_555151_1CB3B248 X-CRM114-Status: GOOD ( 15.31 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Matteo Croce The generic memset is defined as a byte at time write. This is always safe, but it's slower than a 4 byte or even 8 byte write. Write a generic memset which fills the data one byte at time until the destination is aligned, then fills using the largest size allowed, and finally fills the remaining data one byte at time. Signed-off-by: Matteo Croce --- lib/string.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/lib/string.c b/lib/string.c index 69adec252597..598ece5434e9 100644 --- a/lib/string.c +++ b/lib/string.c @@ -811,10 +811,36 @@ EXPORT_SYMBOL(__sysfs_match_string); */ void *memset(void *s, int c, size_t count) { - char *xs = s; + union types dest = { .as_u8 = s }; + if (count >= MIN_THRESHOLD) { + unsigned long cu = (unsigned long)c; + + /* Compose an ulong with 'c' repeated 4/8 times */ + cu |= cu << 8; + cu |= cu << 16; +#if BITS_PER_LONG == 64 + cu |= cu << 32; +#endif + + if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + /* + * Fill the buffer one byte at time until + * the destination is word aligned. + */ + for (; count && dest.as_uptr & word_mask; count--) + *dest.as_u8++ = c; + } + + /* Copy using the largest size allowed */ + for (; count >= bytes_long; count -= bytes_long) + *dest.as_ulong++ = cu; + } + + /* copy the remainder */ while (count--) - *xs++ = c; + *dest.as_u8++ = c; + return s; } EXPORT_SYMBOL(memset);