From patchwork Fri Feb 9 04:06:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 826A2C48297 for ; Fri, 9 Feb 2024 04:07:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6FA78D0011; Thu, 8 Feb 2024 23:07:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1F728D0001; Thu, 8 Feb 2024 23:07:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC0518D0011; Thu, 8 Feb 2024 23:07:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B6E458D0001 for ; Thu, 8 Feb 2024 23:07:40 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 943484019E for ; Fri, 9 Feb 2024 04:07:40 +0000 (UTC) X-FDA: 81770931480.02.CC9B5C1 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf09.hostedemail.com (Postfix) with ESMTP id BF2DB140032 for ; Fri, 9 Feb 2024 04:07:38 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xux0hNhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451658; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=XR+Ex508yDnLk8ZIWngzWeNS18a3PPFZQcvNUU6kamxdaP6GF5UwXxUAfn7l6CgOemh72Q OFn+wTXdFwm+j5EQzXhSs14YnRWAU6hdV1GX+DA9qp7H79ELMup0gbEahf4pGn0ZkhN/Pk KB6+29Ho/tfD4VYFx8Nck1JqM/i+hQI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xux0hNhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451658; a=rsa-sha256; cv=none; b=u5Z8Qya06u0VgQZZ7luPXCeWYB2WaW4NmQIxMRqAbgH5Q44bFIK8MLAuijdk43xMLV+bj6 XT3sR9WFL0mto+0wVjzF9ldYF9+Q0v7htumlQyw46ExxQ5tSNuFzIlYEpmS1I+GInJ9DgK KosSSNKbAZZYpLKBW/dPiumCwqdIUSE= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1d780a392fdso3994745ad.3 for ; Thu, 08 Feb 2024 20:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451657; x=1708056457; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=Xux0hNhz8ST1kaggi9hr/yJcdfFWyvpsWP0Q2A0dqtHGxznY9srlcBXCdqcAiBryEj In4Evw9wOeun9vL7fir2v4VxCqAFZiEKDdSwmxRlzqMwp7CVDk78NXmhjIXzMuWjyU8E k6dodMDdvQ2uE27ySKvVTJBbDbgWaHi/3vU9B3buviDV2++AeTMTEFISxgIL+bcMqm3M C2sS9QDMoFkKITd+4SuJX+SR74n14AQlX50KwL1NqYXox0/q9Th5SZ1Ma+cKAea80YUM 78pkmsQ0rKEmpAKZcZhmXQ7qdB9T4JRi6Jqw1mjddQ9aFTZpdfiY27pycNtKkQv3nQVa qbjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451657; x=1708056457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=AlIDGGCWBlL3vZMGPJ0YHwwR4NdOcbJMeuqd3CkRuUeciqZp9NlNAuiipYKURIvej2 SEyMCsP548BdqhDUWjQiL4EDxyDaOjWy/FqINriPzbRS7hts3ymLrkeyFVkyhbpR020I k9uM+oxuGzftxkH0x9B0hNfmzWwLjVpK+OCHqOb1FGKxGH/CqSZFaEzusm+hSdHji0sh MrohSXUkfqi8+CXkLrXHRyQt92CEPidL/8MMReHVU6EodpPzvYw68anVjctXr/8UJIk6 Bylnl389yUHDW0xB2SvfHdQLX9SOLAaR8YQBabrcjNGDj3WTVFTHG1jqk9odBYDEb57y bskA== X-Gm-Message-State: AOJu0Yyb8q897kwFNJoBdzHBZ1GH970eRgnqCASBqXgprMbEqRZGqWsO dZCa/A8xMPTqCn0xm+cH9GCwEk8S7zIyprXSyVLGug4adr8bb5DK X-Google-Smtp-Source: AGHT+IGkOiYtxyH9h4Hxmx0HmJv/rugkioUJ3yihq6NNOSDK+ZXse18c8Q3zRElKlWdCYTq+E6Xxsw== X-Received: by 2002:a17:903:2a90:b0:1d6:f185:f13b with SMTP id lv16-20020a1709032a9000b001d6f185f13bmr618327plb.17.1707451657685; Thu, 08 Feb 2024 20:07:37 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXDiS092+UkDyF4aEZPoblkWtOoQh1U8TGTdcCSduROIhOu+T04/eDIqGY6NzkwsOOzzZi0ztZBRPjETZoAFTsBxT2B3gdn7EYY0CrHwnxfASTjykvyk4yUPd99WN1zKGqA6dczojNPrJ0u4cqZeHkVclQfcV0afn+ij1NLT26mTYwX8tDsBlFcO3vI1tfpczHFjzQ5zEfAj3sIJaM7Ybtr2QS+kHiCwRn1K+R9vmundeWwOcBO4F+N9D7ITsVvn4Z59MKMI8lC9bdSbwy4VqUhhg30uZJOtPtrVLHLhwPbV64heLkaKzgW5Tk+DG2lHuTnYn9hyuJbcl0Hv6xLbiFvHOEwp1ebFpnSWQc32i3o5DeLAiU5HQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id l4-20020a170902d04400b001d9fcd344afsm541162pll.222.2024.02.08.20.07.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:37 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 20/20] selftests/bpf: Convert simple page_frag allocator to per-cpu. Date: Thu, 8 Feb 2024 20:06:08 -0800 Message-Id: <20240209040608.98927-21-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BF2DB140032 X-Stat-Signature: enu55b9cdgokpugbmqjjofgno8ay11dj X-Rspam-User: X-HE-Tag: 1707451658-72902 X-HE-Meta: U2FsdGVkX19nGH3Vk4ulw44LYEjx+FH3GK1+Cgj6TrvG0p36IoSuE7YrEsJyQ9i2NbbawmMJFBQwO+vj6U6X3HbcKxjle9bQK0bfPLLHaNjCjHJ6dOgB46WMHWNg+gSYUzO2DD/id8LoCtlXtwB7x1E2dZX1afgbNJiZdKaAcwY8BmcqvkftYe29IOnuM8WBDjCHcth3ijttzzAwPCUTw3I5dEBiX58xo76vpD00NzN8jE6fTkYf8CZgGjLT94D8SgT3qrUmtQFmO1n0LBBPcOXxpZV5syGdWbEWOer9d3Ye/AEkzUururxzF4ISQeOkJtKF/Uu1fQ/bdUTP6+ZDHdO2VZH/4aO8EOMwFlL3XP3vq2HnWgnX8qtHNTJvDo/n7QM0h5vl4nH8O5YjJTOGrrLB1v7CemYKDNA/dsdNgd5CQIEke6LvU7XsFAEyEl64jCuYEf/7tj+hVz7x9z2q1MS7OPFtZZ7UgapPurizFDemcng5zJ5s1emhmV7aGh7C1sooN6VdLy0Tn66k0fMC9/ZZXY/eeXMI5Csrk9eePGijMRXHYcxFxna+NV8nnAQIsHjdI0U9ofpTHglJa0T3SsxHbXCNuoh0T1wucespfofgWHcGydkEMwaG0iSqWWxVngBin+GF44Qt6DxtH2rnMXL4uUaH2pJzQadtRIL/5AhH/a4+gZktUAcNF2GYecEwU5M0UBSKAQ/zr9u7PqYVXSopIwxkRChwplcK2J7ofPDnwxzjoeK0JXzy2/Wmizu/vjj7L+ifRPrwb8xC5DV10JTvPtL8kSvfnhxjIZbxCKxP5dVkJ8N1+yBAw4Gck6ZLvENZes6MvmI7ENim27mmwWHCXvG7dQZGINNfUKXeMcYqEWuEhO2pQbl+HTW7KuJB7IzW5h8l+tWiSseZ5kQ816nVkkZNq0XsFZHz9EtSrSLF4XEaAtO4tsIf3j8MV8KOeD6EvfdQxetNelMh7cP 7ODUMkB6 7seJ72tRLg0mAqKSveT7ptxW2vAGLb7UEQvMSzybOC/UtQ/Daj5u3iLiccIr/5PhPWRK429NxeRVR+vY/TJPk9vFP22C6nfsY0bhXQqjrRdXJOkOnsnBE2dLARcI0aV1YcKVbs+HoO2iJJ7hDJMzaKr7F3LkhFORkHiWrO/UI2MP05xYK+wvrNytA8zBLOtwt45kYhmcPbMVab3CbrG20hGV04fKYyETjM5wCIOS8AlMOYXT3/+lMYVRb3m+cKCCi3S20MfUUcTkSbBHDrt2Q0P18CsknJ3TiA2z5Oc4vdIdQKqg5VEyLSFK7bs2yj1e2M4VOwUX4dUlww0/4bHAwTeUF3jQOdKaJyj71cubQJi1DNldYRLQK3e6KhimR91+KsPCPxHcb01thGGy+ASOw6LsNL8HSz4kdpLv9KcjUQw+wFFtA2hn1VhtpiasARYaw+uDpLwnSyaWzvrhf/dscs5iAM5pK1xtJXv4yf1hrIIVgDJqe5IjqFSO5VrhH+mGb7Inl1Mh2hCQM+zIqY5jT10VlMsJNvzbHxEO/mU+JFylQYbJWwN7k3Lcijlboy7iL0RTPmUkDoOuQvQMmP+0L6OZFVbn5yuFGE8CK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Convert simple page_frag allocator to per-cpu page_frag to further stress test a combination of __arena global and static variables and alloc/free from arena. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/bpf_arena_alloc.h | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_arena_alloc.h b/tools/testing/selftests/bpf/bpf_arena_alloc.h index 0f4cb399b4c7..c27678299e0c 100644 --- a/tools/testing/selftests/bpf/bpf_arena_alloc.h +++ b/tools/testing/selftests/bpf/bpf_arena_alloc.h @@ -10,14 +10,19 @@ #define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1) #endif -void __arena *cur_page; -int cur_offset; +#ifdef __BPF__ +#define NR_CPUS (sizeof(struct cpumask) * 8) + +static void __arena * __arena page_frag_cur_page[NR_CPUS]; +static int __arena page_frag_cur_offset[NR_CPUS]; /* Simple page_frag allocator */ static inline void __arena* bpf_alloc(unsigned int size) { __u64 __arena *obj_cnt; - void __arena *page = cur_page; + __u32 cpu = bpf_get_smp_processor_id(); + void __arena *page = page_frag_cur_page[cpu]; + int __arena *cur_offset = &page_frag_cur_offset[cpu]; int offset; size = round_up(size, 8); @@ -29,8 +34,8 @@ static inline void __arena* bpf_alloc(unsigned int size) if (!page) return NULL; cast_kern(page); - cur_page = page; - cur_offset = PAGE_SIZE - 8; + page_frag_cur_page[cpu] = page; + *cur_offset = PAGE_SIZE - 8; obj_cnt = page + PAGE_SIZE - 8; *obj_cnt = 0; } else { @@ -38,12 +43,12 @@ static inline void __arena* bpf_alloc(unsigned int size) obj_cnt = page + PAGE_SIZE - 8; } - offset = cur_offset - size; + offset = *cur_offset - size; if (offset < 0) goto refill; (*obj_cnt)++; - cur_offset = offset; + *cur_offset = offset; return page + offset; } @@ -56,3 +61,7 @@ static inline void bpf_free(void __arena *addr) if (--(*obj_cnt) == 0) bpf_arena_free_pages(&arena, addr, 1); } +#else +static inline void __arena* bpf_alloc(unsigned int size) { return NULL; } +static inline void bpf_free(void __arena *addr) {} +#endif