From patchwork Tue Jun 2 11:34:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "zhujianwei (C)" X-Patchwork-Id: 11583623 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14068138C for ; Tue, 2 Jun 2020 11:34:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F082B2068D for ; Tue, 2 Jun 2020 11:34:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725919AbgFBLeH (ORCPT ); Tue, 2 Jun 2020 07:34:07 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:2515 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725958AbgFBLeH (ORCPT ); Tue, 2 Jun 2020 07:34:07 -0400 Received: from DGGEMM403-HUB.china.huawei.com (unknown [172.30.72.56]) by Forcepoint Email with ESMTP id 09CCC7A09DDE914F5275; Tue, 2 Jun 2020 19:34:05 +0800 (CST) Received: from dggema706-chm.china.huawei.com (10.3.20.70) by DGGEMM403-HUB.china.huawei.com (10.3.20.211) with Microsoft SMTP Server (TLS) id 14.3.487.0; Tue, 2 Jun 2020 19:34:04 +0800 Received: from dggema758-chm.china.huawei.com (10.1.198.200) by dggema706-chm.china.huawei.com (10.3.20.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1913.5; Tue, 2 Jun 2020 19:34:04 +0800 Received: from dggema758-chm.china.huawei.com ([10.9.48.193]) by dggema758-chm.china.huawei.com ([10.9.48.193]) with mapi id 15.01.1913.007; Tue, 2 Jun 2020 19:34:04 +0800 From: "zhujianwei (C)" To: Alexei Starovoitov , Kees Cook CC: "bpf@vger.kernel.org" , "linux-security-module@vger.kernel.org" , Hehuazhen , Lennart Poettering , Christian Ehrhardt , =?utf-8?q?Zbigniew_J?= =?utf-8?q?=C4=99drzejewski-Szmek?= Subject: =?utf-8?b?562U5aSNOiDnrZTlpI06IG5ldyBzZWNjb21wIG1vZGUgYWltcyB0byBp?= =?utf-8?b?bXByb3ZlIHBlcmZvcm1hbmNl?= Thread-Topic: =?utf-8?q?=E7=AD=94=E5=A4=8D=3A_new_seccomp_mode_aims_to_impro?= =?utf-8?q?ve_performance?= Thread-Index: AdY1q17j91IY6CMiRsq40mFg/pmPz///wxIAgAAHIgCAADc1gP/7503AgAfEWYD//fXnAIADmuoA//74EOD//e1ysA== Date: Tue, 2 Jun 2020 11:34:04 +0000 Message-ID: <07ce4c1273054955a350e67f2dc35812@huawei.com> References: <202005290903.11E67AB0FD@keescook> <202005291043.A63D910A8@keescook> <7dacac003a9949ea8163fca5125a2cae@huawei.com> <20200602032446.7sn2fmzsea2v2wbs@ast-mbp.dhcp.thefacebook.com> Accept-Language: en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.166.215.96] MIME-Version: 1.0 X-CFilter-Loop: Reflected Sender: owner-linux-security-module@vger.kernel.org Precedence: bulk List-ID: > > > This is the test result on linux 5.7.0-rc7 for aarch64. > > > And retpoline disabled default. > > > #cat /sys/devices/system/cpu/vulnerabilities/spectre_v2 > > > Not affected > > > > > > bpf_jit_enable 1 > > > bpf_jit_harden 0 > > > > > > We run unixbench syscall benchmark on the original kernel and the new one(replace bpf_prog_run_pin_on_cpu() with immediately returning 'allow' one). > > > The unixbench syscall testcase runs 5 system calls（close/umask/dup/getpid/getuid, extra 15 syscalls needed to run it） in a loop for 10 seconds, counts the number and finally output it. We also add some more filters (each with the same rules) to evaluate the situation just like kees mentioned(case like systemd-resolve), and we find it is right: more filters, more overhead. The following is our result (./syscall 10 m): > > > > > > original: > > > seccomp_off: 10684939 > > > seccomp_on_1_filters: 8513805 overhead：19.8% > > > seccomp_on_4_filters: 7105592 overhead：33.0% > > > seccomp_on_32_filters: 2308677 overhead：78.3% > > > > > > after replacing bpf_prog_run_pin_on_cpu: > > > seccomp_off: 10685244 > > > seccomp_on_1_filters: 9146483 overhead：14.1% > > > seccomp_on_4_filters: 8969886 overhead：16.0% > > > seccomp_on_32_filters: 6454372 overhead：39.6% > > > > > > N-filter bpf overhead: > > > 1_filters: 5.7% > > > 4_filters: 17.0% > > > 32_filters: 38.7% > > > > > > // kernel code modification place > > > static noinline u32 bpf_prog_run_pin_on_cpu_allow(const struct > > > bpf_prog *prog, const void *ctx) { > > > return SECCOMP_RET_ALLOW; > > > } > > > > >This is apples to oranges. > > >As explained earlier: > > >https://lore.kernel.org/netdev/20200531171915.wsxvdjeetmhpsdv2@ast-mb > > >p.dhcp.thefacebook.com/T/#u Please use __weak instead of static and > > >redo the numbers. > > > > > > we have replaced ‘static’ with ‘__weak’, tested with the same way, and got almostly the same result, in our test environment(aarch64). > > > > -static noinline u32 bpf_prog_run_pin_on_cpu_allow(const struct > > bpf_prog *prog, const void *ctx) > > +__weak noinline u32 bpf_prog_run_pin_on_cpu_allow(const struct > > +bpf_prog *prog, const void *ctx) > > > > original: > > seccomp_off: 10684939 > > seccomp_on_1_filters: 8513805 overhead：19.8% > > seccomp_on_4_filters: 7105592 overhead：33.0% > > seccomp_on_32_filters: 2308677 overhead：78.3% > > > > after replacing bpf_prog_run_pin_on_cpu: > > seccomp_off: 10667195 > > seccomp_on_1_filters: 9147454 overhead：14.2% > > seccomp_on_4_filters: 8927605 overhead：16.1% > > seccomp_on_32_filters: 6355476 overhead：40.6% > > are you saying that by replacing 'static' with '__weak' it got slower?! > > Something doesn't add up. Please check generated assembly. > > By having such 'static noinline bpf_prog_run_pin_on_cpu' you're telling compiler to remove most of seccomp_run_filters() code which now will return only two possible values. Which further means that large 'switch' > > statement in __seccomp_filter() is also optimized. populate_seccomp_data() is removed. Etc, etc. That explains 14% vs 19% difference. > > May be you have some debug on? Like cant_migrate() is not a nop? > > Or static_branch is not supported? > > The sure way is to check assembly. > No, we say that by replacing 'static' with '__weak' it got the same result, in our testcase which filters 20 allowed syscall num (for details, see the previous post). > > static case: > N-filter bpf overhead: > 1_filters: 5.7% > 4_filters: 17.0% > 32_filters: 38.7% > > __weak case: > N-filter bpf overhead: > 1_filters: 5.6% > 4_filters: 16.9% > 32_filters: 37.7% And in many scenarios, the requirement for syscall filter is usually simple, and does not need complex filter rules, for example, just configure a syscall black or white list. However, we have noticed that seccomp will have a performance overhead that cannot be ignored in this simple scenario. For example, referring to Kees's t est data, this cost is almost 41/636 = 6.5%, and Alex's data is 17/226 = 7.5%, based on single rule of filtering (getpid); Our data for this overhead is 19.8% (refer to the previous 'orignal' test results), filtering based on our 20 rules (unixbench syscall). To improve the filtering performance in these scenarios, as a PoC, we replaced calling seccomp_computing() in syscall_trace_enter(), with a light_syscall_filter() which uses a bitmap for syscall filter, and only filter syscall-num-only case without the syscall-args case. The light_syscall_filter() implemented as a sub-branch in seccomp_computing(). To measure the performance, we use the same unixbench syscall testcase with 20 allowed syscall-num rules. The result shows that the light_syscall_filter only imposed 1.2% overhead. Can we take a deep discussion to add this light filter mode? This is the details: light_syscall_filter: //run unixbench syscall testcase 10 times and get the average. seccomp_off: 10684018 seccomp_on_1_filters: 10553233 overhead: 1.2% // kernel modification + } if (test_thread_flag(TIF_SYSCALL_TRACEPOINT)) trace_sys_enter(regs, regs->syscallno); --- linux-5.7-rc7_1/arch/arm64/kernel/ptrace.c 2020-05-25 06:32:54.000000000 +0800 +++ linux-5.7-rc7/arch/arm64/kernel/ptrace.c 2020-06-02 12:35:04.412000000 +0800 @@ -1827,6 +1827,46 @@ regs->regs[regno] = saved_reg; } +#define PID_MAX 1000000 +#define SYSNUM_MAX 0x220 + +/* all zero*/ +bool g_light_filter_switch[PID_MAX] = {0}; +bool g_light_filter_bitmap[PID_MAX][SYSNUM_MAX] = {0}; + + +static int __light_syscall_filter(void) { + int pid; + int this_syscall; + + pid = current->pid; + this_syscall = syscall_get_nr(current, task_pt_regs(current)); + + if(g_light_filter_bitmap[pid][this_syscall] == true) { + printk(KERN_ERR "light syscall filter: syscall num %d denied.\n", this_syscall); + goto skip; + } + + return 0; +skip: + return -1; +} + +static inline int light_syscall_filter(void) { + if (unlikely(test_thread_flag(TIF_SECCOMP))) { + return __light_syscall_filter(); + } + + return 0; +} + int syscall_trace_enter(struct pt_regs *regs) { unsigned long flags = READ_ONCE(current_thread_info()->flags); @@ -1837,9 +1877,10 @@ return -1; } - /* Do the secure computing after ptrace; failures should be fast. */ - if (secure_computing() == -1) + /* light check for syscall-num-only rule. */ + if (light_syscall_filter() == -1) { return -1;