From patchwork Mon Dec 17 16:23:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vasily Averin X-Patchwork-Id: 10733767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7173C13BF for ; Mon, 17 Dec 2018 16:23:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D49E286EE for ; Mon, 17 Dec 2018 16:23:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C2CE287BF; Mon, 17 Dec 2018 16:23:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,HEXHASH_WORD, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 92C912870C for ; Mon, 17 Dec 2018 16:23:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387865AbeLQQXh (ORCPT ); Mon, 17 Dec 2018 11:23:37 -0500 Received: from relay.sw.ru ([185.231.240.75]:51788 "EHLO relay.sw.ru" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387823AbeLQQXg (ORCPT ); Mon, 17 Dec 2018 11:23:36 -0500 Received: from [172.16.24.21] by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1gYvg6-0000zk-5J; Mon, 17 Dec 2018 19:23:34 +0300 From: Vasily Averin Subject: [PATCH 0/4] use-after-free in svc_process_common() To: "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker Cc: Chuck Lever , "linux-nfs@vger.kernel.org" , Evgenii Shatokhin , Konstantin Khorenko Message-ID: <8452a70e-b64b-8afc-d51e-e41a5dfcd240@virtuozzo.com> Date: Mon, 17 Dec 2018 19:23:33 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Language: en-US Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Unfortunately nfsv41+ clients are still not properly net-namespace-filied. OpenVz got report on crash in svc_process_common() abd founf that bc_svc_process() cannot use serv->sv_bc_xprt as a pointer. serv is global structure, but sv_bc_xprt is assigned per-netnamespace. If nfsv41+ shares (with the same minorversion) are mounted in several containers together then bc_svc_process() can use wrong backchannel or even access freed memory. OpenVz got report on crash svc_process_common(), and after careful investigations Evgenii Shatokhin have found its reproducer. Then I've reproduced the problem on last mainline kernel. In described scenario you need to have: - nodeA: VM with 2 interfaces and debug kernel with enabled KASAN. - nodeB: any other node - NFS-SRV: NFSv41+ server (4.2 is used in exaple below) 1) nodeA: mount nfsv41+ share # mount -t nfs4 -o vers=4.2 NFS-SRV:/export/ /mnt/ns1 VvS: here serv->sv_bc_xprt is assigned first time, in xs_tcp_bc_up() it is assigned to svc_xprt of mount's backchannel 2) nodeA: create net namespace, and mount the same (or any other) NFSv41+ share # ip netns add second # ip link set ens2 netns second # ip netns exec second bash (inside netns second) # dhclient ens2 VvS: now nets got access to external network (inside netns second) # mount -t nfs4 -o vers=4.2 NFS-SRV:/export/ /mnt/ns2 VvS: now serv->sv_bc_xprt is overwritten by reference to svc_xprt of new mount's backchannel NB: you can mount any other NFS share but minorversion must be the same. NB2: if hardware allows you can use rdma transport here NB3: you can access nothing in mounted share, problem's trigger was enabled already. 3) NodeA, destroy mount inside netns and then netns itself. (inside netns second) # umount /mnt/ns2 (inside netns second) # ip link set ens2 netns 1 (inside netns second) # exit VvS: return to init_net # ip netns del second VvS: now second NFS mount and second net namespace was destroyed. 4) Node A: prepare backchannel event # echo test1 > /mnt/ns1/test1.txt # echo test2 > /mnt/ns1/test2.txt # python >>> fl=open('/mnt/ns1/test1.txt','r') >>> 4) Node B: replace file open by NodeA # mount -t nfs -o vers=4.2 NFS-SRV:/export/ /mnt/ # mv /mnt/test2.txt /mnt/test1.txt ===> KASAN on nodeA detect an access to already freed memory. (see dmesg example below for details) svc_process_common() /* Setup reply header */ rqstp->rq_xprt->xpt_ops->xpo_prep_reply_hdr(rqstp); <<< HERE svc_process_common() uses already freed rqstp->rq_xprt, it was assigned in bc_svc_process() where it was taken from serv->sv_bc_xprt. serv->sv_bc_xprt cannot be used as a pointer, it can be assigned per net-namespace, either in svc_bc_tcp_create() or in xprt_rdma_bc_up(). (Hopefully both transports cannot be used together in the same netns) To fix this problem I've added new callback to struct rpc_xprt_ops, it calls svc_find_xprt with proper name of transport's backchannel. serv->sv_bc_xprt is used in svc_is_backchannel() too. Here this filed is used not as pointer but as some mark of backchannel-compatible svc servers. My 2nd patch replaces sv_bc_xprt pointer to boolean flag, I hope it helps to prevent misuse of sv_bc_xprt in future. 3rd and 4th pathces are minor cleanup in debug messages. Vasily Averin (4): nfs: serv->sv_bc_xprt misuse in bc_svc_process() nfs: remove sv_bc_enabled using in svc_is_backchannel() nfs: minor typo in nfs4_callback_up_net() nfs: fix debug message in svc_create_xprt() fs/nfs/callback.c | 2 +- include/linux/sunrpc/bc_xprt.h | 10 ++++------ include/linux/sunrpc/svc.h | 2 +- include/linux/sunrpc/xprt.h | 1 + net/sunrpc/svc.c | 22 ++++++++++++++++------ net/sunrpc/svc_xprt.c | 4 ++-- net/sunrpc/svcsock.c | 2 +- net/sunrpc/xprtrdma/backchannel.c | 5 +++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- net/sunrpc/xprtrdma/transport.c | 1 + net/sunrpc/xprtrdma/xprt_rdma.h | 1 + net/sunrpc/xprtsock.c | 7 +++++++ 12 files changed, 41 insertions(+), 18 deletions(-)