From patchwork Sun Mar 19 00:20:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 13180173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DDD6C7618A for ; Sun, 19 Mar 2023 00:20:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37BE9900009; Sat, 18 Mar 2023 20:20:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 29002900004; Sat, 18 Mar 2023 20:20:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 108F7900009; Sat, 18 Mar 2023 20:20:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id ED0EB900004 for ; Sat, 18 Mar 2023 20:20:24 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D281540C17 for ; Sun, 19 Mar 2023 00:20:24 +0000 (UTC) X-FDA: 80583741168.13.62A094B Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by imf09.hostedemail.com (Postfix) with ESMTP id E8ABE140003 for ; Sun, 19 Mar 2023 00:20:22 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=huWPKdm+; spf=pass (imf09.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.54 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679185223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MTgm8eMaEME071JBKDzGLJgbn/ShH65iS57yVvkZwv4=; b=62sHf36tyobZcI6h7i57uQqmsUuaLvLdGrIEHd5n8LOay54FcqwTr9P0a+zwhyEBYiliLl XhSMw4uEOI1ssBsJHKSndJkWrj92b03PtUTOvnKq3/CRArlJm1xNRjQgqOyUEZ+m2ckhBM O/GeUo7JjMm5FCkgc6fYH9BWpp8Dnj8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=huWPKdm+; spf=pass (imf09.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.54 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679185223; a=rsa-sha256; cv=none; b=Buv5/mSECb1TPRmBd7SknbHxbcxKAblOBLP3WWM0DKXWou7cj1AYMfLckadeTvw83c50fn tDI4PQj8UO+g1LPyDKTcikHwqd/wTu0RWvW27ay9LccQS4PiyLFjSvGzmTsN1DhhZIvWyo tGC71Gp5KZl4bAE8DGk+DawSGBWSB3k= Received: by mail-wr1-f54.google.com with SMTP id o7so7383538wrg.5 for ; Sat, 18 Mar 2023 17:20:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679185221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MTgm8eMaEME071JBKDzGLJgbn/ShH65iS57yVvkZwv4=; b=huWPKdm+tRAG9YrVt08wts9oXX1u/L6VFZzCMnd846EM/acbas78Ee3zKgv7qmyDtD bQ4WNX1Lu+YmfrpGKGF9SyFdXzzkh/lHE4Iecb+gC8DONfvV1VWDU6hLr7f8aTtTmN7t HsZ+37pEqNuXSetVcUlvZvK5lC1BPpSHqEtT5J2tC7y9SBf3pcvrpBPKD0wEiJ0KPZiC ECZxUl8khPH3gNwSahF7FMoeapYF2Ooz4nvhyNQPEhGwJDLaneWyb6hY1U7a+bvCC0nr vFI3/O5GQGCBZMjn7KZkrweyNUhrDcliQX2NCWuxFeYw+PB+NuIWEt//techJMcX6FDx cMHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679185221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MTgm8eMaEME071JBKDzGLJgbn/ShH65iS57yVvkZwv4=; b=TKTzCqH67Y++epHxh78a+JuazrvSnKl1tiJdtFI6D3mW1BXcvKKb1ar5kSPOcwpjgR bG4gjOHN6af5cvEXiWMFI0nEYsRHlLdEyGkN6H7+zsFWBLz8+IpQXuvFdZWiFcTJPQUp /v1Nef59wXVfKJQ8JqyM9JQ1k2N+NgWZLccAVGDIxB/dXcC/aZtNegVTwFpLK2YrbNsc E7e9REEZMvmGgOP9shiVWDPIFMo4Oa13nbmfofA39b6qMM9rY/H5SweE3YxDVQp4I6cO AqsBowLBsTs6Z/ICwR1CT0B4MkoqxD4oPLPPIqtt0SMfRz4ah7ULHRITlYckO8f4FXWM FcWA== X-Gm-Message-State: AO0yUKV0NT0ZsGaL+IWShp/EEhVJ7L8XYJRWoVIjdohr5c1k8K78WuH6 k0wUF9jopEkz69trY7B+dJONjbXkMp0= X-Google-Smtp-Source: AK7set/8N5Yaps2ztad7B6HhBHCnmCxkP9fmkUJlMKQ2+v0qjXt1bYYTjp1A7l8qM/wVSHyDxqm1wQ== X-Received: by 2002:adf:fd12:0:b0:2ce:306d:6515 with SMTP id e18-20020adffd12000000b002ce306d6515mr9876348wrr.34.1679185221216; Sat, 18 Mar 2023 17:20:21 -0700 (PDT) Received: from lucifer.home (host86-146-209-214.range86-146.btcentralplus.com. [86.146.209.214]) by smtp.googlemail.com with ESMTPSA id x14-20020adfdd8e000000b002cff0c57b98sm5399639wrl.18.2023.03.18.17.20.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 18 Mar 2023 17:20:20 -0700 (PDT) From: Lorenzo Stoakes To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Andrew Morton Cc: Baoquan He , Uladzislau Rezki , Matthew Wilcox , David Hildenbrand , Liu Shixin , Jiri Olsa , Lorenzo Stoakes Subject: [PATCH 4/4] mm: vmalloc: convert vread() to vread_iter() Date: Sun, 19 Mar 2023 00:20:12 +0000 Message-Id: <119871ea9507eac7be5d91db38acdb03981e049e.1679183626.git.lstoakes@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: zt185yopr4gdg7ffot1cjqd9nfps1wm8 X-Rspam-User: X-Rspamd-Queue-Id: E8ABE140003 X-Rspamd-Server: rspam06 X-HE-Tag: 1679185222-371831 X-HE-Meta: U2FsdGVkX19QoUkR4mM/5wsUFXvhmIONnhXmtjyq8fs4hi2o0DEc/DvL8aa/wmjgnlrm7OoOsSEFVx80YJmVkEb45eNoauV8xWpr+gFdAQj9wPIKQpHNlnCj53lhAVq40aVZsNRwnU4xzqu0mQ5WPhpBg9YZcEmPD4cU9l3gpTVwV4U9lrFznRIjpJ/IGh4Me0xyDEU8ggXs/QqUjk5MJqGmIZUYgm+S43ZkKbrDHi3+2fotjoR8qVj3skACIS+JDCSFTvVVDIHVHY6Ri6c8xcmabwVzJozRbo6kee1pUJHHFWcnNasCH/h8GrRc0u+7vVs1xdLf6STmfPPES4nEkts1eYZC6scHXw/PgekhyHaYz9YWmeg6Tmn8nWST59tEjNHNmrehgoCTLnBFVxmOSgybh3NXn8acYhqDiHs/IOChVD4MGEPXzC+A3UepD81/PlEbSRDCgVWk7oVn1bcuDpW1p1Eye3ij00yBcNj2BBLxt5NAAt4ZGAMtLjsZXBbC7Od8ohjW7NZlAbXaJ4MMNvmk1nam1WUiA2OWsGQ3fCZjzsY3iCMi4CryHGrEa1eV3A3ce8OUUyx2Dk3eWY/vmGv1OcVBDStmWB0SoN2Kmhx0sznucu7resC46TydxAJwYRFv6ToNKukQLUs3TtBNdU+Hn4840UYDFTb/JOhVHScFjP4sZP5TF0CTgme4IT4SLQ1JYhZq2I/u3Pgxn4OC/5RA9lTd2b1Q1WrnZI7L23Gl8pQgWKMkw9bTzZ0Lzlvb2hFVROjo2ah5PdhTNNjbXffPMoZ+wwNQ63x8vTY7OYqo5sLc0vd8zCzJzIxkDSOIS9zmMRlrBzChPgcj1VkrzVB1v2NqIp1im3xukYsnwx8hDgQTHBl5yfJH4ZUC320dyPFe76xpdhui9LXIPDryATTFTup7HFqqlCrCWNIFF3/6KXWeSReZF9DesVFr5udrRLoTi7U389jiDu22GTL ZiYi961e SPEJPPb+4E5vClhJmsHP0RZkSG7opK7+rnbSO9Jr2Wa+YRK02xshUmhbFEh/oUwuHZUeKUYNSeZkxIRvzUxUjHVap4EPQ4rdwNumgvQLzwaDHB9JmwBGz/BEv3pMWOYgZy4Yc6/YQSBP2RqqEpWZ/zw47LnHoqYi5PKBMqIUBZOUvn8E3Ri6/45/vfA/frAwZnB2Wo64i3QS9feiYBlLXdDh/rZDB9rSDUA5rkfD+STuxxWgzj1GXgbWmly78MoeeYgh69jvn9JlhFmsMLapPA4fokA3i/cf34uMStf3KMWreqK6PUARNXiZBmVlWzECTw8qCrCBb/noEbhReSgYOgQ0GvdwJ5gQjC7q7IpIMn/rzZicTyvSTrPtnJRRArXIrntuSoF608Ac6hDmjEMSwGpkCS3dG3GEz+UfSP9ByZ0QO9pxOZqgd2f2oQvo13CVvVfna1vVrtFmfTrAeFcLjHBzPf/h7akuhQiH6UWBjFRSqLLxDRkDK6PXj3cy1Mnls1kIJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Having previously laid the foundation for converting vread() to an iterator function, pull the trigger and do so. This patch attempts to provide minimal refactoring and to reflect the existing logic as best we can, with the exception of aligned_vread_iter() which drops the use of the deprecated kmap_atomic() in favour of kmap_local_page(). All existing logic to zero portions of memory not read remain and there should be no functional difference other than a performance improvement in /proc/kcore access to vmalloc regions. Now we have discarded with the need for a bounce buffer at all in read_kcore_iter(), we dispense with the one allocated there altogether. Signed-off-by: Lorenzo Stoakes --- fs/proc/kcore.c | 21 +-------- include/linux/vmalloc.h | 3 +- mm/vmalloc.c | 101 +++++++++++++++++++++------------------- 3 files changed, 57 insertions(+), 68 deletions(-) diff --git a/fs/proc/kcore.c b/fs/proc/kcore.c index 25e0eeb8d498..8a07f04c9203 100644 --- a/fs/proc/kcore.c +++ b/fs/proc/kcore.c @@ -307,13 +307,9 @@ static void append_kcore_note(char *notes, size_t *i, const char *name, *i = ALIGN(*i + descsz, 4); } -static ssize_t -read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) +static ssize_t read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) { - struct file *file = iocb->ki_filp; - char *buf = file->private_data; loff_t *ppos = &iocb->ki_pos; - size_t phdrs_offset, notes_offset, data_offset; size_t page_offline_frozen = 1; size_t phdrs_len, notes_len; @@ -507,9 +503,7 @@ read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) switch (m->type) { case KCORE_VMALLOC: - vread(buf, (char *)start, tsz); - /* we have to zero-fill user buffer even if no read */ - if (copy_to_iter(buf, tsz, iter) != tsz) { + if (vread_iter((char *)start, tsz, iter) != tsz) { ret = -EFAULT; goto out; } @@ -582,10 +576,6 @@ static int open_kcore(struct inode *inode, struct file *filp) if (ret) return ret; - filp->private_data = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!filp->private_data) - return -ENOMEM; - if (kcore_need_update) kcore_update_ram(); if (i_size_read(inode) != proc_root_kcore->size) { @@ -596,16 +586,9 @@ static int open_kcore(struct inode *inode, struct file *filp) return 0; } -static int release_kcore(struct inode *inode, struct file *file) -{ - kfree(file->private_data); - return 0; -} - static const struct proc_ops kcore_proc_ops = { .proc_read_iter = read_kcore_iter, .proc_open = open_kcore, - .proc_release = release_kcore, .proc_lseek = default_llseek, }; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 69250efa03d1..f70ebdf21f22 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -9,6 +9,7 @@ #include /* pgprot_t */ #include #include +#include #include @@ -251,7 +252,7 @@ static inline void set_vm_flush_reset_perms(void *addr) #endif /* for /proc/kcore */ -extern long vread(char *buf, char *addr, unsigned long count); +extern long vread_iter(char *addr, size_t count, struct iov_iter *iter); /* * Internals. Don't use.. diff --git a/mm/vmalloc.c b/mm/vmalloc.c index c24b27664a97..3a32754266dc 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include @@ -3446,20 +3445,20 @@ EXPORT_SYMBOL(vmalloc_32_user); * small helper routine , copy contents to buf from addr. * If the page is not present, fill zero. */ - -static int aligned_vread(char *buf, char *addr, unsigned long count) +static void aligned_vread_iter(char *addr, size_t count, + struct iov_iter *iter) { - struct page *p; - int copied = 0; + struct page *page; - while (count) { + while (count > 0) { unsigned long offset, length; + size_t copied = 0; offset = offset_in_page(addr); length = PAGE_SIZE - offset; if (length > count) length = count; - p = vmalloc_to_page(addr); + page = vmalloc_to_page(addr); /* * To do safe access to this _mapped_ area, we need * lock. But adding lock here means that we need to add @@ -3467,23 +3466,24 @@ static int aligned_vread(char *buf, char *addr, unsigned long count) * interface, rarely used. Instead of that, we'll use * kmap() and get small overhead in this access function. */ - if (p) { + if (page) { /* We can expect USER0 is not used -- see vread() */ - void *map = kmap_atomic(p); - memcpy(buf, map + offset, length); - kunmap_atomic(map); - } else - memset(buf, 0, length); + void *map = kmap_local_page(page); + + copied = copy_to_iter(map + offset, length, iter); + kunmap_local(map); + } + + if (copied < length) + iov_iter_zero(length - copied, iter); addr += length; - buf += length; - copied += length; count -= length; } - return copied; } -static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags) +static void vmap_ram_vread_iter(char *addr, int count, unsigned long flags, + struct iov_iter *iter) { char *start; struct vmap_block *vb; @@ -3496,7 +3496,7 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags * handle it here. */ if (!(flags & VMAP_BLOCK)) { - aligned_vread(buf, addr, count); + aligned_vread_iter(addr, count, iter); return; } @@ -3517,22 +3517,24 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags if (!count) break; start = vmap_block_vaddr(vb->va->va_start, rs); - while (addr < start) { + + if (addr < start) { + size_t to_zero = min_t(size_t, start - addr, count); + + iov_iter_zero(to_zero, iter); + addr += to_zero; + count -= (int)to_zero; if (count == 0) goto unlock; - *buf = '\0'; - buf++; - addr++; - count--; } + /*it could start reading from the middle of used region*/ offset = offset_in_page(addr); n = ((re - rs + 1) << PAGE_SHIFT) - offset; if (n > count) n = count; - aligned_vread(buf, start+offset, n); + aligned_vread_iter(start + offset, n, iter); - buf += n; addr += n; count -= n; } @@ -3541,15 +3543,15 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags finished: /* zero-fill the left dirty or free regions */ - if (count) - memset(buf, 0, count); + if (count > 0) + iov_iter_zero(count, iter); } /** - * vread() - read vmalloc area in a safe way. - * @buf: buffer for reading data - * @addr: vm address. - * @count: number of bytes to be read. + * vread_iter() - read vmalloc area in a safe way to an iterator. + * @addr: vm address. + * @count: number of bytes to be read. + * @iter: the iterator to which data should be written. * * This function checks that addr is a valid vmalloc'ed area, and * copy data from that area to a given buffer. If the given memory range @@ -3569,13 +3571,13 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags * (same number as @count) or %0 if [addr...addr+count) doesn't * include any intersection with valid vmalloc area */ -long vread(char *buf, char *addr, unsigned long count) +long vread_iter(char *addr, size_t count, struct iov_iter *iter) { struct vmap_area *va; struct vm_struct *vm; - char *vaddr, *buf_start = buf; - unsigned long buflen = count; - unsigned long n, size, flags; + char *vaddr; + size_t buflen = count; + size_t n, size, flags; might_sleep(); @@ -3595,7 +3597,7 @@ long vread(char *buf, char *addr, unsigned long count) goto finished; list_for_each_entry_from(va, &vmap_area_list, list) { - if (!count) + if (count == 0) break; vm = va->vm; @@ -3619,36 +3621,39 @@ long vread(char *buf, char *addr, unsigned long count) if (addr >= vaddr + size) continue; - while (addr < vaddr) { + + if (addr < vaddr) { + size_t to_zero = min_t(size_t, vaddr - addr, count); + + iov_iter_zero(to_zero, iter); + addr += to_zero; + count -= to_zero; if (count == 0) goto finished; - *buf = '\0'; - buf++; - addr++; - count--; } + n = vaddr + size - addr; if (n > count) n = count; if (flags & VMAP_RAM) - vmap_ram_vread(buf, addr, n, flags); + vmap_ram_vread_iter(addr, n, flags, iter); else if (!(vm->flags & VM_IOREMAP)) - aligned_vread(buf, addr, n); + aligned_vread_iter(addr, n, iter); else /* IOREMAP area is treated as memory hole */ - memset(buf, 0, n); - buf += n; + iov_iter_zero(n, iter); + addr += n; count -= n; } finished: up_read(&vmap_area_lock); - if (buf == buf_start) + if (count == buflen) return 0; /* zero-fill memory holes */ - if (buf != buf_start + buflen) - memset(buf, 0, buflen - (buf - buf_start)); + if (count > 0) + iov_iter_zero(count, iter); return buflen; }