| Message ID | e9ee508d2db4499387d1a2d175ca269d@zhaoxin.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | 答复: A bug about cache inconsistency report | expand |
On Tue, 07 Aug 2018 12:59:04 +0200, Hans Hu(SH-RD) wrote: > > > >Then the next step would be to fake sg-buffer from this straight > > >buffer. Revert the above, and modify sgbuf.c to the following: > > >- Allocate a large continuous buffer > > >- Assign each page in this large buffer > > > > >If this still works, it's not about vmap, but it just means that the > > >physically ordered pages do matter -- implicitly showing that the > > >snooping behavior isn't properly turned on / off on the controller. > > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose. > > >Not really what I meant. Rather something like below (totally untested). > > [Hans :] I know what you mean now and complete code like below, but the result is still noise. OK, so indeed the vmapped address does seem matter. Interesting. What kind of user access does produce the noise? Does it via aplay mmap mode, too? In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct). Takashi --- diff --git a/sound/pci/hda/hda_intel.c b/sound/pci/hda/hda_intel.c --- a/sound/pci/hda/hda_intel.c +++ b/sound/pci/hda/hda_intel.c @@ -410,12 +410,23 @@ static void __mark_pages_wc(struct azx *chip, struct snd_dma_buffer *dmab, bool #ifdef CONFIG_SND_DMA_SGBUF if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_SG) { struct snd_sg_buf *sgbuf = dmab->private_data; + pgprot_t prot; + void *new_area; + if (chip->driver_type == AZX_DRIVER_CMEDIA) return; /* deal with only CORB/RIRB buffers */ - if (on) + if (on) { set_pages_array_wc(sgbuf->page_table, sgbuf->pages); - else + prot = pgprot_noncached(PAGE_KERNEL); + } else { set_pages_array_wb(sgbuf->page_table, sgbuf->pages); + prot = PAGE_KERNEL; + } + new_area = vmap(sgbuf->page_table, sgbuf->pages, VM_MAP, prot); + if (WARN_ON(!new_area)) + return; + vunmap(dmab->area); + dmab->area = new_area; return; } #endif
> > >Then the next step would be to fake sg-buffer from this straight > > >buffer. Revert the above, and modify sgbuf.c to the following: > > >- Allocate a large continuous buffer > > >- Assign each page in this large buffer > > > > >If this still works, it's not about vmap, but it just means that > > >the physically ordered pages do matter -- implicitly showing that > > >the snooping behavior isn't properly turned on / off on the controller. > > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose. > > >Not really what I meant. Rather something like below (totally untested). > > [Hans :] I know what you mean now and complete code like below, but the result is still noise. >OK, so indeed the vmapped address does seem matter. Interesting. >What kind of user access does produce the noise? Does it via aplay mmap mode, too? >In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct). [Hans :] As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to uncacheable type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user(). Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used by most music players. [Hans :] I complete the patch in your last mail like below, not used the vunmap & re-vmap way(it has a little problem). Result is OK now(no noise anymore). --- a/sound/pci/hda/hda_intel.c +++ b/sound/pci/hda/hda_intel.c #ifdef CONFIG_X86 +#ifdef CONFIG_SND_DMA_SGBUF +#include <linux/vmalloc.h> +#include <linux/sched.h> +#include <asm/paravirt.h> + +static int bad_address(void *p) +{ +unsigned long dummy; + +return probe_kernel_address((unsigned long *)p, dummy); +} + +static pte_t * dump_pte(unsigned long address) +{ +pgd_t *base = __va(read_cr3() & PHYSICAL_PAGE_MASK); +pgd_t *pgd = base + pgd_index(address); +pud_t *pud; +pmd_t *pmd; +pte_t *pte; + +if (bad_address(pgd)) +goto bad; + +if (!pgd_present(*pgd)) +goto bad; + +pud = pud_offset(pgd, address); +if (bad_address(pud)) +goto bad; + +if (!pud_present(*pud) || pud_large(*pud)) +goto bad; + +pmd = pmd_offset(pud, address); +if (bad_address(pmd)) +goto bad; + +if (!pmd_present(*pmd) || pmd_large(*pmd)) +goto bad; + +pte = pte_offset_kernel(pmd, address); +if (bad_address(pte)) +goto bad; + +return pte; +bad: +return NULL; +} +#endif static void __mark_pages_wc(struct azx *chip, struct snd_dma_buffer *dmab, bool on) { int pages; @@ -410,12 +459,28 @@ static void __mark_pages_wc(struct azx *chip, struct snd_dma_buffer *dmab, bool #ifdef CONFIG_SND_DMA_SGBUF if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_SG) { struct snd_sg_buf *sgbuf = dmab->private_data; +pgprot_t prot; +int i; +unsigned long addr; +pte_t * pte; if (chip->driver_type == AZX_DRIVER_CMEDIA) return; /* deal with only CORB/RIRB buffers */ -if (on) +if (on){ set_pages_array_wc(sgbuf->page_table, sgbuf->pages); -else +prot = pgprot_writecombine(PAGE_KERNEL); +} +else{ set_pages_array_wb(sgbuf->page_table, sgbuf->pages); +prot = PAGE_KERNEL; +} +for (i = 0; i < sgbuf->pages; i ++) +{ +addr = (unsigned long)dmab->area + PAGE_SIZE * i; +pte = dump_pte(addr); +if (pte) +set_pte_at(&init_mm, addr, pte, mk_pte(sgbuf->page_table[i], prot)); +else WARN_ON(1); +} return; } #endif 保密声明： 本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。 CONFIDENTIAL NOTE: This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
On Wed, 08 Aug 2018 11:54:57 +0200, Hans Hu(SH-RD) wrote: > > > > >Then the next step would be to fake sg-buffer from this straight > > > >buffer. Revert the above, and modify sgbuf.c to the following: > > > >- Allocate a large continuous buffer > > > >- Assign each page in this large buffer > > > > > > >If this still works, it's not about vmap, but it just means that > > > >the physically ordered pages do matter -- implicitly showing that > > > >the snooping behavior isn't properly turned on / off on the controller. > > > > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose. > > > > >Not really what I meant. Rather something like below (totally untested). > > > > [Hans :] I know what you mean now and complete code like below, but the result is still noise. > > >OK, so indeed the vmapped address does seem matter. Interesting. > >What kind of user access does produce the noise? Does it via aplay mmap mode, too? > > >In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct). > > [Hans :] As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to uncacheable type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user(). Well, that's usually no problem regarding that cache coherency. At least it hasn't been any issue with Intel and AMD CPUs. Does the problem happen with Intel CPU instead of VIA? > Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used by most music players. > > [Hans :] I complete the patch in your last mail like below, not used the vunmap & re-vmap way(it has a little problem). Result is OK now(no noise anymore). What problem did you get with vunmap / vmap? Takashi
> > > >Then the next step would be to fake sg-buffer from this straight > > > >buffer. Revert the above, and modify sgbuf.c to the following: > > > >- Allocate a large continuous buffer > > > >- Assign each page in this large buffer > > > > > > >If this still works, it's not about vmap, but it just means that > > > >the physically ordered pages do matter -- implicitly showing that > > > >the snooping behavior isn't properly turned on / off on the controller. > > > > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose. > > > > >Not really what I meant. Rather something like below (totally untested). > > > > [Hans :] I know what you mean now and complete code like below, but the result is still noise. > > >OK, so indeed the vmapped address does seem matter. Interesting. > >What kind of user access does produce the noise? Does it via aplay mmap mode, too? > > >In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct). > > [Hans :] As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to uncacheable type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user(). >Well, that's usually no problem regarding that cache coherency. >At least it hasn't been any issue with Intel and AMD CPUs. >Does the problem happen with Intel CPU instead of VIA? [Hans :] I don't have enough Intel/AMD machine here, up to now, only find one Intel's : HW: mother board is Dell 042P49, HDA controller is 8086:1c20, codec is cx20641. > Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used by most music players. > > [Hans :] I complete the patch in your last mail like below, not used the vunmap & re-vmap way(it has a little problem). Result is OK now(no noise anymore). >What problem did you get with vunmap / vmap? [Hans :] if patch like below, even the new vmapped address new_area be assigned to dmab->area, it will not use this time, but next. This time used dmab->area still the last one, but the old one had been vunmap, so Oops reported. +new_area = vmap(sgbuf->page_table, sgbuf->pages, VM_MAP, prot); +if (WARN_ON(!new_area)) +return; +vunmap(dmab->area); +dmab->area = new_area; Thanks, Hans 保密声明： 本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。 CONFIDENTIAL NOTE: This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
On Wed, 08 Aug 2018 12:44:00 +0200, Hans Hu(SH-RD) wrote: > > > > > >Then the next step would be to fake sg-buffer from this straight > > > > >buffer. Revert the above, and modify sgbuf.c to the following: > > > > >- Allocate a large continuous buffer > > > > >- Assign each page in this large buffer > > > > > > > > >If this still works, it's not about vmap, but it just means that > > > > >the physically ordered pages do matter -- implicitly showing that > > > > >the snooping behavior isn't properly turned on / off on the controller. > > > > > > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose. > > > > > > >Not really what I meant. Rather something like below (totally untested). > > > > > > [Hans :] I know what you mean now and complete code like below, but the result is still noise. > > > > >OK, so indeed the vmapped address does seem matter. Interesting. > > >What kind of user access does produce the noise? Does it via aplay mmap mode, too? > > > > >In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct). > > > > [Hans :] As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to uncacheable type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user(). > > > >Well, that's usually no problem regarding that cache coherency. > >At least it hasn't been any issue with Intel and AMD CPUs. > >Does the problem happen with Intel CPU instead of VIA? > > [Hans :] I don't have enough Intel/AMD machine here, up to now, only find one Intel's : HW: mother board is Dell 042P49, HDA controller is 8086:1c20, codec is cx20641. And the same issue is seen on that machine? > > Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used by most music players. > > > > [Hans :] I complete the patch in your last mail like below, not used the vunmap & re-vmap way(it has a little problem). Result is OK now(no noise anymore). > > >What problem did you get with vunmap / vmap? > > [Hans :] if patch like below, even the new vmapped address new_area be assigned to dmab->area, it will not use this time, but next. This time used dmab->area still the last one, but the old one had been vunmap, so Oops reported. That's bad. But the problem isn't as you guessed, I guess. The allocator re-uses the pre-allocated buffer if available, and this makes things inconsistent. I'll cook later. Takashi
diff --git a/sound/core/sgbuf.c b/sound/core/sgbuf.c index 84fffabdd129..643f6cb2048c 100644 --- a/sound/core/sgbuf.c +++ b/sound/core/sgbuf.c @@ -68,7 +68,7 @@ void *snd_malloc_sgbuf_pages(struct device *device, size_t *res_size) { struct snd_sg_buf *sgbuf; -unsigned int i, pages, chunk, maxpages; +unsigned int i, pages, chunk; struct snd_dma_buffer tmpb; struct snd_sg_page *table; struct page **pgtable; @@ -90,38 +90,18 @@ void *snd_malloc_sgbuf_pages(struct device *device, goto _failed; sgbuf->page_table = pgtable; -/* allocate pages */ -maxpages = MAX_ALLOC_PAGES; -while (pages > 0) { -chunk = pages; -/* don't be too eager to take a huge chunk */ -if (chunk > maxpages) -chunk = maxpages; -chunk <<= PAGE_SHIFT; -if (snd_dma_alloc_pages_fallback(SNDRV_DMA_TYPE_DEV, device, - chunk, &tmpb) < 0) { -if (!sgbuf->pages) -goto _failed; -if (!res_size) -goto _failed; -size = sgbuf->pages * PAGE_SIZE; -break; -} -chunk = tmpb.bytes >> PAGE_SHIFT; -for (i = 0; i < chunk; i++) { -table->buf = tmpb.area; -table->addr = tmpb.addr; -if (!i) -table->addr |= chunk; /* mark head */ -table++; -*pgtable++ = virt_to_page(tmpb.area); -tmpb.area += PAGE_SIZE; -tmpb.addr += PAGE_SIZE; -} -sgbuf->pages += chunk; -pages -= chunk; -if (chunk < maxpages) -maxpages = chunk; +if (snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, device, size, &tmpb) < 0) +goto _failed; +chunk = tmpb.bytes >> PAGE_SHIFT; +for (i = 0; i < pages; i++) { +table->buf = tmpb.area; +table->addr = tmpb.addr; +if (!i) +table->addr |= pages; /* mark head */ +table++; +*pgtable++ = virt_to_page(tmpb.area); +tmpb.area += PAGE_SIZE; +tmpb.addr += PAGE_SIZE; +sgbuf->pages += chunk; } sgbuf->size = size;