diff mbox series

[v1,4/5] mtd: rawnand: meson: clear OOB buffer before read

Message ID 20230412061700.1492474-5-AVKrasnov@sberdevices.ru (mailing list archive)
State Superseded
Headers show
Series refactoring and fix for Meson NAND | expand

Commit Message

Arseniy Krasnov April 12, 2023, 6:16 a.m. UTC
This NAND reads only few user's bytes in ECC mode (not full OOB), so
fill OOB buffer with zeroes to not return garbage from previous reads
to user.
Otherwise 'nanddump' utility prints something like this for just erased
page:

...
0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
  OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
  OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
  OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
  OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e

Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
---
 drivers/mtd/nand/raw/meson_nand.c | 5 +++++
 1 file changed, 5 insertions(+)

Comments

Miquel Raynal April 12, 2023, 7:44 a.m. UTC | #1
Hi Arseniy,

AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:

> This NAND reads only few user's bytes in ECC mode (not full OOB), so

"This NAND reads" does not look right, do you mean "Subpage reads do
not retrieve all the OOB bytes,"?

> fill OOB buffer with zeroes to not return garbage from previous reads
> to user.
> Otherwise 'nanddump' utility prints something like this for just erased
> page:
> 
> ...
> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> 
> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> ---
>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> index f84a10238e4d..f2f2472cb511 100644
> --- a/drivers/mtd/nand/raw/meson_nand.c
> +++ b/drivers/mtd/nand/raw/meson_nand.c
> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>  				   int oob_required, int page)
>  {
> +	struct mtd_info *mtd = nand_to_mtd(nand);
>  	u8 *oob_buf = nand->oob_poi;
>  	int ret;
>  
> +	memset(oob_buf, 0, mtd->oobsize);

I'm surprised raw reads do not read the entire OOB?

> +
>  	ret = meson_nfc_read_page_sub(nand, page, 1);
>  	if (ret)
>  		return ret;
> @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
>  	u8 *oob_buf = nand->oob_poi;
>  	int ret, i;
>  
> +	memset(oob_buf, 0, mtd->oobsize);
> +
>  	ret = meson_nfc_read_page_sub(nand, page, 0);
>  	if (ret)
>  		return ret;


Thanks,
Miquèl
Miquel Raynal April 12, 2023, 7:47 a.m. UTC | #2
Hi again,

miquel.raynal@bootlin.com wrote on Wed, 12 Apr 2023 09:44:00 +0200:

> Hi Arseniy,
> 
> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> 
> > This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> 
> "This NAND reads" does not look right, do you mean "Subpage reads do
> not retrieve all the OOB bytes,"?
> 
> > fill OOB buffer with zeroes to not return garbage from previous reads
> > to user.
> > Otherwise 'nanddump' utility prints something like this for just erased
> > page:
> > 
> > ...
> > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > 
> > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > ---
> >  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > index f84a10238e4d..f2f2472cb511 100644
> > --- a/drivers/mtd/nand/raw/meson_nand.c
> > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >  				   int oob_required, int page)
> >  {
> > +	struct mtd_info *mtd = nand_to_mtd(nand);
> >  	u8 *oob_buf = nand->oob_poi;
> >  	int ret;
> >  
> > +	memset(oob_buf, 0, mtd->oobsize);  

Should use 0xff instead of 0x00, that's the default state of a NAND
cell.

And also this memset should be conditioned to 'oob_required' I
guess?

> 
> I'm surprised raw reads do not read the entire OOB?
> 
> > +
> >  	ret = meson_nfc_read_page_sub(nand, page, 1);
> >  	if (ret)
> >  		return ret;
> > @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
> >  	u8 *oob_buf = nand->oob_poi;
> >  	int ret, i;
> >  
> > +	memset(oob_buf, 0, mtd->oobsize);
> > +
> >  	ret = meson_nfc_read_page_sub(nand, page, 0);
> >  	if (ret)
> >  		return ret;  
> 
> 
> Thanks,
> Miquèl


Thanks,
Miquèl
Arseniy Krasnov April 12, 2023, 9:20 a.m. UTC | #3
On 12.04.2023 10:44, Miquel Raynal wrote:
> Hi Arseniy,
> 
> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> 
>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
> 
> "This NAND reads" does not look right, do you mean "Subpage reads do
> not retrieve all the OOB bytes,"?
> 
>> fill OOB buffer with zeroes to not return garbage from previous reads
>> to user.
>> Otherwise 'nanddump' utility prints something like this for just erased
>> page:
>>
>> ...
>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>
>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>> ---
>>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>> index f84a10238e4d..f2f2472cb511 100644
>> --- a/drivers/mtd/nand/raw/meson_nand.c
>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>  				   int oob_required, int page)
>>  {
>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>  	u8 *oob_buf = nand->oob_poi;
>>  	int ret;
>>  
>> +	memset(oob_buf, 0, mtd->oobsize);
> 
> I'm surprised raw reads do not read the entire OOB?

Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
still depends on ECC parameters: for each portion of data covered with ECC code we can
read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.

Thanks, Arseniy

> 
>> +
>>  	ret = meson_nfc_read_page_sub(nand, page, 1);
>>  	if (ret)
>>  		return ret;
>> @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
>>  	u8 *oob_buf = nand->oob_poi;
>>  	int ret, i;
>>  
>> +	memset(oob_buf, 0, mtd->oobsize);
>> +
>>  	ret = meson_nfc_read_page_sub(nand, page, 0);
>>  	if (ret)
>>  		return ret;
> 
> 
> Thanks,
> Miquèl
Miquel Raynal April 12, 2023, 9:36 a.m. UTC | #4
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:

> On 12.04.2023 10:44, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> >   
> >> This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> > 
> > "This NAND reads" does not look right, do you mean "Subpage reads do
> > not retrieve all the OOB bytes,"?
> >   
> >> fill OOB buffer with zeroes to not return garbage from previous reads
> >> to user.
> >> Otherwise 'nanddump' utility prints something like this for just erased
> >> page:
> >>
> >> ...
> >> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>
> >> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >> ---
> >>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>  1 file changed, 5 insertions(+)
> >>
> >> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >> index f84a10238e4d..f2f2472cb511 100644
> >> --- a/drivers/mtd/nand/raw/meson_nand.c
> >> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>  				   int oob_required, int page)
> >>  {
> >> +	struct mtd_info *mtd = nand_to_mtd(nand);
> >>  	u8 *oob_buf = nand->oob_poi;
> >>  	int ret;
> >>  
> >> +	memset(oob_buf, 0, mtd->oobsize);  
> > 
> > I'm surprised raw reads do not read the entire OOB?  
> 
> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> still depends on ECC parameters: for each portion of data covered with ECC code we can
> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.

In all modes, when you read OOB, you should get the full OOB. The fact
that ECC correction is enabled or disabled does not matter. If the NAND
features OOB sections of 64 bytes, you should get the 64 bytes.

What happens sometimes, is that some of the bytes are not protected
against bitflips, but the policy is to return the full buffer.

> 
> Thanks, Arseniy
> 
> >   
> >> +
> >>  	ret = meson_nfc_read_page_sub(nand, page, 1);
> >>  	if (ret)
> >>  		return ret;
> >> @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
> >>  	u8 *oob_buf = nand->oob_poi;
> >>  	int ret, i;
> >>  
> >> +	memset(oob_buf, 0, mtd->oobsize);
> >> +
> >>  	ret = meson_nfc_read_page_sub(nand, page, 0);
> >>  	if (ret)
> >>  		return ret;  
> > 
> > 
> > Thanks,
> > Miquèl  


Thanks,
Miquèl
Arseniy Krasnov April 12, 2023, 10:14 a.m. UTC | #5
On 12.04.2023 12:36, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> 
>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>   
>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so  
>>>
>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>> not retrieve all the OOB bytes,"?
>>>   
>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>> to user.
>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>> page:
>>>>
>>>> ...
>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>
>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>> ---
>>>>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>  1 file changed, 5 insertions(+)
>>>>
>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>> index f84a10238e4d..f2f2472cb511 100644
>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>  				   int oob_required, int page)
>>>>  {
>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>>>  	u8 *oob_buf = nand->oob_poi;
>>>>  	int ret;
>>>>  
>>>> +	memset(oob_buf, 0, mtd->oobsize);  
>>>
>>> I'm surprised raw reads do not read the entire OOB?  
>>
>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
> 
> In all modes, when you read OOB, you should get the full OOB. The fact
> that ECC correction is enabled or disabled does not matter. If the NAND
> features OOB sections of 64 bytes, you should get the 64 bytes.
> 
> What happens sometimes, is that some of the bytes are not protected
> against bitflips, but the policy is to return the full buffer.

Ok, so to clarify case for this NAND controller:
1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
   + ECC codes)?
2) If I have access to only 32 bytes of OOB (in case above), I must report that size
   of OOB is only 32 bytes during initialization?

Thanks, Arseniy

> 
>>
>> Thanks, Arseniy
>>
>>>   
>>>> +
>>>>  	ret = meson_nfc_read_page_sub(nand, page, 1);
>>>>  	if (ret)
>>>>  		return ret;
>>>> @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
>>>>  	u8 *oob_buf = nand->oob_poi;
>>>>  	int ret, i;
>>>>  
>>>> +	memset(oob_buf, 0, mtd->oobsize);
>>>> +
>>>>  	ret = meson_nfc_read_page_sub(nand, page, 0);
>>>>  	if (ret)
>>>>  		return ret;  
>>>
>>>
>>> Thanks,
>>> Miquèl  
> 
> 
> Thanks,
> Miquèl
Liang Yang April 12, 2023, 10:51 a.m. UTC | #6
Hi Arseniy and Miquel,

On 2023/4/12 18:14, Arseniy Krasnov wrote:
> [ EXTERNAL EMAIL ]
> 
> 
> 
> On 12.04.2023 12:36, Miquel Raynal wrote:
>> Hi Arseniy,
>>
>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>
>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>> Hi Arseniy,
>>>>
>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>    
>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>
>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>> not retrieve all the OOB bytes,"?
>>>>    
>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>> to user.
>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>> page:
>>>>>
>>>>> ...
>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>
>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>> ---
>>>>>   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>   1 file changed, 5 insertions(+)
>>>>>
>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>   				   int oob_required, int page)
>>>>>   {
>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>   	u8 *oob_buf = nand->oob_poi;
>>>>>   	int ret;
>>>>>   
>>>>> +	memset(oob_buf, 0, mtd->oobsize);
>>>>
>>>> I'm surprised raw reads do not read the entire OOB?
>>>
>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>
>> In all modes, when you read OOB, you should get the full OOB. The fact
>> that ECC correction is enabled or disabled does not matter. If the NAND
>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>
>> What happens sometimes, is that some of the bytes are not protected
>> against bitflips, but the policy is to return the full buffer.
> 
> Ok, so to clarify case for this NAND controller:
> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>     + ECC codes)?
> 2) If I have access to only 32 bytes of OOB (in case above), I must report that size
>     of OOB is only 32 bytes during initialization?
> 
> Thanks, Arseniy

Yes. it should return all the OOB data. i make a mistake on raw read and 
there is wrong code in meson_nfc_read_page_raw().
	meson_nfc_get_data_oob(nand, buf, oob_buf);
changed to:
	if (oob_required)
		memcpy(oob_buf, buf + mtd->writesize, mtd->oobsize)

for the ECC mode, i define the meson_ooblayout_ops in host driver.

> 
>>
>>>
>>> Thanks, Arseniy
>>>
>>>>    
>>>>> +
>>>>>   	ret = meson_nfc_read_page_sub(nand, page, 1);
>>>>>   	if (ret)
>>>>>   		return ret;
>>>>> @@ -881,6 +884,8 @@ static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
>>>>>   	u8 *oob_buf = nand->oob_poi;
>>>>>   	int ret, i;
>>>>>   
>>>>> +	memset(oob_buf, 0, mtd->oobsize);
>>>>> +
>>>>>   	ret = meson_nfc_read_page_sub(nand, page, 0);
>>>>>   	if (ret)
>>>>>   		return ret;
>>>>
>>>>
>>>> Thanks,
>>>> Miquèl
>>
>>
>> Thanks,
>> Miquèl
>
Liang Yang April 12, 2023, 11:36 a.m. UTC | #7
Hi,

On 2023/4/12 18:51, Liang Yang wrote:
>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c 
>>>>>> b/drivers/mtd/nand/raw/meson_nand.c
>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct 
>>>>>> nand_chip *nand,
>>>>>>   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>                      int oob_required, int page)
>>>>>>   {
>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>       u8 *oob_buf = nand->oob_poi;
>>>>>>       int ret;
>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>
>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>
>>>> Yes! Seems in case of raw access (what i see in this driver) number 
>>>> of OOB bytes read
>>>> still depends on ECC parameters: for each portion of data covered 
>>>> with ECC code we can
>>>> read it's ECC code and "user bytes" from OOB - it is what i see by 
>>>> dumping DMA buffer by
>>>> printk(). For example I'm working with 2K NAND pages, each page has 
>>>> 2 x 1K ECC blocks.
>>>> For each ECC block I have 16 OOB bytes which I can access by 
>>>> read/write. Each 16 bytes
>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I 
>>>> read page in raw mode
>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is 
>>>> reported as 64 bytes.
>>>
>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>
>>> What happens sometimes, is that some of the bytes are not protected
>>> against bitflips, but the policy is to return the full buffer.
>>
>> Ok, so to clarify case for this NAND controller:
>> 1) In both ECC and raw modes i need to return the same raw OOB data 
>> (e.g. user bytes
>>     + ECC codes)?
>> 2) If I have access to only 32 bytes of OOB (in case above), I must 
>> report that size
>>     of OOB is only 32 bytes during initialization?
>>
>> Thanks, Arseniy
> 
> Yes. it should return all the OOB data. i make a mistake on raw read and 
> there is wrong code in meson_nfc_read_page_raw().
>      meson_nfc_get_data_oob(nand, buf, oob_buf);
> changed to:
>      if (oob_required)
>          memcpy(oob_buf, buf + mtd->writesize, mtd->oobsize)

Sorry, please ignore this. the previous code is right.

the controller changes the layout of one page; the physical layout is 
2048 main data + 64 oob data. after writing into NAND page, it is stored
like this: 1024 main data + 2 user bytes + 14 ECC parity bytes + 1024 
main data + 2 user bytes + 14 ECC parity bytes. so that is right we only 
get 4 user bytes and 28 ECC parity bytes, total 32 bytes. that is the 
behavior of the controller that transferring one ECC page(1KB) brings 
back only 2 user bytes.

because layout is changed by controller, so go back to the function. 
meson_nfc_get_data_oob(nand, buf, oob_buf) try to get the right user and 
ecc parity bytes from the right pos. after that, the other oob bytes is 
not reading from NAND flash.
Dmitry Rokosov April 12, 2023, 11:43 a.m. UTC | #8
Hello Liang,

On Wed, Apr 12, 2023 at 07:36:30PM +0800, Liang Yang wrote:
> Hi,
> 
> On 2023/4/12 18:51, Liang Yang wrote:
> > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > @@ -858,9 +858,12 @@ static int
> > > > > > > meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > >                      int oob_required, int page)
> > > > > > >   {
> > > > > > > +    struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > >       u8 *oob_buf = nand->oob_poi;
> > > > > > >       int ret;
> > > > > > > +    memset(oob_buf, 0, mtd->oobsize);
> > > > > > 
> > > > > > I'm surprised raw reads do not read the entire OOB?
> > > > > 
> > > > > Yes! Seems in case of raw access (what i see in this driver)
> > > > > number of OOB bytes read
> > > > > still depends on ECC parameters: for each portion of data
> > > > > covered with ECC code we can
> > > > > read it's ECC code and "user bytes" from OOB - it is what i
> > > > > see by dumping DMA buffer by
> > > > > printk(). For example I'm working with 2K NAND pages, each
> > > > > page has 2 x 1K ECC blocks.
> > > > > For each ECC block I have 16 OOB bytes which I can access by
> > > > > read/write. Each 16 bytes
> > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So
> > > > > when I read page in raw mode
> > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB
> > > > > is reported as 64 bytes.
> > > > 
> > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > 
> > > > What happens sometimes, is that some of the bytes are not protected
> > > > against bitflips, but the policy is to return the full buffer.
> > > 
> > > Ok, so to clarify case for this NAND controller:
> > > 1) In both ECC and raw modes i need to return the same raw OOB data
> > > (e.g. user bytes
> > >     + ECC codes)?
> > > 2) If I have access to only 32 bytes of OOB (in case above), I must
> > > report that size
> > >     of OOB is only 32 bytes during initialization?
> > > 
> > > Thanks, Arseniy
> > 
> > Yes. it should return all the OOB data. i make a mistake on raw read and
> > there is wrong code in meson_nfc_read_page_raw().
> >      meson_nfc_get_data_oob(nand, buf, oob_buf);
> > changed to:
> >      if (oob_required)
> >          memcpy(oob_buf, buf + mtd->writesize, mtd->oobsize)
> 
> Sorry, please ignore this. the previous code is right.
> 
> the controller changes the layout of one page; the physical layout is 2048
> main data + 64 oob data. after writing into NAND page, it is stored
> like this: 1024 main data + 2 user bytes + 14 ECC parity bytes + 1024 main
> data + 2 user bytes + 14 ECC parity bytes. so that is right we only get 4
> user bytes and 28 ECC parity bytes, total 32 bytes. that is the behavior of
> the controller that transferring one ECC page(1KB) brings back only 2 user
> bytes.
> 
> because layout is changed by controller, so go back to the function.
> meson_nfc_get_data_oob(nand, buf, oob_buf) try to get the right user and ecc
> parity bytes from the right pos. after that, the other oob bytes is not
> reading from NAND flash.

I have always been under the impression that NAND OOB layout falls under the
responsibility of the flash driver. Is this specific to the Amlogic NAND,
and does it map the flash layout to the internal controller layout?
For example, different OOB layouts exist between Macronix and ESMT.

Apologies for any confusion, and thank you in advance for any help in
clarifying this matter.
Arseniy Krasnov April 12, 2023, 11:47 a.m. UTC | #9
On 12.04.2023 14:43, Dmitry Rokosov wrote:
> Hello Liang,
> 
> On Wed, Apr 12, 2023 at 07:36:30PM +0800, Liang Yang wrote:
>> Hi,
>>
>> On 2023/4/12 18:51, Liang Yang wrote:
>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> @@ -858,9 +858,12 @@ static int
>>>>>>>> meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>                      int oob_required, int page)
>>>>>>>>   {
>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>       u8 *oob_buf = nand->oob_poi;
>>>>>>>>       int ret;
>>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>
>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>
>>>>>> Yes! Seems in case of raw access (what i see in this driver)
>>>>>> number of OOB bytes read
>>>>>> still depends on ECC parameters: for each portion of data
>>>>>> covered with ECC code we can
>>>>>> read it's ECC code and "user bytes" from OOB - it is what i
>>>>>> see by dumping DMA buffer by
>>>>>> printk(). For example I'm working with 2K NAND pages, each
>>>>>> page has 2 x 1K ECC blocks.
>>>>>> For each ECC block I have 16 OOB bytes which I can access by
>>>>>> read/write. Each 16 bytes
>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So
>>>>>> when I read page in raw mode
>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB
>>>>>> is reported as 64 bytes.
>>>>>
>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>
>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>> against bitflips, but the policy is to return the full buffer.
>>>>
>>>> Ok, so to clarify case for this NAND controller:
>>>> 1) In both ECC and raw modes i need to return the same raw OOB data
>>>> (e.g. user bytes
>>>>     + ECC codes)?
>>>> 2) If I have access to only 32 bytes of OOB (in case above), I must
>>>> report that size
>>>>     of OOB is only 32 bytes during initialization?
>>>>
>>>> Thanks, Arseniy
>>>
>>> Yes. it should return all the OOB data. i make a mistake on raw read and
>>> there is wrong code in meson_nfc_read_page_raw().
>>>      meson_nfc_get_data_oob(nand, buf, oob_buf);
>>> changed to:
>>>      if (oob_required)
>>>          memcpy(oob_buf, buf + mtd->writesize, mtd->oobsize)
>>
>> Sorry, please ignore this. the previous code is right.
>>
>> the controller changes the layout of one page; the physical layout is 2048
>> main data + 64 oob data. after writing into NAND page, it is stored
>> like this: 1024 main data + 2 user bytes + 14 ECC parity bytes + 1024 main
>> data + 2 user bytes + 14 ECC parity bytes. so that is right we only get 4
>> user bytes and 28 ECC parity bytes, total 32 bytes. that is the behavior of
>> the controller that transferring one ECC page(1KB) brings back only 2 user
>> bytes.

So i perform the following test, using file with the following pattern:
[ 1024 x 0x11 ][ 1024 x 0x22 ][ 16 x 0x33 ][ 16 x 0x44 ][ 16 x 0x55 ][ 16 x 0x66 ]

Total 2112 bytes (2048 data + 64 oob)

I write it using 'nandwrite' tool with flags '--oob' and '--noecc' - e.g. raw mode.
meson_nfc_set_data_oob() places to DMA buffer only parts with 0x11, 0x22, 0x33, 0x44.
After write data bytes will be 0x11 and 0x22, first 32 bytes of OOB will be 0x33 and 0x44.
Parts with 0x55 and 0x66 will be obviously missed.

So it is Amlogic limitation for 32 bytes of OOB (in case of 1K ECC as You mentioned above)?

And there is no way to access to the whole 64 bytes of OOB even in raw mode (and therefore in
ECC mode)?

Thanks, Arseniy

>>
>> because layout is changed by controller, so go back to the function.
>> meson_nfc_get_data_oob(nand, buf, oob_buf) try to get the right user and ecc
>> parity bytes from the right pos. after that, the other oob bytes is not
>> reading from NAND flash.
> 
> I have always been under the impression that NAND OOB layout falls under the
> responsibility of the flash driver. Is this specific to the Amlogic NAND,
> and does it map the flash layout to the internal controller layout?
> For example, different OOB layouts exist between Macronix and ESMT.
> 
> Apologies for any confusion, and thank you in advance for any help in
> clarifying this matter.
>
Miquel Raynal April 12, 2023, 12:18 p.m. UTC | #10
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:

> On 12.04.2023 12:36, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> >   
> >> On 12.04.2023 10:44, Miquel Raynal wrote:  
> >>> Hi Arseniy,
> >>>
> >>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> >>>     
> >>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so    
> >>>
> >>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>> not retrieve all the OOB bytes,"?
> >>>     
> >>>> fill OOB buffer with zeroes to not return garbage from previous reads
> >>>> to user.
> >>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>> page:
> >>>>
> >>>> ...
> >>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>
> >>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>> ---
> >>>>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>  1 file changed, 5 insertions(+)
> >>>>
> >>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>> index f84a10238e4d..f2f2472cb511 100644
> >>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>  				   int oob_required, int page)
> >>>>  {
> >>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>  	u8 *oob_buf = nand->oob_poi;
> >>>>  	int ret;
> >>>>  
> >>>> +	memset(oob_buf, 0, mtd->oobsize);    
> >>>
> >>> I'm surprised raw reads do not read the entire OOB?    
> >>
> >> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> > 
> > In all modes, when you read OOB, you should get the full OOB. The fact
> > that ECC correction is enabled or disabled does not matter. If the NAND
> > features OOB sections of 64 bytes, you should get the 64 bytes.
> > 
> > What happens sometimes, is that some of the bytes are not protected
> > against bitflips, but the policy is to return the full buffer.  
> 
> Ok, so to clarify case for this NAND controller:
> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>    + ECC codes)?

Well, you need to cover the same amount of data, yes. But in the ECC
case the data won't be raw (at least not all of it).

> 2) If I have access to only 32 bytes of OOB (in case above), I must report that size
>    of OOB is only 32 bytes during initialization?

I believe it's just an implementation error in the controller driver.
This controller can be used on NAND chips with other geometries, please
don't play with it.

Thanks,
Miquèl
Arseniy Krasnov April 12, 2023, 12:22 p.m. UTC | #11
On 12.04.2023 15:18, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
> 
>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>   
>>>> On 12.04.2023 10:44, Miquel Raynal wrote:  
>>>>> Hi Arseniy,
>>>>>
>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>     
>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so    
>>>>>
>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>> not retrieve all the OOB bytes,"?
>>>>>     
>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>> to user.
>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>> page:
>>>>>>
>>>>>> ...
>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>
>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>> ---
>>>>>>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>  1 file changed, 5 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>  				   int oob_required, int page)
>>>>>>  {
>>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>  	u8 *oob_buf = nand->oob_poi;
>>>>>>  	int ret;
>>>>>>  
>>>>>> +	memset(oob_buf, 0, mtd->oobsize);    
>>>>>
>>>>> I'm surprised raw reads do not read the entire OOB?    
>>>>
>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
>>>
>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>
>>> What happens sometimes, is that some of the bytes are not protected
>>> against bitflips, but the policy is to return the full buffer.  
>>
>> Ok, so to clarify case for this NAND controller:
>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>    + ECC codes)?
> 
> Well, you need to cover the same amount of data, yes. But in the ECC
> case the data won't be raw (at least not all of it).

So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?

Thanks, Arseniy

> 
>> 2) If I have access to only 32 bytes of OOB (in case above), I must report that size
>>    of OOB is only 32 bytes during initialization?
> 
> I believe it's just an implementation error in the controller driver.
> This controller can be used on NAND chips with other geometries, please
> don't play with it.
> 
> Thanks,
> Miquèl
Liang Yang April 12, 2023, 12:28 p.m. UTC | #12
Hi Dmitry,

On 2023/4/12 19:43, Dmitry Rokosov wrote:
> [ EXTERNAL EMAIL ]
> 
> Hello Liang,
> 
> On Wed, Apr 12, 2023 at 07:36:30PM +0800, Liang Yang wrote:
>> Hi,
>>
>> On 2023/4/12 18:51, Liang Yang wrote:
>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> @@ -858,9 +858,12 @@ static int
>>>>>>>> meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>                       int oob_required, int page)
>>>>>>>>    {
>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>        int ret;
>>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>
>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>
>>>>>> Yes! Seems in case of raw access (what i see in this driver)
>>>>>> number of OOB bytes read
>>>>>> still depends on ECC parameters: for each portion of data
>>>>>> covered with ECC code we can
>>>>>> read it's ECC code and "user bytes" from OOB - it is what i
>>>>>> see by dumping DMA buffer by
>>>>>> printk(). For example I'm working with 2K NAND pages, each
>>>>>> page has 2 x 1K ECC blocks.
>>>>>> For each ECC block I have 16 OOB bytes which I can access by
>>>>>> read/write. Each 16 bytes
>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So
>>>>>> when I read page in raw mode
>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB
>>>>>> is reported as 64 bytes.
>>>>>
>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>
>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>> against bitflips, but the policy is to return the full buffer.
>>>>
>>>> Ok, so to clarify case for this NAND controller:
>>>> 1) In both ECC and raw modes i need to return the same raw OOB data
>>>> (e.g. user bytes
>>>>      + ECC codes)?
>>>> 2) If I have access to only 32 bytes of OOB (in case above), I must
>>>> report that size
>>>>      of OOB is only 32 bytes during initialization?
>>>>
>>>> Thanks, Arseniy
>>>
>>> Yes. it should return all the OOB data. i make a mistake on raw read and
>>> there is wrong code in meson_nfc_read_page_raw().
>>>       meson_nfc_get_data_oob(nand, buf, oob_buf);
>>> changed to:
>>>       if (oob_required)
>>>           memcpy(oob_buf, buf + mtd->writesize, mtd->oobsize)
>>
>> Sorry, please ignore this. the previous code is right.
>>
>> the controller changes the layout of one page; the physical layout is 2048
>> main data + 64 oob data. after writing into NAND page, it is stored
>> like this: 1024 main data + 2 user bytes + 14 ECC parity bytes + 1024 main
>> data + 2 user bytes + 14 ECC parity bytes. so that is right we only get 4
>> user bytes and 28 ECC parity bytes, total 32 bytes. that is the behavior of
>> the controller that transferring one ECC page(1KB) brings back only 2 user
>> bytes.
>>
>> because layout is changed by controller, so go back to the function.
>> meson_nfc_get_data_oob(nand, buf, oob_buf) try to get the right user and ecc
>> parity bytes from the right pos. after that, the other oob bytes is not
>> reading from NAND flash.
> 
> I have always been under the impression that NAND OOB layout falls under the
> responsibility of the flash driver. Is this specific to the Amlogic NAND,
> and does it map the flash layout to the internal controller layout?
> For example, different OOB layouts exist between Macronix and ESMT.
> 
> Apologies for any confusion, and thank you in advance for any help in
> clarifying this matter.
> 

This is the behavior of Amlogic NAND controller, not related to NAND 
device.

this design of Amlogic NAND controller is reducing the fifo(sram) size 
inside the controller.

To the NAND device with <2KB(main) + 64B(oob)> page, usually if the 
controller place user bytes and ECC parity bytes after 2048(column 
address). to finish the ECC calculation and checking, the controller has 
to read the data and its ECC parity bytes. if they are not placed 
continuously, the controllers have to read the whole page data including 
the oob data together or send random read command with column address. 
so consider that the controller needs a fifo of at least (2048 + 64) 
bytes inside. if the NAND device has bigger size of NAND page, the size 
of the fifo should be bigger.
in Amlogic NAND controller, after changing the data layout of the NAND 
page, it just needs <1KB + 16B> fifo size. the controller firstly sends 
only <1024 + 16> clocks and get 1st ECC page data and its ECC parity 
bytes, and then it can start ECC calculation and checking. after that or 
at the same time, loop next ECC page data and its ECC parity.
Miquel Raynal April 12, 2023, 12:57 p.m. UTC | #13
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:

> On 12.04.2023 15:18, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
> >   
> >> On 12.04.2023 12:36, Miquel Raynal wrote:  
> >>> Hi Arseniy,
> >>>
> >>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> >>>     
> >>>> On 12.04.2023 10:44, Miquel Raynal wrote:    
> >>>>> Hi Arseniy,
> >>>>>
> >>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> >>>>>       
> >>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so      
> >>>>>
> >>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>>>> not retrieve all the OOB bytes,"?
> >>>>>       
> >>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
> >>>>>> to user.
> >>>>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>>>> page:
> >>>>>>
> >>>>>> ...
> >>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>>>   OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>>>   OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>>>   OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>>>   OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>>>
> >>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>>>> ---
> >>>>>>  drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>>>  1 file changed, 5 insertions(+)
> >>>>>>
> >>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>> index f84a10238e4d..f2f2472cb511 100644
> >>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>>>  static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>>>  				   int oob_required, int page)
> >>>>>>  {
> >>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>>>  	u8 *oob_buf = nand->oob_poi;
> >>>>>>  	int ret;
> >>>>>>  
> >>>>>> +	memset(oob_buf, 0, mtd->oobsize);      
> >>>>>
> >>>>> I'm surprised raw reads do not read the entire OOB?      
> >>>>
> >>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
> >>>
> >>> In all modes, when you read OOB, you should get the full OOB. The fact
> >>> that ECC correction is enabled or disabled does not matter. If the NAND
> >>> features OOB sections of 64 bytes, you should get the 64 bytes.
> >>>
> >>> What happens sometimes, is that some of the bytes are not protected
> >>> against bitflips, but the policy is to return the full buffer.    
> >>
> >> Ok, so to clarify case for this NAND controller:
> >> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> >>    + ECC codes)?  
> > 
> > Well, you need to cover the same amount of data, yes. But in the ECC
> > case the data won't be raw (at least not all of it).  
> 
> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?

If the page are 2k+64B you should read 2k+64B when OOB are requested.

If the controller only returns 2k+32B, then perform a random read to
just move the read pointer to mtd->size + mtd->oobsize - 32 and
retrieve the missing 32 bytes?

This applies to the two modes, the only difference is:
- with correction (commonly named "ECC mode"): the user bytes and ECC
  bytes should be fixed if there are any bitflips
- without correction (commonly referred as "raw mode"): no correction
  applies, if there are bitflips, give them

Please mind the raw mode can be slow, it's meant for debugging and
testing, mainly. Page reads however should be fast, so if just moving
the column pointer works, then do it, otherwise we'll consider
returning FFs.

Thanks,
Miquèl
Liang Yang April 12, 2023, 2:04 p.m. UTC | #14
Hi Miquel and Arseniy,

On 2023/4/12 20:57, Miquel Raynal wrote:
> [ EXTERNAL EMAIL ]
> 
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> 
>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>    
>>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>> Hi Arseniy,
>>>>>
>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>      
>>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>> Hi Arseniy,
>>>>>>>
>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>        
>>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>
>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>        
>>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>>>> to user.
>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>> page:
>>>>>>>>
>>>>>>>> ...
>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>
>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>> ---
>>>>>>>>   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>   1 file changed, 5 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>   				   int oob_required, int page)
>>>>>>>>   {
>>>>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>   	u8 *oob_buf = nand->oob_poi;
>>>>>>>>   	int ret;
>>>>>>>>   
>>>>>>>> +	memset(oob_buf, 0, mtd->oobsize);
>>>>>>>
>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>
>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>>>>
>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>
>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>> against bitflips, but the policy is to return the full buffer.
>>>>
>>>> Ok, so to clarify case for this NAND controller:
>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>     + ECC codes)?
>>>
>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>> case the data won't be raw (at least not all of it).
>>
>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
> 
> If the page are 2k+64B you should read 2k+64B when OOB are requested.
> 
> If the controller only returns 2k+32B, then perform a random read to
> just move the read pointer to mtd->size + mtd->oobsize - 32 and
> retrieve the missing 32 bytes?

1) raw read can read out the whole page data 2k+64B, decided by the len 
in the controller raw read command:
	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
after that, the missing oob bytes(not used) can be copied from 
meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() 
is like this if need.
	{
		......
		meson_nfc_read_page_sub(nand, page, 1);
		meson_nfc_get_data_oob(nand, buf, oob_buf);
		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize 
- oob_len);

	}
2) In ECC mode, the controller can't bring back the missing OOB bytes. 
it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.

> 
> This applies to the two modes, the only difference is:
> - with correction (commonly named "ECC mode"): the user bytes and ECC
>    bytes should be fixed if there are any bitflips
> - without correction (commonly referred as "raw mode"): no correction
>    applies, if there are bitflips, give them
> 
> Please mind the raw mode can be slow, it's meant for debugging and
> testing, mainly. Page reads however should be fast, so if just moving
> the column pointer works, then do it, otherwise we'll consider
> returning FFs.
> 
> Thanks,
> Miquèl
>
Miquel Raynal April 12, 2023, 2:32 p.m. UTC | #15
Hello,

liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:

> Hi Miquel and Arseniy,
> 
> On 2023/4/12 20:57, Miquel Raynal wrote:
> > [ EXTERNAL EMAIL ]
> > 
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> >   
> >> On 12.04.2023 15:18, Miquel Raynal wrote:  
> >>> Hi Arseniy,
> >>>
> >>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:  
> >>>    >>>> On 12.04.2023 12:36, Miquel Raynal wrote:  
> >>>>> Hi Arseniy,
> >>>>>
> >>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:  
> >>>>>      >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:  
> >>>>>>> Hi Arseniy,
> >>>>>>>
> >>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:  
> >>>>>>>        >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> >>>>>>>
> >>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>>>>>> not retrieve all the OOB bytes,"?  
> >>>>>>>        >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads  
> >>>>>>>> to user.
> >>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>>>>>> page:
> >>>>>>>>
> >>>>>>>> ...
> >>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>>>>>    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>>>>>    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>>>>>    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>>>>>    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>>>>>
> >>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>>>>>> ---
> >>>>>>>>   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>>>>>   1 file changed, 5 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>> index f84a10238e4d..f2f2472cb511 100644
> >>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>>>>>   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>>>>>   				   int oob_required, int page)
> >>>>>>>>   {
> >>>>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>>>>>   	u8 *oob_buf = nand->oob_poi;
> >>>>>>>>   	int ret;  
> >>>>>>>>   >>>>>>>> +	memset(oob_buf, 0, mtd->oobsize);  
> >>>>>>>
> >>>>>>> I'm surprised raw reads do not read the entire OOB?  
> >>>>>>
> >>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> >>>>>
> >>>>> In all modes, when you read OOB, you should get the full OOB. The fact
> >>>>> that ECC correction is enabled or disabled does not matter. If the NAND
> >>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
> >>>>>
> >>>>> What happens sometimes, is that some of the bytes are not protected
> >>>>> against bitflips, but the policy is to return the full buffer.  
> >>>>
> >>>> Ok, so to clarify case for this NAND controller:
> >>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> >>>>     + ECC codes)?  
> >>>
> >>> Well, you need to cover the same amount of data, yes. But in the ECC
> >>> case the data won't be raw (at least not all of it).  
> >>
> >> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> >> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> >> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
> > 
> > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > 
> > If the controller only returns 2k+32B, then perform a random read to
> > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > retrieve the missing 32 bytes?  
> 
> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
> 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
> 	{
> 		......
> 		meson_nfc_read_page_sub(nand, page, 1);
> 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
> 
> 	}
> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.

And then (if oob_required) you can bring the missing bytes with
something along:
nand_change_read_column_op(chip, mtd->writesize + oob_len,
			   oob_buf + oob_len,
			   mtd->oobsize - oob_len,
			   false);
Should not be a huge performance hit.

> 
> > 
> > This applies to the two modes, the only difference is:
> > - with correction (commonly named "ECC mode"): the user bytes and ECC
> >    bytes should be fixed if there are any bitflips
> > - without correction (commonly referred as "raw mode"): no correction
> >    applies, if there are bitflips, give them
> > 
> > Please mind the raw mode can be slow, it's meant for debugging and
> > testing, mainly. Page reads however should be fast, so if just moving
> > the column pointer works, then do it, otherwise we'll consider
> > returning FFs.
> > 
> > Thanks,
> > Miquèl
> >   
> 


Thanks,
Miquèl
Dmitry Rokosov April 12, 2023, 7:15 p.m. UTC | #16
On Wed, Apr 12, 2023 at 10:04:28PM +0800, Liang Yang wrote:
> Hi Miquel and Arseniy,
> 
> On 2023/4/12 20:57, Miquel Raynal wrote:
> > [ EXTERNAL EMAIL ]
> > 
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> > 
> > > On 12.04.2023 15:18, Miquel Raynal wrote:
> > > > Hi Arseniy,
> > > > 
> > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
> > > > > On 12.04.2023 12:36, Miquel Raynal wrote:
> > > > > > Hi Arseniy,
> > > > > > 
> > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> > > > > > > On 12.04.2023 10:44, Miquel Raynal wrote:
> > > > > > > > Hi Arseniy,
> > > > > > > > 
> > > > > > > > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> > > > > > > > > This NAND reads only few user's bytes in ECC mode (not full OOB), so
> > > > > > > > 
> > > > > > > > "This NAND reads" does not look right, do you mean "Subpage reads do
> > > > > > > > not retrieve all the OOB bytes,"?
> > > > > > > > > fill OOB buffer with zeroes to not return garbage from previous reads
> > > > > > > > > to user.
> > > > > > > > > Otherwise 'nanddump' utility prints something like this for just erased
> > > > > > > > > page:
> > > > > > > > > 
> > > > > > > > > ...
> > > > > > > > > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > > > > >    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> > > > > > > > >    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> > > > > > > > >    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> > > > > > > > >    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > > > > > > > > ---
> > > > > > > > >   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> > > > > > > > >   1 file changed, 5 insertions(+)
> > > > > > > > > 
> > > > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > > > >   				   int oob_required, int page)
> > > > > > > > >   {
> > > > > > > > > +	struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > > > >   	u8 *oob_buf = nand->oob_poi;
> > > > > > > > >   	int ret;
> > > > > > > > > +	memset(oob_buf, 0, mtd->oobsize);
> > > > > > > > 
> > > > > > > > I'm surprised raw reads do not read the entire OOB?
> > > > > > > 
> > > > > > > Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> > > > > > > still depends on ECC parameters: for each portion of data covered with ECC code we can
> > > > > > > read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> > > > > > > printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> > > > > > > For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> > > > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> > > > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
> > > > > > 
> > > > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > > > 
> > > > > > What happens sometimes, is that some of the bytes are not protected
> > > > > > against bitflips, but the policy is to return the full buffer.
> > > > > 
> > > > > Ok, so to clarify case for this NAND controller:
> > > > > 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> > > > >     + ECC codes)?
> > > > 
> > > > Well, you need to cover the same amount of data, yes. But in the ECC
> > > > case the data won't be raw (at least not all of it).
> > > 
> > > So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> > > OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> > > behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
> > 
> > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > 
> > If the controller only returns 2k+32B, then perform a random read to
> > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > retrieve the missing 32 bytes?
> 
> 1) raw read can read out the whole page data 2k+64B, decided by the len in
> the controller raw read command:
> 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> after that, the missing oob bytes(not used) can be copied from
> meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is
> like this if need.
> 	{
> 		......
> 		meson_nfc_read_page_sub(nand, page, 1);
> 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize -
> oob_len);
> 
> 	}
> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it
> can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> 

How does the Meson controller know the actual NAND flash layout when the
OOB is split into protected and unprotected areas, such as Free and ECC
areas? If the controller has a static OOB layout, where is the mapping
located?

> > 
> > This applies to the two modes, the only difference is:
> > - with correction (commonly named "ECC mode"): the user bytes and ECC
> >    bytes should be fixed if there are any bitflips
> > - without correction (commonly referred as "raw mode"): no correction
> >    applies, if there are bitflips, give them
> > 
> > Please mind the raw mode can be slow, it's meant for debugging and
> > testing, mainly. Page reads however should be fast, so if just moving
> > the column pointer works, then do it, otherwise we'll consider
> > returning FFs.
Miquel Raynal April 12, 2023, 8:56 p.m. UTC | #17
Hi Dmitry,

ddrokosov@sberdevices.ru wrote on Wed, 12 Apr 2023 22:15:48 +0300:

> On Wed, Apr 12, 2023 at 10:04:28PM +0800, Liang Yang wrote:
> > Hi Miquel and Arseniy,
> > 
> > On 2023/4/12 20:57, Miquel Raynal wrote:  
> > > [ EXTERNAL EMAIL ]
> > > 
> > > Hi Arseniy,
> > > 
> > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> > >   
> > > > On 12.04.2023 15:18, Miquel Raynal wrote:  
> > > > > Hi Arseniy,
> > > > > 
> > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:  
> > > > > > On 12.04.2023 12:36, Miquel Raynal wrote:  
> > > > > > > Hi Arseniy,
> > > > > > > 
> > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:  
> > > > > > > > On 12.04.2023 10:44, Miquel Raynal wrote:  
> > > > > > > > > Hi Arseniy,
> > > > > > > > > 
> > > > > > > > > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:  
> > > > > > > > > > This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> > > > > > > > > 
> > > > > > > > > "This NAND reads" does not look right, do you mean "Subpage reads do
> > > > > > > > > not retrieve all the OOB bytes,"?  
> > > > > > > > > > fill OOB buffer with zeroes to not return garbage from previous reads
> > > > > > > > > > to user.
> > > > > > > > > > Otherwise 'nanddump' utility prints something like this for just erased
> > > > > > > > > > page:
> > > > > > > > > > 
> > > > > > > > > > ...
> > > > > > > > > > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > > > > > >    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> > > > > > > > > >    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> > > > > > > > > >    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> > > > > > > > > >    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > > > > > > > > > ---
> > > > > > > > > >   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> > > > > > > > > >   1 file changed, 5 insertions(+)
> > > > > > > > > > 
> > > > > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > > > > >   				   int oob_required, int page)
> > > > > > > > > >   {
> > > > > > > > > > +	struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > > > > >   	u8 *oob_buf = nand->oob_poi;
> > > > > > > > > >   	int ret;
> > > > > > > > > > +	memset(oob_buf, 0, mtd->oobsize);  
> > > > > > > > > 
> > > > > > > > > I'm surprised raw reads do not read the entire OOB?  
> > > > > > > > 
> > > > > > > > Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> > > > > > > > still depends on ECC parameters: for each portion of data covered with ECC code we can
> > > > > > > > read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> > > > > > > > printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> > > > > > > > For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> > > > > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> > > > > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> > > > > > > 
> > > > > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > > > > 
> > > > > > > What happens sometimes, is that some of the bytes are not protected
> > > > > > > against bitflips, but the policy is to return the full buffer.  
> > > > > > 
> > > > > > Ok, so to clarify case for this NAND controller:
> > > > > > 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> > > > > >     + ECC codes)?  
> > > > > 
> > > > > Well, you need to cover the same amount of data, yes. But in the ECC
> > > > > case the data won't be raw (at least not all of it).  
> > > > 
> > > > So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> > > > OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> > > > behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
> > > 
> > > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > > 
> > > If the controller only returns 2k+32B, then perform a random read to
> > > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > > retrieve the missing 32 bytes?  
> > 
> > 1) raw read can read out the whole page data 2k+64B, decided by the len in
> > the controller raw read command:
> > 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> > after that, the missing oob bytes(not used) can be copied from
> > meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is
> > like this if need.
> > 	{
> > 		......
> > 		meson_nfc_read_page_sub(nand, page, 1);
> > 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> > 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> > 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize -
> > oob_len);
> > 
> > 	}
> > 2) In ECC mode, the controller can't bring back the missing OOB bytes. it
> > can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> >   
> 
> How does the Meson controller know the actual NAND flash layout when the
> OOB is split into protected and unprotected areas, such as Free and ECC
> areas? If the controller has a static OOB layout, where is the mapping
> located?

It's usually a set of values hardcoded in the driver. It's a per
geometry set.

> 
> > > 
> > > This applies to the two modes, the only difference is:
> > > - with correction (commonly named "ECC mode"): the user bytes and ECC
> > >    bytes should be fixed if there are any bitflips
> > > - without correction (commonly referred as "raw mode"): no correction
> > >    applies, if there are bitflips, give them
> > > 
> > > Please mind the raw mode can be slow, it's meant for debugging and
> > > testing, mainly. Page reads however should be fast, so if just moving
> > > the column pointer works, then do it, otherwise we'll consider
> > > returning FFs.  
> 


Thanks,
Miquèl
Liang Yang April 13, 2023, 5:32 a.m. UTC | #18
Hi Miquel,

On 2023/4/12 22:32, Miquel Raynal wrote:
> [ EXTERNAL EMAIL ]
> 
> Hello,
> 
> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
> 
>> Hi Miquel and Arseniy,
>>
>> On 2023/4/12 20:57, Miquel Raynal wrote:
>>> [ EXTERNAL EMAIL ]
>>>
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
>>>    
>>>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>>>> Hi Arseniy,
>>>>>
>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>>>> Hi Arseniy,
>>>>>>>
>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>>>> Hi Arseniy,
>>>>>>>>>
>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>
>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>>>>>> to user.
>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>> page:
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>> ---
>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>    				   int oob_required, int page)
>>>>>>>>>>    {
>>>>>>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>    	u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>    	int ret;
>>>>>>>>>>    >>>>>>>> +	memset(oob_buf, 0, mtd->oobsize);
>>>>>>>>>
>>>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>>>
>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>>>>>>
>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>
>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>> against bitflips, but the policy is to return the full buffer.
>>>>>>
>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>      + ECC codes)?
>>>>>
>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>> case the data won't be raw (at least not all of it).
>>>>
>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
>>>
>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>
>>> If the controller only returns 2k+32B, then perform a random read to
>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>> retrieve the missing 32 bytes?
>>
>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>> 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>> 	{
>> 		......
>> 		meson_nfc_read_page_sub(nand, page, 1);
>> 		meson_nfc_get_data_oob(nand, buf, oob_buf);
>> 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>> 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>
>> 	}
>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> 
> And then (if oob_required) you can bring the missing bytes with
> something along:
> nand_change_read_column_op(chip, mtd->writesize + oob_len,
> 			   oob_buf + oob_len,
> 			   mtd->oobsize - oob_len,
> 			   false);
> Should not be a huge performance hit.

After finishing ECC mode reading, the column address internal in NAND 
device should be the right pos; it doesn't need to change the column 
again. so adding controller raw read for the missing bytes after ECC 
reading may works.

> 
>>
>>>
>>> This applies to the two modes, the only difference is:
>>> - with correction (commonly named "ECC mode"): the user bytes and ECC
>>>     bytes should be fixed if there are any bitflips
>>> - without correction (commonly referred as "raw mode"): no correction
>>>     applies, if there are bitflips, give them
>>>
>>> Please mind the raw mode can be slow, it's meant for debugging and
>>> testing, mainly. Page reads however should be fast, so if just moving
>>> the column pointer works, then do it, otherwise we'll consider
>>> returning FFs.
>>>
>>> Thanks,
>>> Miquèl
>>>    
>>
> 
> 
> Thanks,
> Miquèl
>
Liang Yang April 13, 2023, 6:11 a.m. UTC | #19
On 2023/4/13 13:32, Liang Yang wrote:
> Hi Miquel,
> 
> On 2023/4/12 22:32, Miquel Raynal wrote:
>> [ EXTERNAL EMAIL ]
>>
>> Hello,
>>
>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>
>>> Hi Miquel and Arseniy,
>>>
>>> On 2023/4/12 20:57, Miquel Raynal wrote:
>>>> [ EXTERNAL EMAIL ]
>>>>
>>>> Hi Arseniy,
>>>>
>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>>>>> Hi Arseniy,
>>>>>>>>
>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>
>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 
>>>>>>>>>> +0300:
>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC 
>>>>>>>>>> mode (not full OOB), so
>>>>>>>>>>
>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage 
>>>>>>>>>> reads do
>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return 
>>>>>>>>>> garbage from previous reads
>>>>>>>>>>> to user.
>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for 
>>>>>>>>>>> just erased
>>>>>>>>>>> page:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>> ---
>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c 
>>>>>>>>>>> b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>> @@ -858,9 +858,12 @@ static int 
>>>>>>>>>>> meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, 
>>>>>>>>>>> u8 *buf,
>>>>>>>>>>>                       int oob_required, int page)
>>>>>>>>>>>    {
>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>        int ret;
>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>>>>
>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>>>>
>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) 
>>>>>>>>> number of OOB bytes read
>>>>>>>>> still depends on ECC parameters: for each portion of data 
>>>>>>>>> covered with ECC code we can
>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see 
>>>>>>>>> by dumping DMA buffer by
>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page 
>>>>>>>>> has 2 x 1K ECC blocks.
>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by 
>>>>>>>>> read/write. Each 16 bytes
>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when 
>>>>>>>>> I read page in raw mode
>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is 
>>>>>>>>> reported as 64 bytes.
>>>>>>>>
>>>>>>>> In all modes, when you read OOB, you should get the full OOB. 
>>>>>>>> The fact
>>>>>>>> that ECC correction is enabled or disabled does not matter. If 
>>>>>>>> the NAND
>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>
>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>> against bitflips, but the policy is to return the full buffer.
>>>>>>>
>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB 
>>>>>>> data (e.g. user bytes
>>>>>>>      + ECC codes)?
>>>>>>
>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>> case the data won't be raw (at least not all of it).
>>>>>
>>>>> So "same amount of data", in ECC mode current implementation 
>>>>> returns only user OOB bytes (e.g.
>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC 
>>>>> codes. IIUC correct
>>>>> behaviour is to always return user bytes + ECC codes as OOB data 
>>>>> even in ECC mode ?
>>>>
>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>
>>>> If the controller only returns 2k+32B, then perform a random read to
>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>> retrieve the missing 32 bytes?
>>>
>>> 1) raw read can read out the whole page data 2k+64B, decided by the 
>>> len in the controller raw read command:
>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>> after that, the missing oob bytes(not used) can be copied from 
>>> meson_chip->data_buf. so the implementation of 
>>> meson_nfc_read_page_raw() is like this if need.
>>>     {
>>>         ......
>>>         meson_nfc_read_page_sub(nand, page, 1);
>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, 
>>> mtd->oobsize - oob_len);
>>>
>>>     }
>>> 2) In ECC mode, the controller can't bring back the missing OOB 
>>> bytes. it can read out the user bytes and ecc bytes per 
>>> meson_ooblayout_ops define.
>>
>> And then (if oob_required) you can bring the missing bytes with
>> something along:
>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>                oob_buf + oob_len,
>>                mtd->oobsize - oob_len,
>>                false);
>> Should not be a huge performance hit.
> 
> After finishing ECC mode reading, the column address internal in NAND 
> device should be the right pos; it doesn't need to change the column 
> again. so adding controller raw read for the missing bytes after ECC 
> reading may works.
> 
use raw read for the missing bytes, but they are not protected by host 
ECC. to the NAND type of storage, is it ok or missing bytes better to be 
filled with 0xff?

>>
>>>
>>>>
>>>> This applies to the two modes, the only difference is:
>>>> - with correction (commonly named "ECC mode"): the user bytes and ECC
>>>>     bytes should be fixed if there are any bitflips
>>>> - without correction (commonly referred as "raw mode"): no correction
>>>>     applies, if there are bitflips, give them
>>>>
>>>> Please mind the raw mode can be slow, it's meant for debugging and
>>>> testing, mainly. Page reads however should be fast, so if just moving
>>>> the column pointer works, then do it, otherwise we'll consider
>>>> returning FFs.
>>>>
>>>> Thanks,
>>>> Miquèl
>>>
>>
>>
>> Thanks,
>> Miquèl
>>
>
Arseniy Krasnov April 13, 2023, 7 a.m. UTC | #20
On 13.04.2023 09:11, Liang Yang wrote:
> 
> On 2023/4/13 13:32, Liang Yang wrote:
>> Hi Miquel,
>>
>> On 2023/4/12 22:32, Miquel Raynal wrote:
>>> [ EXTERNAL EMAIL ]
>>>
>>> Hello,
>>>
>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>
>>>> Hi Miquel and Arseniy,
>>>>
>>>> On 2023/4/12 20:57, Miquel Raynal wrote:
>>>>> [ EXTERNAL EMAIL ]
>>>>>
>>>>> Hi Arseniy,
>>>>>
>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>>>>>> Hi Arseniy,
>>>>>>>
>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>>>>>> Hi Arseniy,
>>>>>>>>>
>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>
>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>
>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>>>>>>>> to user.
>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>> page:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>> ---
>>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>                       int oob_required, int page)
>>>>>>>>>>>>    {
>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>        int ret;
>>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>>>>>
>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>>>>>
>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>>>>>>>>
>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>
>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>> against bitflips, but the policy is to return the full buffer.
>>>>>>>>
>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>      + ECC codes)?
>>>>>>>
>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>> case the data won't be raw (at least not all of it).
>>>>>>
>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
>>>>>
>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>
>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>> retrieve the missing 32 bytes?
>>>>
>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>     {
>>>>         ......
>>>>         meson_nfc_read_page_sub(nand, page, 1);
>>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>
>>>>     }
>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
>>>
>>> And then (if oob_required) you can bring the missing bytes with
>>> something along:
>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>                oob_buf + oob_len,
>>>                mtd->oobsize - oob_len,
>>>                false);
>>> Should not be a huge performance hit.
>>
>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>
> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?

IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.
RAW mode it is user bytes(not corrected by ECC) + ECC + missing bytes


Also @Liang, is this valid code (drivers/mtd/nand/raw/meson_nand.c)?

	ret = nand_check_erased_ecc_chunk(data, ecc->size,
                                          oob, ecc->bytes + 2,
                                          NULL, 0,      
                                          ecc->strength);

It confused me, because 'oob' buffer contains both user bytes and ECC code,
'ecc->bytes + 2' is 16. May be it should be:

	ret = nand_check_erased_ecc_chunk(data, ecc->size,
                                          oob + 2, ecc->bytes,
                                          NULL, 0,      
                                          ecc->strength);

For example let's look on Tegra's driver (drivers/mtd/nand/raw/tegra_nand.c):

                        u8 *oob = chip->oob_poi + nand->ecc.offset +    
                                  (chip->ecc.bytes * bit);              
                                                                        
                        ret = nand_check_erased_ecc_chunk(data, chip->ecc.size,
                                                          oob, chip->ecc.bytes,
                                                          NULL, 0,      
                                                          chip->ecc.strength);

'oob' contains 'nand->ecc.offset', and ECC length does not account user bytes length
(e.g. 2) - it is just 'chip->ecc.bytes'

Please correct me if i'm wrong.

Thanks, Arseniy

> 
>>>
>>>>
>>>>>
>>>>> This applies to the two modes, the only difference is:
>>>>> - with correction (commonly named "ECC mode"): the user bytes and ECC
>>>>>     bytes should be fixed if there are any bitflips
>>>>> - without correction (commonly referred as "raw mode"): no correction
>>>>>     applies, if there are bitflips, give them
>>>>>
>>>>> Please mind the raw mode can be slow, it's meant for debugging and
>>>>> testing, mainly. Page reads however should be fast, so if just moving
>>>>> the column pointer works, then do it, otherwise we'll consider
>>>>> returning FFs.
>>>>>
>>>>> Thanks,
>>>>> Miquèl
>>>>
>>>
>>>
>>> Thanks,
>>> Miquèl
>>>
>>
>
Miquel Raynal April 13, 2023, 8:22 a.m. UTC | #21
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:

> On 13.04.2023 09:11, Liang Yang wrote:
> > 
> > On 2023/4/13 13:32, Liang Yang wrote:  
> >> Hi Miquel,
> >>
> >> On 2023/4/12 22:32, Miquel Raynal wrote:  
> >>> [ EXTERNAL EMAIL ]
> >>>
> >>> Hello,
> >>>
> >>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
> >>>  
> >>>> Hi Miquel and Arseniy,
> >>>>
> >>>> On 2023/4/12 20:57, Miquel Raynal wrote:  
> >>>>> [ EXTERNAL EMAIL ]
> >>>>>
> >>>>> Hi Arseniy,
> >>>>>
> >>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:  
> >>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:  
> >>>>>>> Hi Arseniy,
> >>>>>>>
> >>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
> >>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:  
> >>>>>>>>> Hi Arseniy,
> >>>>>>>>>
> >>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> >>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:  
> >>>>>>>>>>> Hi Arseniy,
> >>>>>>>>>>>
> >>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> >>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
> >>>>>>>>>>>
> >>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>>>>>>>>>> not retrieve all the OOB bytes,"?
> >>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads  
> >>>>>>>>>>>> to user.
> >>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>>>>>>>>>> page:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ...
> >>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>>>>>>>>>
> >>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>>>>>>>>>    1 file changed, 5 insertions(+)
> >>>>>>>>>>>>
> >>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
> >>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>>>>>>>>>                       int oob_required, int page)
> >>>>>>>>>>>>    {
> >>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
> >>>>>>>>>>>>        int ret;
> >>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);  
> >>>>>>>>>>>
> >>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?  
> >>>>>>>>>>
> >>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> >>>>>>>>>
> >>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
> >>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
> >>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
> >>>>>>>>>
> >>>>>>>>> What happens sometimes, is that some of the bytes are not protected
> >>>>>>>>> against bitflips, but the policy is to return the full buffer.  
> >>>>>>>>
> >>>>>>>> Ok, so to clarify case for this NAND controller:
> >>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> >>>>>>>>      + ECC codes)?  
> >>>>>>>
> >>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
> >>>>>>> case the data won't be raw (at least not all of it).  
> >>>>>>
> >>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> >>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> >>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
> >>>>>
> >>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
> >>>>>
> >>>>> If the controller only returns 2k+32B, then perform a random read to
> >>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
> >>>>> retrieve the missing 32 bytes?  
> >>>>
> >>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
> >>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> >>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
> >>>>     {
> >>>>         ......
> >>>>         meson_nfc_read_page_sub(nand, page, 1);
> >>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
> >>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> >>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
> >>>>
> >>>>     }
> >>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.  
> >>>
> >>> And then (if oob_required) you can bring the missing bytes with
> >>> something along:
> >>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
> >>>                oob_buf + oob_len,
> >>>                mtd->oobsize - oob_len,
> >>>                false);
> >>> Should not be a huge performance hit.  
> >>
> >> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
> >>  
> > use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?  
> 
> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.

I believe the ECC bytes you'll get will be corrected.
You can check this by using nandflipbits in mtd-utils.

> RAW mode it is user bytes(not corrected by ECC) + ECC + missing bytes
> 
> 
> Also @Liang, is this valid code (drivers/mtd/nand/raw/meson_nand.c)?
> 
> 	ret = nand_check_erased_ecc_chunk(data, ecc->size,
>                                           oob, ecc->bytes + 2,
>                                           NULL, 0,      
>                                           ecc->strength);
> 
> It confused me, because 'oob' buffer contains both user bytes and ECC code,
> 'ecc->bytes + 2' is 16. May be it should be:
> 
> 	ret = nand_check_erased_ecc_chunk(data, ecc->size,
>                                           oob + 2, ecc->bytes,
>                                           NULL, 0,      
>                                           ecc->strength);

When you check for an erased chunk you should probably check the whole
OOB area.

> 
> For example let's look on Tegra's driver (drivers/mtd/nand/raw/tegra_nand.c):
> 
>                         u8 *oob = chip->oob_poi + nand->ecc.offset +    
>                                   (chip->ecc.bytes * bit);              
>                                                                         
>                         ret = nand_check_erased_ecc_chunk(data, chip->ecc.size,
>                                                           oob, chip->ecc.bytes,
>                                                           NULL, 0,      
>                                                           chip->ecc.strength);
> 
> 'oob' contains 'nand->ecc.offset', and ECC length does not account user bytes length
> (e.g. 2) - it is just 'chip->ecc.bytes'

I haven't looked carefully, but be aware there are two user bytes
reserved at the beginning of the OOB area for marking bad blocks. There
*may* be a confusion somewhere. I am not saying there is one, just a
hint if you can't find an explanation.


Thanks,
Miquèl
Miquel Raynal April 13, 2023, 8:22 a.m. UTC | #22
Hi Liang,

liang.yang@amlogic.com wrote on Thu, 13 Apr 2023 13:32:03 +0800:

> Hi Miquel,
> 
> On 2023/4/12 22:32, Miquel Raynal wrote:
> > [ EXTERNAL EMAIL ]
> > 
> > Hello,
> > 
> > liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
> >   
> >> Hi Miquel and Arseniy,
> >>
> >> On 2023/4/12 20:57, Miquel Raynal wrote:  
> >>> [ EXTERNAL EMAIL ]
> >>>
> >>> Hi Arseniy,
> >>>
> >>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:  
> >>>    >>>> On 12.04.2023 15:18, Miquel Raynal wrote:  
> >>>>> Hi Arseniy,
> >>>>>
> >>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:  
> >>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:  
> >>>>>>> Hi Arseniy,
> >>>>>>>
> >>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:  
> >>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:  
> >>>>>>>>> Hi Arseniy,
> >>>>>>>>>
> >>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:  
> >>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> >>>>>>>>>
> >>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>>>>>>>> not retrieve all the OOB bytes,"?  
> >>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads  
> >>>>>>>>>> to user.
> >>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>>>>>>>> page:
> >>>>>>>>>>
> >>>>>>>>>> ...
> >>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>>>>>>>> ---
> >>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>>>>>>>    1 file changed, 5 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
> >>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>>>>>>>    				   int oob_required, int page)
> >>>>>>>>>>    {
> >>>>>>>>>> +	struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>>>>>>>    	u8 *oob_buf = nand->oob_poi;
> >>>>>>>>>>    	int ret;  
> >>>>>>>>>>    >>>>>>>> +	memset(oob_buf, 0, mtd->oobsize);  
> >>>>>>>>>
> >>>>>>>>> I'm surprised raw reads do not read the entire OOB?  
> >>>>>>>>
> >>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> >>>>>>>
> >>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
> >>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
> >>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
> >>>>>>>
> >>>>>>> What happens sometimes, is that some of the bytes are not protected
> >>>>>>> against bitflips, but the policy is to return the full buffer.  
> >>>>>>
> >>>>>> Ok, so to clarify case for this NAND controller:
> >>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> >>>>>>      + ECC codes)?  
> >>>>>
> >>>>> Well, you need to cover the same amount of data, yes. But in the ECC
> >>>>> case the data won't be raw (at least not all of it).  
> >>>>
> >>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> >>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> >>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
> >>>
> >>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
> >>>
> >>> If the controller only returns 2k+32B, then perform a random read to
> >>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
> >>> retrieve the missing 32 bytes?  
> >>
> >> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
> >> 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> >> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
> >> 	{
> >> 		......
> >> 		meson_nfc_read_page_sub(nand, page, 1);
> >> 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> >> 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> >> 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
> >>
> >> 	}
> >> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.  
> > 
> > And then (if oob_required) you can bring the missing bytes with
> > something along:
> > nand_change_read_column_op(chip, mtd->writesize + oob_len,
> > 			   oob_buf + oob_len,
> > 			   mtd->oobsize - oob_len,
> > 			   false);
> > Should not be a huge performance hit.  
> 
> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.

Yes, if the last byte read is the one right before the "missing" bytes,
then that will work.

Thanks,
Miquèl
Dmitry Rokosov April 13, 2023, 9:27 a.m. UTC | #23
On Wed, Apr 12, 2023 at 10:56:03PM +0200, Miquel Raynal wrote:
> Hi Dmitry,
> 
> ddrokosov@sberdevices.ru wrote on Wed, 12 Apr 2023 22:15:48 +0300:
> 
> > On Wed, Apr 12, 2023 at 10:04:28PM +0800, Liang Yang wrote:
> > > Hi Miquel and Arseniy,
> > > 
> > > On 2023/4/12 20:57, Miquel Raynal wrote:  
> > > > [ EXTERNAL EMAIL ]
> > > > 
> > > > Hi Arseniy,
> > > > 
> > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> > > >   
> > > > > On 12.04.2023 15:18, Miquel Raynal wrote:  
> > > > > > Hi Arseniy,
> > > > > > 
> > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:  
> > > > > > > On 12.04.2023 12:36, Miquel Raynal wrote:  
> > > > > > > > Hi Arseniy,
> > > > > > > > 
> > > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:  
> > > > > > > > > On 12.04.2023 10:44, Miquel Raynal wrote:  
> > > > > > > > > > Hi Arseniy,
> > > > > > > > > > 
> > > > > > > > > > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:  
> > > > > > > > > > > This NAND reads only few user's bytes in ECC mode (not full OOB), so  
> > > > > > > > > > 
> > > > > > > > > > "This NAND reads" does not look right, do you mean "Subpage reads do
> > > > > > > > > > not retrieve all the OOB bytes,"?  
> > > > > > > > > > > fill OOB buffer with zeroes to not return garbage from previous reads
> > > > > > > > > > > to user.
> > > > > > > > > > > Otherwise 'nanddump' utility prints something like this for just erased
> > > > > > > > > > > page:
> > > > > > > > > > > 
> > > > > > > > > > > ...
> > > > > > > > > > > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > > > > > > >    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> > > > > > > > > > >    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> > > > > > > > > > >    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> > > > > > > > > > >    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > > > > > > > > > > 
> > > > > > > > > > > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > > > > > > > > > > ---
> > > > > > > > > > >   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> > > > > > > > > > >   1 file changed, 5 insertions(+)
> > > > > > > > > > > 
> > > > > > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > > > > > >   				   int oob_required, int page)
> > > > > > > > > > >   {
> > > > > > > > > > > +	struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > > > > > >   	u8 *oob_buf = nand->oob_poi;
> > > > > > > > > > >   	int ret;
> > > > > > > > > > > +	memset(oob_buf, 0, mtd->oobsize);  
> > > > > > > > > > 
> > > > > > > > > > I'm surprised raw reads do not read the entire OOB?  
> > > > > > > > > 
> > > > > > > > > Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> > > > > > > > > still depends on ECC parameters: for each portion of data covered with ECC code we can
> > > > > > > > > read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> > > > > > > > > printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> > > > > > > > > For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> > > > > > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> > > > > > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
> > > > > > > > 
> > > > > > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > > > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > > > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > > > > > 
> > > > > > > > What happens sometimes, is that some of the bytes are not protected
> > > > > > > > against bitflips, but the policy is to return the full buffer.  
> > > > > > > 
> > > > > > > Ok, so to clarify case for this NAND controller:
> > > > > > > 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> > > > > > >     + ECC codes)?  
> > > > > > 
> > > > > > Well, you need to cover the same amount of data, yes. But in the ECC
> > > > > > case the data won't be raw (at least not all of it).  
> > > > > 
> > > > > So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> > > > > OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> > > > > behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
> > > > 
> > > > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > > > 
> > > > If the controller only returns 2k+32B, then perform a random read to
> > > > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > > > retrieve the missing 32 bytes?  
> > > 
> > > 1) raw read can read out the whole page data 2k+64B, decided by the len in
> > > the controller raw read command:
> > > 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> > > after that, the missing oob bytes(not used) can be copied from
> > > meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is
> > > like this if need.
> > > 	{
> > > 		......
> > > 		meson_nfc_read_page_sub(nand, page, 1);
> > > 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> > > 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> > > 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize -
> > > oob_len);
> > > 
> > > 	}
> > > 2) In ECC mode, the controller can't bring back the missing OOB bytes. it
> > > can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> > >   
> > 
> > How does the Meson controller know the actual NAND flash layout when the
> > OOB is split into protected and unprotected areas, such as Free and ECC
> > areas? If the controller has a static OOB layout, where is the mapping
> > located?
> 
> It's usually a set of values hardcoded in the driver. It's a per
> geometry set.
> 

Sorry, I'm still confused. Before I developed spinand drivers, the OOB layout
was located on the flash driver side.
Do you mean if the OOB geometry in the rawnand subsystem is under the
responsibility of the controller driver?

> > 
> > > > 
> > > > This applies to the two modes, the only difference is:
> > > > - with correction (commonly named "ECC mode"): the user bytes and ECC
> > > >    bytes should be fixed if there are any bitflips
> > > > - without correction (commonly referred as "raw mode"): no correction
> > > >    applies, if there are bitflips, give them
> > > > 
> > > > Please mind the raw mode can be slow, it's meant for debugging and
> > > > testing, mainly. Page reads however should be fast, so if just moving
> > > > the column pointer works, then do it, otherwise we'll consider
> > > > returning FFs.  
> >
Arseniy Krasnov April 13, 2023, 9:36 a.m. UTC | #24
On 13.04.2023 11:22, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
> 
>> On 13.04.2023 09:11, Liang Yang wrote:
>>>
>>> On 2023/4/13 13:32, Liang Yang wrote:  
>>>> Hi Miquel,
>>>>
>>>> On 2023/4/12 22:32, Miquel Raynal wrote:  
>>>>> [ EXTERNAL EMAIL ]
>>>>>
>>>>> Hello,
>>>>>
>>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>>>  
>>>>>> Hi Miquel and Arseniy,
>>>>>>
>>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:  
>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>
>>>>>>> Hi Arseniy,
>>>>>>>
>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:  
>>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:  
>>>>>>>>> Hi Arseniy,
>>>>>>>>>
>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:  
>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>
>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:  
>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>
>>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>>>
>>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads  
>>>>>>>>>>>>>> to user.
>>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>>>> page:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>>>                       int oob_required, int page)
>>>>>>>>>>>>>>    {
>>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>>>        int ret;
>>>>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);  
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?  
>>>>>>>>>>>>
>>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.  
>>>>>>>>>>>
>>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>>>
>>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>>>> against bitflips, but the policy is to return the full buffer.  
>>>>>>>>>>
>>>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>>>      + ECC codes)?  
>>>>>>>>>
>>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>>>> case the data won't be raw (at least not all of it).  
>>>>>>>>
>>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?  
>>>>>>>
>>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>>>
>>>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>>>> retrieve the missing 32 bytes?  
>>>>>>
>>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>>>     {
>>>>>>         ......
>>>>>>         meson_nfc_read_page_sub(nand, page, 1);
>>>>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>>>
>>>>>>     }
>>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.  
>>>>>
>>>>> And then (if oob_required) you can bring the missing bytes with
>>>>> something along:
>>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>>>                oob_buf + oob_len,
>>>>>                mtd->oobsize - oob_len,
>>>>>                false);
>>>>> Should not be a huge performance hit.  
>>>>
>>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>>>  
>>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?  
>>
>> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
>> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.
> 
> I believe the ECC bytes you'll get will be corrected.
> You can check this by using nandflipbits in mtd-utils.

Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
driver) also as missing bytes of OOB. So IIUC ECC codes are metadata to correct data bytes, and thus couldn't be
corrected.

Thanks, Arseniy

> 
>> RAW mode it is user bytes(not corrected by ECC) + ECC + missing bytes
>>
>>
>> Also @Liang, is this valid code (drivers/mtd/nand/raw/meson_nand.c)?
>>
>> 	ret = nand_check_erased_ecc_chunk(data, ecc->size,
>>                                           oob, ecc->bytes + 2,
>>                                           NULL, 0,      
>>                                           ecc->strength);
>>
>> It confused me, because 'oob' buffer contains both user bytes and ECC code,
>> 'ecc->bytes + 2' is 16. May be it should be:
>>
>> 	ret = nand_check_erased_ecc_chunk(data, ecc->size,
>>                                           oob + 2, ecc->bytes,
>>                                           NULL, 0,      
>>                                           ecc->strength);
> 
> When you check for an erased chunk you should probably check the whole
> OOB area.

Ok, I see

> 
>>
>> For example let's look on Tegra's driver (drivers/mtd/nand/raw/tegra_nand.c):
>>
>>                         u8 *oob = chip->oob_poi + nand->ecc.offset +    
>>                                   (chip->ecc.bytes * bit);              
>>                                                                         
>>                         ret = nand_check_erased_ecc_chunk(data, chip->ecc.size,
>>                                                           oob, chip->ecc.bytes,
>>                                                           NULL, 0,      
>>                                                           chip->ecc.strength);
>>
>> 'oob' contains 'nand->ecc.offset', and ECC length does not account user bytes length
>> (e.g. 2) - it is just 'chip->ecc.bytes'
> 
> I haven't looked carefully, but be aware there are two user bytes
> reserved at the beginning of the OOB area for marking bad blocks. There
> *may* be a confusion somewhere. I am not saying there is one, just a
> hint if you can't find an explanation.
> 
> 
> Thanks,
> Miquèl
Miquel Raynal April 13, 2023, 10:22 a.m. UTC | #25
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:36:24 +0300:

> On 13.04.2023 11:22, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
> >   
> >> On 13.04.2023 09:11, Liang Yang wrote:  
> >>>
> >>> On 2023/4/13 13:32, Liang Yang wrote:    
> >>>> Hi Miquel,
> >>>>
> >>>> On 2023/4/12 22:32, Miquel Raynal wrote:    
> >>>>> [ EXTERNAL EMAIL ]
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
> >>>>>    
> >>>>>> Hi Miquel and Arseniy,
> >>>>>>
> >>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:    
> >>>>>>> [ EXTERNAL EMAIL ]
> >>>>>>>
> >>>>>>> Hi Arseniy,
> >>>>>>>
> >>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:    
> >>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:    
> >>>>>>>>> Hi Arseniy,
> >>>>>>>>>
> >>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
> >>>>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:    
> >>>>>>>>>>> Hi Arseniy,
> >>>>>>>>>>>
> >>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
> >>>>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:    
> >>>>>>>>>>>>> Hi Arseniy,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
> >>>>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
> >>>>>>>>>>>>> not retrieve all the OOB bytes,"?
> >>>>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads    
> >>>>>>>>>>>>>> to user.
> >>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
> >>>>>>>>>>>>>> page:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>>>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> >>>>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> >>>>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> >>>>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> >>>>>>>>>>>>>>    1 file changed, 5 insertions(+)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
> >>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
> >>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> >>>>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> >>>>>>>>>>>>>>                       int oob_required, int page)
> >>>>>>>>>>>>>>    {
> >>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
> >>>>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
> >>>>>>>>>>>>>>        int ret;
> >>>>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);    
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?    
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> >>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
> >>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> >>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> >>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> >>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> >>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
> >>>>>>>>>>>
> >>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
> >>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
> >>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
> >>>>>>>>>>>
> >>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
> >>>>>>>>>>> against bitflips, but the policy is to return the full buffer.    
> >>>>>>>>>>
> >>>>>>>>>> Ok, so to clarify case for this NAND controller:
> >>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> >>>>>>>>>>      + ECC codes)?    
> >>>>>>>>>
> >>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
> >>>>>>>>> case the data won't be raw (at least not all of it).    
> >>>>>>>>
> >>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> >>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> >>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?    
> >>>>>>>
> >>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
> >>>>>>>
> >>>>>>> If the controller only returns 2k+32B, then perform a random read to
> >>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
> >>>>>>> retrieve the missing 32 bytes?    
> >>>>>>
> >>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
> >>>>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> >>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
> >>>>>>     {
> >>>>>>         ......
> >>>>>>         meson_nfc_read_page_sub(nand, page, 1);
> >>>>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
> >>>>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> >>>>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
> >>>>>>
> >>>>>>     }
> >>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.    
> >>>>>
> >>>>> And then (if oob_required) you can bring the missing bytes with
> >>>>> something along:
> >>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
> >>>>>                oob_buf + oob_len,
> >>>>>                mtd->oobsize - oob_len,
> >>>>>                false);
> >>>>> Should not be a huge performance hit.    
> >>>>
> >>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
> >>>>    
> >>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?    
> >>
> >> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
> >> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.  
> > 
> > I believe the ECC bytes you'll get will be corrected.
> > You can check this by using nandflipbits in mtd-utils.  
> 
> Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
> driver) also as missing bytes of OOB.

Gasp. Yeah that's a controller limitation, okay.

> So IIUC ECC codes are metadata to correct data bytes, and thus
> couldn't be corrected.

We consider them metadata, but they are fully part of the ECC scheme
and thus their correction is part of the process, bitflips in the ECC
bytes will count as data bitflips actually.

I talked a bit about ECC engines at a previous conference if it can
help:
https://elinux.org/ELC_Europe_2020_Presentations
'Understand ECC Support for NAND Flash Devices in Linux'
And also wrote a blog post with a chapter about ECC engines:
https://bootlin.com/blog/supporting-a-misbehaving-nand-ecc-engine/

Thanks,
Miquèl
Miquel Raynal April 13, 2023, 10:29 a.m. UTC | #26
Hi Dmitry,

ddrokosov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:27:06 +0300:

> On Wed, Apr 12, 2023 at 10:56:03PM +0200, Miquel Raynal wrote:
> > Hi Dmitry,
> > 
> > ddrokosov@sberdevices.ru wrote on Wed, 12 Apr 2023 22:15:48 +0300:
> >   
> > > On Wed, Apr 12, 2023 at 10:04:28PM +0800, Liang Yang wrote:  
> > > > Hi Miquel and Arseniy,
> > > > 
> > > > On 2023/4/12 20:57, Miquel Raynal wrote:    
> > > > > [ EXTERNAL EMAIL ]
> > > > > 
> > > > > Hi Arseniy,
> > > > > 
> > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> > > > >     
> > > > > > On 12.04.2023 15:18, Miquel Raynal wrote:    
> > > > > > > Hi Arseniy,
> > > > > > > 
> > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:    
> > > > > > > > On 12.04.2023 12:36, Miquel Raynal wrote:    
> > > > > > > > > Hi Arseniy,
> > > > > > > > > 
> > > > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:    
> > > > > > > > > > On 12.04.2023 10:44, Miquel Raynal wrote:    
> > > > > > > > > > > Hi Arseniy,
> > > > > > > > > > > 
> > > > > > > > > > > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:    
> > > > > > > > > > > > This NAND reads only few user's bytes in ECC mode (not full OOB), so    
> > > > > > > > > > > 
> > > > > > > > > > > "This NAND reads" does not look right, do you mean "Subpage reads do
> > > > > > > > > > > not retrieve all the OOB bytes,"?    
> > > > > > > > > > > > fill OOB buffer with zeroes to not return garbage from previous reads
> > > > > > > > > > > > to user.
> > > > > > > > > > > > Otherwise 'nanddump' utility prints something like this for just erased
> > > > > > > > > > > > page:
> > > > > > > > > > > > 
> > > > > > > > > > > > ...
> > > > > > > > > > > > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > > > > > > > >    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> > > > > > > > > > > >    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> > > > > > > > > > > >    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> > > > > > > > > > > >    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > > > > > > > > > > > 
> > > > > > > > > > > > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > > > > > > > > > > > ---
> > > > > > > > > > > >   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> > > > > > > > > > > >   1 file changed, 5 insertions(+)
> > > > > > > > > > > > 
> > > > > > > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > > > > > > >   				   int oob_required, int page)
> > > > > > > > > > > >   {
> > > > > > > > > > > > +	struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > > > > > > >   	u8 *oob_buf = nand->oob_poi;
> > > > > > > > > > > >   	int ret;
> > > > > > > > > > > > +	memset(oob_buf, 0, mtd->oobsize);    
> > > > > > > > > > > 
> > > > > > > > > > > I'm surprised raw reads do not read the entire OOB?    
> > > > > > > > > > 
> > > > > > > > > > Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> > > > > > > > > > still depends on ECC parameters: for each portion of data covered with ECC code we can
> > > > > > > > > > read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> > > > > > > > > > printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> > > > > > > > > > For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> > > > > > > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> > > > > > > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
> > > > > > > > > 
> > > > > > > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > > > > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > > > > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > > > > > > 
> > > > > > > > > What happens sometimes, is that some of the bytes are not protected
> > > > > > > > > against bitflips, but the policy is to return the full buffer.    
> > > > > > > > 
> > > > > > > > Ok, so to clarify case for this NAND controller:
> > > > > > > > 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> > > > > > > >     + ECC codes)?    
> > > > > > > 
> > > > > > > Well, you need to cover the same amount of data, yes. But in the ECC
> > > > > > > case the data won't be raw (at least not all of it).    
> > > > > > 
> > > > > > So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> > > > > > OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> > > > > > behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?    
> > > > > 
> > > > > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > > > > 
> > > > > If the controller only returns 2k+32B, then perform a random read to
> > > > > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > > > > retrieve the missing 32 bytes?    
> > > > 
> > > > 1) raw read can read out the whole page data 2k+64B, decided by the len in
> > > > the controller raw read command:
> > > > 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> > > > after that, the missing oob bytes(not used) can be copied from
> > > > meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is
> > > > like this if need.
> > > > 	{
> > > > 		......
> > > > 		meson_nfc_read_page_sub(nand, page, 1);
> > > > 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> > > > 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> > > > 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize -
> > > > oob_len);
> > > > 
> > > > 	}
> > > > 2) In ECC mode, the controller can't bring back the missing OOB bytes. it
> > > > can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> > > >     
> > > 
> > > How does the Meson controller know the actual NAND flash layout when the
> > > OOB is split into protected and unprotected areas, such as Free and ECC
> > > areas? If the controller has a static OOB layout, where is the mapping
> > > located?  
> > 
> > It's usually a set of values hardcoded in the driver. It's a per
> > geometry set.
> >   
> 
> Sorry, I'm still confused. Before I developed spinand drivers, the OOB layout
> was located on the flash driver side.

The spinand subsystem is different, most of the chips have on-die ECC,
which means the ECC layout is most of the time per-chip.

In the raw NAND layer, most of the time the ECC engine is merged with
the NAND controller and thus the list of available layouts depend on
the controller. But these layouts often adapt to the NAND chip
geometry, so you can only now which layout to use after identifying the
chip (there is an ->attach_chip() hook to handle things related to
geometry after controller base initialization.

> Do you mean if the OOB geometry in the rawnand subsystem is under the
> responsibility of the controller driver?

The geometry is NAND chip specific.
The OOB layout depends on the chip geometry, the ECC engine
capabilities and configuration (like the strength).
The discovery of the geometry is performed by the core (using raw NAND
controller driver callbacks of course).
If on-host ECC engine is picked (the default in the raw NAND
subsystem), then the controller driver picks the correct OOB layout and
refuses to probe otherwise.

Hope its clearer now :)

Thanks,
Miquèl
Arseniy Krasnov April 13, 2023, 10:35 a.m. UTC | #27
On 13.04.2023 13:22, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:36:24 +0300:
> 
>> On 13.04.2023 11:22, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
>>>   
>>>> On 13.04.2023 09:11, Liang Yang wrote:  
>>>>>
>>>>> On 2023/4/13 13:32, Liang Yang wrote:    
>>>>>> Hi Miquel,
>>>>>>
>>>>>> On 2023/4/12 22:32, Miquel Raynal wrote:    
>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>>>>>    
>>>>>>>> Hi Miquel and Arseniy,
>>>>>>>>
>>>>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:    
>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>
>>>>>>>>> Hi Arseniy,
>>>>>>>>>
>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:    
>>>>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:    
>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>
>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:    
>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>
>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:    
>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads    
>>>>>>>>>>>>>>>> to user.
>>>>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>>>>>> page:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>>>>>                       int oob_required, int page)
>>>>>>>>>>>>>>>>    {
>>>>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>>>>>        int ret;
>>>>>>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);    
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?    
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
>>>>>>>>>>>>>
>>>>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>>>>>> against bitflips, but the policy is to return the full buffer.    
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>>>>>      + ECC codes)?    
>>>>>>>>>>>
>>>>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>>>>>> case the data won't be raw (at least not all of it).    
>>>>>>>>>>
>>>>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?    
>>>>>>>>>
>>>>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>>>>>
>>>>>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>>>>>> retrieve the missing 32 bytes?    
>>>>>>>>
>>>>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>>>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>>>>>     {
>>>>>>>>         ......
>>>>>>>>         meson_nfc_read_page_sub(nand, page, 1);
>>>>>>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>>>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>>>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>>>>>
>>>>>>>>     }
>>>>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.    
>>>>>>>
>>>>>>> And then (if oob_required) you can bring the missing bytes with
>>>>>>> something along:
>>>>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>>>>>                oob_buf + oob_len,
>>>>>>>                mtd->oobsize - oob_len,
>>>>>>>                false);
>>>>>>> Should not be a huge performance hit.    
>>>>>>
>>>>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>>>>>    
>>>>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?    
>>>>
>>>> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
>>>> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.  
>>>
>>> I believe the ECC bytes you'll get will be corrected.
>>> You can check this by using nandflipbits in mtd-utils.  
>>
>> Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
>> driver) also as missing bytes of OOB.
> 
> Gasp. Yeah that's a controller limitation, okay.
> 
>> So IIUC ECC codes are metadata to correct data bytes, and thus
>> couldn't be corrected.
> 
> We consider them metadata, but they are fully part of the ECC scheme
> and thus their correction is part of the process, bitflips in the ECC
> bytes will count as data bitflips actually.
> 
> I talked a bit about ECC engines at a previous conference if it can
> help:
> https://elinux.org/ELC_Europe_2020_Presentations
> 'Understand ECC Support for NAND Flash Devices in Linux'
> And also wrote a blog post with a chapter about ECC engines:
> https://bootlin.com/blog/supporting-a-misbehaving-nand-ecc-engine/


Thanks for this!

Thanks, Arseniy

> 
> Thanks,
> Miquèl
Dmitry Rokosov April 13, 2023, 2:03 p.m. UTC | #28
On Thu, Apr 13, 2023 at 12:29:16PM +0200, Miquel Raynal wrote:
> Hi Dmitry,
> 
> ddrokosov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:27:06 +0300:
> 
> > On Wed, Apr 12, 2023 at 10:56:03PM +0200, Miquel Raynal wrote:
> > > Hi Dmitry,
> > > 
> > > ddrokosov@sberdevices.ru wrote on Wed, 12 Apr 2023 22:15:48 +0300:
> > >   
> > > > On Wed, Apr 12, 2023 at 10:04:28PM +0800, Liang Yang wrote:  
> > > > > Hi Miquel and Arseniy,
> > > > > 
> > > > > On 2023/4/12 20:57, Miquel Raynal wrote:    
> > > > > > [ EXTERNAL EMAIL ]
> > > > > > 
> > > > > > Hi Arseniy,
> > > > > > 
> > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
> > > > > >     
> > > > > > > On 12.04.2023 15:18, Miquel Raynal wrote:    
> > > > > > > > Hi Arseniy,
> > > > > > > > 
> > > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:    
> > > > > > > > > On 12.04.2023 12:36, Miquel Raynal wrote:    
> > > > > > > > > > Hi Arseniy,
> > > > > > > > > > 
> > > > > > > > > > avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:    
> > > > > > > > > > > On 12.04.2023 10:44, Miquel Raynal wrote:    
> > > > > > > > > > > > Hi Arseniy,
> > > > > > > > > > > > 
> > > > > > > > > > > > AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:    
> > > > > > > > > > > > > This NAND reads only few user's bytes in ECC mode (not full OOB), so    
> > > > > > > > > > > > 
> > > > > > > > > > > > "This NAND reads" does not look right, do you mean "Subpage reads do
> > > > > > > > > > > > not retrieve all the OOB bytes,"?    
> > > > > > > > > > > > > fill OOB buffer with zeroes to not return garbage from previous reads
> > > > > > > > > > > > > to user.
> > > > > > > > > > > > > Otherwise 'nanddump' utility prints something like this for just erased
> > > > > > > > > > > > > page:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > > > > > > > > >    OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
> > > > > > > > > > > > >    OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
> > > > > > > > > > > > >    OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
> > > > > > > > > > > > >    OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > >   drivers/mtd/nand/raw/meson_nand.c | 5 +++++
> > > > > > > > > > > > >   1 file changed, 5 insertions(+)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > > index f84a10238e4d..f2f2472cb511 100644
> > > > > > > > > > > > > --- a/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > > +++ b/drivers/mtd/nand/raw/meson_nand.c
> > > > > > > > > > > > > @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
> > > > > > > > > > > > >   static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
> > > > > > > > > > > > >   				   int oob_required, int page)
> > > > > > > > > > > > >   {
> > > > > > > > > > > > > +	struct mtd_info *mtd = nand_to_mtd(nand);
> > > > > > > > > > > > >   	u8 *oob_buf = nand->oob_poi;
> > > > > > > > > > > > >   	int ret;
> > > > > > > > > > > > > +	memset(oob_buf, 0, mtd->oobsize);    
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm surprised raw reads do not read the entire OOB?    
> > > > > > > > > > > 
> > > > > > > > > > > Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
> > > > > > > > > > > still depends on ECC parameters: for each portion of data covered with ECC code we can
> > > > > > > > > > > read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
> > > > > > > > > > > printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
> > > > > > > > > > > For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
> > > > > > > > > > > contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
> > > > > > > > > > > controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
> > > > > > > > > > 
> > > > > > > > > > In all modes, when you read OOB, you should get the full OOB. The fact
> > > > > > > > > > that ECC correction is enabled or disabled does not matter. If the NAND
> > > > > > > > > > features OOB sections of 64 bytes, you should get the 64 bytes.
> > > > > > > > > > 
> > > > > > > > > > What happens sometimes, is that some of the bytes are not protected
> > > > > > > > > > against bitflips, but the policy is to return the full buffer.    
> > > > > > > > > 
> > > > > > > > > Ok, so to clarify case for this NAND controller:
> > > > > > > > > 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
> > > > > > > > >     + ECC codes)?    
> > > > > > > > 
> > > > > > > > Well, you need to cover the same amount of data, yes. But in the ECC
> > > > > > > > case the data won't be raw (at least not all of it).    
> > > > > > > 
> > > > > > > So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
> > > > > > > OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
> > > > > > > behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?    
> > > > > > 
> > > > > > If the page are 2k+64B you should read 2k+64B when OOB are requested.
> > > > > > 
> > > > > > If the controller only returns 2k+32B, then perform a random read to
> > > > > > just move the read pointer to mtd->size + mtd->oobsize - 32 and
> > > > > > retrieve the missing 32 bytes?    
> > > > > 
> > > > > 1) raw read can read out the whole page data 2k+64B, decided by the len in
> > > > > the controller raw read command:
> > > > > 	cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
> > > > > after that, the missing oob bytes(not used) can be copied from
> > > > > meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is
> > > > > like this if need.
> > > > > 	{
> > > > > 		......
> > > > > 		meson_nfc_read_page_sub(nand, page, 1);
> > > > > 		meson_nfc_get_data_oob(nand, buf, oob_buf);
> > > > > 		oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
> > > > > 		memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize -
> > > > > oob_len);
> > > > > 
> > > > > 	}
> > > > > 2) In ECC mode, the controller can't bring back the missing OOB bytes. it
> > > > > can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
> > > > >     
> > > > 
> > > > How does the Meson controller know the actual NAND flash layout when the
> > > > OOB is split into protected and unprotected areas, such as Free and ECC
> > > > areas? If the controller has a static OOB layout, where is the mapping
> > > > located?  
> > > 
> > > It's usually a set of values hardcoded in the driver. It's a per
> > > geometry set.
> > >   
> > 
> > Sorry, I'm still confused. Before I developed spinand drivers, the OOB layout
> > was located on the flash driver side.
> 
> The spinand subsystem is different, most of the chips have on-die ECC,
> which means the ECC layout is most of the time per-chip.
> 
> In the raw NAND layer, most of the time the ECC engine is merged with
> the NAND controller and thus the list of available layouts depend on
> the controller. But these layouts often adapt to the NAND chip
> geometry, so you can only now which layout to use after identifying the
> chip (there is an ->attach_chip() hook to handle things related to
> geometry after controller base initialization.
> 
> > Do you mean if the OOB geometry in the rawnand subsystem is under the
> > responsibility of the controller driver?
> 
> The geometry is NAND chip specific.
> The OOB layout depends on the chip geometry, the ECC engine
> capabilities and configuration (like the strength).
> The discovery of the geometry is performed by the core (using raw NAND
> controller driver callbacks of course).
> If on-host ECC engine is picked (the default in the raw NAND
> subsystem), then the controller driver picks the correct OOB layout and
> refuses to probe otherwise.
> 
> Hope its clearer now :)

Yeah, pretty clear!

I wanted to express my special thanks to you for sharing such detailed
information about the raw NAND layer! Your insights will be incredibly
helpful for me and will provide valuable context that I would not have
otherwise had.

Thank you so much for taking the time to share your expertise with me!
Arseniy Krasnov April 18, 2023, 5:12 a.m. UTC | #29
On 13.04.2023 13:35, Arseniy Krasnov wrote:
> 
> 
> On 13.04.2023 13:22, Miquel Raynal wrote:
>> Hi Arseniy,
>>
>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:36:24 +0300:
>>
>>> On 13.04.2023 11:22, Miquel Raynal wrote:
>>>> Hi Arseniy,
>>>>
>>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
>>>>   
>>>>> On 13.04.2023 09:11, Liang Yang wrote:  
>>>>>>
>>>>>> On 2023/4/13 13:32, Liang Yang wrote:    
>>>>>>> Hi Miquel,
>>>>>>>
>>>>>>> On 2023/4/12 22:32, Miquel Raynal wrote:    
>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>>>>>>    
>>>>>>>>> Hi Miquel and Arseniy,
>>>>>>>>>
>>>>>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:    
>>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>>
>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>
>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:    
>>>>>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:    
>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>
>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>>>>>>     >>>> On 12.04.2023 12:36, Miquel Raynal wrote:    
>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>>>>>>       >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:    
>>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>>>>>>         >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>>>>>>         >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads    
>>>>>>>>>>>>>>>>> to user.
>>>>>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>>>>>>> page:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>>>>>>     OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>>>>>>     OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>>>>>>     OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>>>>>>     OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>    drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>>>>>>    1 file changed, 5 insertions(+)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>>>>>>    static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>>>>>>                       int oob_required, int page)
>>>>>>>>>>>>>>>>>    {
>>>>>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>>>>>>        u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>>>>>>        int ret;
>>>>>>>>>>>>>>>>>    >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);    
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?    
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.    
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>>>>>>> against bitflips, but the policy is to return the full buffer.    
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>>>>>>      + ECC codes)?    
>>>>>>>>>>>>
>>>>>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>>>>>>> case the data won't be raw (at least not all of it).    
>>>>>>>>>>>
>>>>>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?    
>>>>>>>>>>
>>>>>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>>>>>>
>>>>>>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>>>>>>> retrieve the missing 32 bytes?    
>>>>>>>>>
>>>>>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>>>>>>     cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>>>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>>>>>>     {
>>>>>>>>>         ......
>>>>>>>>>         meson_nfc_read_page_sub(nand, page, 1);
>>>>>>>>>         meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>>>>>>         oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>>>>>>         memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>>>>>>
>>>>>>>>>     }
>>>>>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.    
>>>>>>>>
>>>>>>>> And then (if oob_required) you can bring the missing bytes with
>>>>>>>> something along:
>>>>>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>>>>>>                oob_buf + oob_len,
>>>>>>>>                mtd->oobsize - oob_len,
>>>>>>>>                false);
>>>>>>>> Should not be a huge performance hit.    
>>>>>>>
>>>>>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>>>>>>    
>>>>>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?    
>>>>>
>>>>> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
>>>>> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.  
>>>>
>>>> I believe the ECC bytes you'll get will be corrected.
>>>> You can check this by using nandflipbits in mtd-utils.  
>>>
>>> Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
>>> driver) also as missing bytes of OOB.
>>
>> Gasp. Yeah that's a controller limitation, okay.
>>
>>> So IIUC ECC codes are metadata to correct data bytes, and thus
>>> couldn't be corrected.
>>
>> We consider them metadata, but they are fully part of the ECC scheme
>> and thus their correction is part of the process, bitflips in the ECC
>> bytes will count as data bitflips actually.
>>
>> I talked a bit about ECC engines at a previous conference if it can
>> help:
>> https://elinux.org/ELC_Europe_2020_Presentations
>> 'Understand ECC Support for NAND Flash Devices in Linux'
>> And also wrote a blog post with a chapter about ECC engines:
>> https://bootlin.com/blog/supporting-a-misbehaving-nand-ecc-engine/
> 
> 
> Thanks for this!
> 
> Thanks, Arseniy

Hello again @Liang @Miquel!

One more question about OOB access, as I can see current driver uses the following
callbacks:

	nand->ecc.write_oob_raw = nand_write_oob_std;                   
	nand->ecc.write_oob = nand_write_oob_std;


Function 'nand_write_oob_std()' writes data to the end of the page. But as I
can see by dumping 'data_buf' during read, physical layout of each page is the
following (1KB ECC):

0x000: [         1 KB of data        ]
0x400: [ 2B user data] [ 14B ECC code]
0x410: [         1 KB of data        ]    (A)
0x810: [ 2B user data] [ 14B ECC code]
0x820: [        32B unused           ]



So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
page will look like this:

0x000: [             0xFF            ]
0x400: [           ........          ]
0x7f0: [             0xFF            ]
0x800: [ 00 .......................  ]
0x830: [ ........................ 3f ]

Here we have two problems:
1) Attempt to display raw data by 'nanddump' utility produces a little bit
   invalid output, as driver relies on layout (A) from above. E.g. OOB data
   is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):

0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
  OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
  OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
  OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
  OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|


2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
   state (I mean was written at least once) and NAND controller tries to use
   ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
   we have strange situation: OOB seems written without any errors, but we can't
   read this page. First idea is to write OOB data to 0x400 and 0x810 in raw mode,
   but this does not work - if there is some data, NAND controller will try to
   use ECC engine to check these user bytes on next attempt to read this page. But
   as these 4 bytes were written in raw mode, ECC codes are missed.

We suggest the following thing: use only area at 0x820 for OOB (see A) - it is not covered
by ECC engine, so write to this zone won't conflict with ECC in future. In this case
we change 'meson_ooblayout_free()' function which instead of describing 2 user bytes
for each ECC block, will return 16 tail bytes for each ECC block.

What to You think?


> 
>>
>> Thanks,
>> Miquèl
Liang Yang April 18, 2023, 12:24 p.m. UTC | #30
Hi Arseniy and Miquel,

On 2023/4/18 13:12, Arseniy Krasnov wrote:
> [ EXTERNAL EMAIL ]
> 
> 
> 
> On 13.04.2023 13:35, Arseniy Krasnov wrote:
>>
>>
>> On 13.04.2023 13:22, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:36:24 +0300:
>>>
>>>> On 13.04.2023 11:22, Miquel Raynal wrote:
>>>>> Hi Arseniy,
>>>>>
>>>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
>>>>>    
>>>>>> On 13.04.2023 09:11, Liang Yang wrote:
>>>>>>>
>>>>>>> On 2023/4/13 13:32, Liang Yang wrote:
>>>>>>>> Hi Miquel,
>>>>>>>>
>>>>>>>> On 2023/4/12 22:32, Miquel Raynal wrote:
>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>>>>>>>     
>>>>>>>>>> Hi Miquel and Arseniy,
>>>>>>>>>>
>>>>>>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:
>>>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>>>
>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>
>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
>>>>>>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>
>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>>>>>>>      >>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>>>>>>>        >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>>>>>>>          >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>>>>>>>          >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>>>>>>>>>>>>>> to user.
>>>>>>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>>>>>>>> page:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>>>>>>>      OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>>>>>>>      OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>>>>>>>      OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>>>>>>>      OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>     drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>>>>>>>     static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>>>>>>>                        int oob_required, int page)
>>>>>>>>>>>>>>>>>>     {
>>>>>>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>>>>>>>         u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>>>>>>>         int ret;
>>>>>>>>>>>>>>>>>>     >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>>>>>>>> against bitflips, but the policy is to return the full buffer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>>>>>>>       + ECC codes)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>>>>>>>> case the data won't be raw (at least not all of it).
>>>>>>>>>>>>
>>>>>>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
>>>>>>>>>>>
>>>>>>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>>>>>>>
>>>>>>>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>>>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>>>>>>>> retrieve the missing 32 bytes?
>>>>>>>>>>
>>>>>>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>>>>>>>      cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>>>>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>>>>>>>      {
>>>>>>>>>>          ......
>>>>>>>>>>          meson_nfc_read_page_sub(nand, page, 1);
>>>>>>>>>>          meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>>>>>>>          oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>>>>>>>          memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>>>>>>>
>>>>>>>>>>      }
>>>>>>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
>>>>>>>>>
>>>>>>>>> And then (if oob_required) you can bring the missing bytes with
>>>>>>>>> something along:
>>>>>>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>>>>>>>                 oob_buf + oob_len,
>>>>>>>>>                 mtd->oobsize - oob_len,
>>>>>>>>>                 false);
>>>>>>>>> Should not be a huge performance hit.
>>>>>>>>
>>>>>>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>>>>>>>     
>>>>>>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?
>>>>>>
>>>>>> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
>>>>>> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.
>>>>>
>>>>> I believe the ECC bytes you'll get will be corrected.
>>>>> You can check this by using nandflipbits in mtd-utils.
>>>>
>>>> Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
>>>> driver) also as missing bytes of OOB.
>>>
>>> Gasp. Yeah that's a controller limitation, okay.
>>>
>>>> So IIUC ECC codes are metadata to correct data bytes, and thus
>>>> couldn't be corrected.
>>>
>>> We consider them metadata, but they are fully part of the ECC scheme
>>> and thus their correction is part of the process, bitflips in the ECC
>>> bytes will count as data bitflips actually.
>>>
>>> I talked a bit about ECC engines at a previous conference if it can
>>> help:
>>> https://elinux.org/ELC_Europe_2020_Presentations
>>> 'Understand ECC Support for NAND Flash Devices in Linux'
>>> And also wrote a blog post with a chapter about ECC engines:
>>> https://bootlin.com/blog/supporting-a-misbehaving-nand-ecc-engine/
>>
>>
>> Thanks for this!
>>
>> Thanks, Arseniy
> 
> Hello again @Liang @Miquel!
> 
> One more question about OOB access, as I can see current driver uses the following
> callbacks:
> 
> 	nand->ecc.write_oob_raw = nand_write_oob_std;
> 	nand->ecc.write_oob = nand_write_oob_std;
> 
> 
> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
> can see by dumping 'data_buf' during read, physical layout of each page is the
> following (1KB ECC):
> 
> 0x000: [         1 KB of data        ]
> 0x400: [ 2B user data] [ 14B ECC code]
> 0x410: [         1 KB of data        ]    (A)
> 0x810: [ 2B user data] [ 14B ECC code]
> 0x820: [        32B unused           ]
> 
> 
> 
> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
> page will look like this:
> 
> 0x000: [             0xFF            ]
> 0x400: [           ........          ]
> 0x7f0: [             0xFF            ]
> 0x800: [ 00 .......................  ]
> 0x830: [ ........................ 3f ]
> 
> Here we have two problems:
> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>     invalid output, as driver relies on layout (A) from above. E.g. OOB data
>     is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
> 
> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>    OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> 
Hi Arseniy,

I realized the write_oob_raw() and write_oob() are wrong in 
meson_nand.c. I suggest both of them should be reworked and follow the 
format of meson nand controller. i.e. firstly format the data in Layout 
(A) and then write. reading is firstly reading the data of layout (A) 
and then compost the layout (B).


> 
> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>     state (I mean was written at least once) and NAND controller tries to use
>     ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus

As i said above, write_oob_raw() and write_oob() should be reworked.
i don't know what do you mean page was written at least once. anyway the 
page should be written once, even just write_oob_raw().

>     we have strange situation: OOB seems written without any errors, but we can't
>     read this page. First idea is to write OOB data to 0x400 and 0x810 in raw mode,
>     but this does not work - if there is some data, NAND controller will try to
>     use ECC engine to check these user bytes on next attempt to read this page. But
>     as these 4 bytes were written in raw mode, ECC codes are missed.
> 
> We suggest the following thing: use only area at 0x820 for OOB (see A) - it is not covered
> by ECC engine, so write to this zone won't conflict with ECC in future. In this case
> we change 'meson_ooblayout_free()' function which instead of describing 2 user bytes
> for each ECC block, will return 16 tail bytes for each ECC block.
> 
> What to You think?

the key point is that the data 0x820-0x840 is not protected by Host 
ECC.so i don't think we have to change it.

Thanks,
Liang
> 
> 
>>
>>>
>>> Thanks,
>>> Miquèl
>
Arseniy Krasnov April 18, 2023, 12:44 p.m. UTC | #31
On 18.04.2023 15:24, Liang Yang wrote:
> Hi Arseniy and Miquel,
> 
> On 2023/4/18 13:12, Arseniy Krasnov wrote:
>> [ EXTERNAL EMAIL ]
>>
>>
>>
>> On 13.04.2023 13:35, Arseniy Krasnov wrote:
>>>
>>>
>>> On 13.04.2023 13:22, Miquel Raynal wrote:
>>>> Hi Arseniy,
>>>>
>>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 12:36:24 +0300:
>>>>
>>>>> On 13.04.2023 11:22, Miquel Raynal wrote:
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> avkrasnov@sberdevices.ru wrote on Thu, 13 Apr 2023 10:00:24 +0300:
>>>>>>   
>>>>>>> On 13.04.2023 09:11, Liang Yang wrote:
>>>>>>>>
>>>>>>>> On 2023/4/13 13:32, Liang Yang wrote:
>>>>>>>>> Hi Miquel,
>>>>>>>>>
>>>>>>>>> On 2023/4/12 22:32, Miquel Raynal wrote:
>>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> liang.yang@amlogic.com wrote on Wed, 12 Apr 2023 22:04:28 +0800:
>>>>>>>>>>    
>>>>>>>>>>> Hi Miquel and Arseniy,
>>>>>>>>>>>
>>>>>>>>>>> On 2023/4/12 20:57, Miquel Raynal wrote:
>>>>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>
>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 15:22:26 +0300:
>>>>>>>>>>>>> On 12.04.2023 15:18, Miquel Raynal wrote:
>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 13:14:52 +0300:
>>>>>>>>>>>>>>      >>>> On 12.04.2023 12:36, Miquel Raynal wrote:
>>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> avkrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 12:20:55 +0300:
>>>>>>>>>>>>>>>>        >>>>>> On 12.04.2023 10:44, Miquel Raynal wrote:
>>>>>>>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> AVKrasnov@sberdevices.ru wrote on Wed, 12 Apr 2023 09:16:58 +0300:
>>>>>>>>>>>>>>>>>>          >>>>>>>> This NAND reads only few user's bytes in ECC mode (not full OOB), so
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> "This NAND reads" does not look right, do you mean "Subpage reads do
>>>>>>>>>>>>>>>>>> not retrieve all the OOB bytes,"?
>>>>>>>>>>>>>>>>>>          >>>>>>>> fill OOB buffer with zeroes to not return garbage from previous reads
>>>>>>>>>>>>>>>>>>> to user.
>>>>>>>>>>>>>>>>>>> Otherwise 'nanddump' utility prints something like this for just erased
>>>>>>>>>>>>>>>>>>> page:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>> 0x000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>>>>>>>>>>>>>>>>      OOB Data: ff ff ff ff 00 00 ff ff 80 cf 22 99 cb ad d3 be
>>>>>>>>>>>>>>>>>>>      OOB Data: 63 27 ae 06 16 0a 2f eb bb dd 46 74 41 8e 88 6e
>>>>>>>>>>>>>>>>>>>      OOB Data: 38 a1 2d e6 77 d4 05 06 f2 a5 7e 25 eb 34 7c ff
>>>>>>>>>>>>>>>>>>>      OOB Data: 38 ea de 14 10 de 9b 40 33 16 6a cc 9d aa 2f 5e
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Signed-off-by: Arseniy Krasnov <AVKrasnov@sberdevices.ru>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>     drivers/mtd/nand/raw/meson_nand.c | 5 +++++
>>>>>>>>>>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>>> index f84a10238e4d..f2f2472cb511 100644
>>>>>>>>>>>>>>>>>>> --- a/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>>> +++ b/drivers/mtd/nand/raw/meson_nand.c
>>>>>>>>>>>>>>>>>>> @@ -858,9 +858,12 @@ static int meson_nfc_read_page_sub(struct nand_chip *nand,
>>>>>>>>>>>>>>>>>>>     static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
>>>>>>>>>>>>>>>>>>>                        int oob_required, int page)
>>>>>>>>>>>>>>>>>>>     {
>>>>>>>>>>>>>>>>>>> +    struct mtd_info *mtd = nand_to_mtd(nand);
>>>>>>>>>>>>>>>>>>>         u8 *oob_buf = nand->oob_poi;
>>>>>>>>>>>>>>>>>>>         int ret;
>>>>>>>>>>>>>>>>>>>     >>>>>>>> +    memset(oob_buf, 0, mtd->oobsize);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm surprised raw reads do not read the entire OOB?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes! Seems in case of raw access (what i see in this driver) number of OOB bytes read
>>>>>>>>>>>>>>>>> still depends on ECC parameters: for each portion of data covered with ECC code we can
>>>>>>>>>>>>>>>>> read it's ECC code and "user bytes" from OOB - it is what i see by dumping DMA buffer by
>>>>>>>>>>>>>>>>> printk(). For example I'm working with 2K NAND pages, each page has 2 x 1K ECC blocks.
>>>>>>>>>>>>>>>>> For each ECC block I have 16 OOB bytes which I can access by read/write. Each 16 bytes
>>>>>>>>>>>>>>>>> contains 2 bytes of user's data and 14 bytes ECC codes. So when I read page in raw mode
>>>>>>>>>>>>>>>>> controller returns 32 bytes (2 x (2 + 14)) of OOB. While OOB is reported as 64 bytes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In all modes, when you read OOB, you should get the full OOB. The fact
>>>>>>>>>>>>>>>> that ECC correction is enabled or disabled does not matter. If the NAND
>>>>>>>>>>>>>>>> features OOB sections of 64 bytes, you should get the 64 bytes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What happens sometimes, is that some of the bytes are not protected
>>>>>>>>>>>>>>>> against bitflips, but the policy is to return the full buffer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok, so to clarify case for this NAND controller:
>>>>>>>>>>>>>>> 1) In both ECC and raw modes i need to return the same raw OOB data (e.g. user bytes
>>>>>>>>>>>>>>>       + ECC codes)?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Well, you need to cover the same amount of data, yes. But in the ECC
>>>>>>>>>>>>>> case the data won't be raw (at least not all of it).
>>>>>>>>>>>>>
>>>>>>>>>>>>> So "same amount of data", in ECC mode current implementation returns only user OOB bytes (e.g.
>>>>>>>>>>>>> OOB data excluding ECC codes), in raw it returns user bytes + ECC codes. IIUC correct
>>>>>>>>>>>>> behaviour is to always return user bytes + ECC codes as OOB data even in ECC mode ?
>>>>>>>>>>>>
>>>>>>>>>>>> If the page are 2k+64B you should read 2k+64B when OOB are requested.
>>>>>>>>>>>>
>>>>>>>>>>>> If the controller only returns 2k+32B, then perform a random read to
>>>>>>>>>>>> just move the read pointer to mtd->size + mtd->oobsize - 32 and
>>>>>>>>>>>> retrieve the missing 32 bytes?
>>>>>>>>>>>
>>>>>>>>>>> 1) raw read can read out the whole page data 2k+64B, decided by the len in the controller raw read command:
>>>>>>>>>>>      cmd = (len & GENMASK(5, 0)) | scrambler | DMA_DIR(dir);
>>>>>>>>>>> after that, the missing oob bytes(not used) can be copied from meson_chip->data_buf. so the implementation of meson_nfc_read_page_raw() is like this if need.
>>>>>>>>>>>      {
>>>>>>>>>>>          ......
>>>>>>>>>>>          meson_nfc_read_page_sub(nand, page, 1);
>>>>>>>>>>>          meson_nfc_get_data_oob(nand, buf, oob_buf);
>>>>>>>>>>>          oob_len = (nand->ecc.bytes + 2) * nand->ecc.steps;
>>>>>>>>>>>          memcpy(oob_buf + oob_len, meson_chip->data_buf + oob_len, mtd->oobsize - oob_len);
>>>>>>>>>>>
>>>>>>>>>>>      }
>>>>>>>>>>> 2) In ECC mode, the controller can't bring back the missing OOB bytes. it can read out the user bytes and ecc bytes per meson_ooblayout_ops define.
>>>>>>>>>>
>>>>>>>>>> And then (if oob_required) you can bring the missing bytes with
>>>>>>>>>> something along:
>>>>>>>>>> nand_change_read_column_op(chip, mtd->writesize + oob_len,
>>>>>>>>>>                 oob_buf + oob_len,
>>>>>>>>>>                 mtd->oobsize - oob_len,
>>>>>>>>>>                 false);
>>>>>>>>>> Should not be a huge performance hit.
>>>>>>>>>
>>>>>>>>> After finishing ECC mode reading, the column address internal in NAND device should be the right pos; it doesn't need to change the column again. so adding controller raw read for the missing bytes after ECC reading may works.
>>>>>>>>>     
>>>>>>>> use raw read for the missing bytes, but they are not protected by host ECC. to the NAND type of storage, is it ok or missing bytes better to be filled with 0xff?
>>>>>>>
>>>>>>> IIUC Miquèl's reply, valid behaviour is to return full OOB data in both modes. So in:
>>>>>>> ECC mode it is user bytes(corrected by ECC, read from info buffer) + ECC + missing bytes. ECC and missing bytes read in RAW mode.
>>>>>>
>>>>>> I believe the ECC bytes you'll get will be corrected.
>>>>>> You can check this by using nandflipbits in mtd-utils.
>>>>>
>>>>> Sorry, didn't get it, i'm new in NAND area. Bytes of ECC codes are available only in raw mode (at least in this NAND
>>>>> driver) also as missing bytes of OOB.
>>>>
>>>> Gasp. Yeah that's a controller limitation, okay.
>>>>
>>>>> So IIUC ECC codes are metadata to correct data bytes, and thus
>>>>> couldn't be corrected.
>>>>
>>>> We consider them metadata, but they are fully part of the ECC scheme
>>>> and thus their correction is part of the process, bitflips in the ECC
>>>> bytes will count as data bitflips actually.
>>>>
>>>> I talked a bit about ECC engines at a previous conference if it can
>>>> help:
>>>> https://elinux.org/ELC_Europe_2020_Presentations
>>>> 'Understand ECC Support for NAND Flash Devices in Linux'
>>>> And also wrote a blog post with a chapter about ECC engines:
>>>> https://bootlin.com/blog/supporting-a-misbehaving-nand-ecc-engine/
>>>
>>>
>>> Thanks for this!
>>>
>>> Thanks, Arseniy
>>
>> Hello again @Liang @Miquel!
>>
>> One more question about OOB access, as I can see current driver uses the following
>> callbacks:
>>
>>     nand->ecc.write_oob_raw = nand_write_oob_std;
>>     nand->ecc.write_oob = nand_write_oob_std;
>>
>>
>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>> can see by dumping 'data_buf' during read, physical layout of each page is the
>> following (1KB ECC):
>>
>> 0x000: [         1 KB of data        ]
>> 0x400: [ 2B user data] [ 14B ECC code]
>> 0x410: [         1 KB of data        ]    (A)
>> 0x810: [ 2B user data] [ 14B ECC code]
>> 0x820: [        32B unused           ]
>>
>>
>>
>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>> page will look like this:
>>
>> 0x000: [             0xFF            ]
>> 0x400: [           ........          ]
>> 0x7f0: [             0xFF            ]
>> 0x800: [ 00 .......................  ]
>> 0x830: [ ........................ 3f ]
>>
>> Here we have two problems:
>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>     invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>     is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>
>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>    OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>
> Hi Arseniy,
> 
> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).

IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
written data (we can't update bits from 0 to 1).

> 
> 
>>
>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>     state (I mean was written at least once) and NAND controller tries to use
>>     ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
> 
> As i said above, write_oob_raw() and write_oob() should be reworked.
> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().

Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.

Thanks, Arseniy

> 
>>     we have strange situation: OOB seems written without any errors, but we can't
>>     read this page. First idea is to write OOB data to 0x400 and 0x810 in raw mode,
>>     but this does not work - if there is some data, NAND controller will try to
>>     use ECC engine to check these user bytes on next attempt to read this page. But
>>     as these 4 bytes were written in raw mode, ECC codes are missed.
>>
>> We suggest the following thing: use only area at 0x820 for OOB (see A) - it is not covered
>> by ECC engine, so write to this zone won't conflict with ECC in future. In this case
>> we change 'meson_ooblayout_free()' function which instead of describing 2 user bytes
>> for each ECC block, will return 16 tail bytes for each ECC block.
>>
>> What to You think?
> 
> the key point is that the data 0x820-0x840 is not protected by Host ECC.so i don't think we have to change it.
> 
> Thanks,
> Liang
>>
>>
>>>
>>>>
>>>> Thanks,
>>>> Miquèl
>>
Miquel Raynal April 18, 2023, 1:25 p.m. UTC | #32
Hi Arseniy,

> >> Hello again @Liang @Miquel!
> >>
> >> One more question about OOB access, as I can see current driver uses the following
> >> callbacks:
> >>
> >>     nand->ecc.write_oob_raw = nand_write_oob_std;
> >>     nand->ecc.write_oob = nand_write_oob_std;
> >>
> >>
> >> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
> >> can see by dumping 'data_buf' during read, physical layout of each page is the
> >> following (1KB ECC):
> >>
> >> 0x000: [         1 KB of data        ]
> >> 0x400: [ 2B user data] [ 14B ECC code]
> >> 0x410: [         1 KB of data        ]    (A)
> >> 0x810: [ 2B user data] [ 14B ECC code]
> >> 0x820: [        32B unused           ]
> >>
> >>
> >>
> >> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
> >> page will look like this:
> >>
> >> 0x000: [             0xFF            ]
> >> 0x400: [           ........          ]
> >> 0x7f0: [             0xFF            ]
> >> 0x800: [ 00 .......................  ]
> >> 0x830: [ ........................ 3f ]
> >>
> >> Here we have two problems:
> >> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
> >>     invalid output, as driver relies on layout (A) from above. E.g. OOB data
> >>     is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
> >>
> >> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
> >>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>    OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
> >>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>  
> > Hi Arseniy,
> > 
> > I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).  
> 
> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
> written data (we can't update bits from 0 to 1).
> 
> > 
> >   
> >>
> >> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
> >>     state (I mean was written at least once) and NAND controller tries to use
> >>     ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus  
> > 
> > As i said above, write_oob_raw() and write_oob() should be reworked.
> > i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().  
> 
> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.

A page is written after two steps:
- loading the data into the NAND chip cache (that's when you use the
  bus)
- programming the NAND array with the data loaded in cache (that's when
  you wait)

In theory it does not matter where you write in the cache, it's regular
DRAM, you can make random writes there with the appropriate NAND
commands. Of course when using embedded hardware ECC engines, the
controllers usually expect to be fed in a certain way in order to
produce the ECC bytes and put them at the right location in cache.

And then, when you actually send the "program" command, the NAND cells
actually get programmed based on what has been loaded in cache.

Thanks,
Miquèl
Arseniy Krasnov April 18, 2023, 2:57 p.m. UTC | #33
On 18.04.2023 16:25, Miquel Raynal wrote:
> Hi Arseniy,
> 
>>>> Hello again @Liang @Miquel!
>>>>
>>>> One more question about OOB access, as I can see current driver uses the following
>>>> callbacks:
>>>>
>>>>     nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>     nand->ecc.write_oob = nand_write_oob_std;
>>>>
>>>>
>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>> following (1KB ECC):
>>>>
>>>> 0x000: [         1 KB of data        ]
>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>> 0x410: [         1 KB of data        ]    (A)
>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>> 0x820: [        32B unused           ]
>>>>
>>>>
>>>>
>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>> page will look like this:
>>>>
>>>> 0x000: [             0xFF            ]
>>>> 0x400: [           ........          ]
>>>> 0x7f0: [             0xFF            ]
>>>> 0x800: [ 00 .......................  ]
>>>> 0x830: [ ........................ 3f ]
>>>>
>>>> Here we have two problems:
>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>     invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>     is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>
>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>    OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>  
>>> Hi Arseniy,
>>>
>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).  
>>
>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>> written data (we can't update bits from 0 to 1).
>>
>>>
>>>   
>>>>
>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>     state (I mean was written at least once) and NAND controller tries to use
>>>>     ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus  
>>>
>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().  
>>
>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.

@Liang, I'll describe current test case in details:
1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
   at 0x400 and 0x810 are updated. All other bytes still 0xff.
4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
   happens because from controller point of view ECC codes are invalid for current
   data (all ECCs are 0xff). Is this behaviour is ok?
5) Ok, don't care on these ECC errors, let's go further.
6) I'm going to write same page in ECC mode - how to do it correctly? There is already
   4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).

That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
leave user's bytes untouched). In this case OOB access is free and not linked with ECC
codes which also covers data.

Thanks, Arseniy
   

> 
> A page is written after two steps:
> - loading the data into the NAND chip cache (that's when you use the
>   bus)
> - programming the NAND array with the data loaded in cache (that's when
>   you wait)
> 
> In theory it does not matter where you write in the cache, it's regular
> DRAM, you can make random writes there with the appropriate NAND
> commands. Of course when using embedded hardware ECC engines, the
> controllers usually expect to be fed in a certain way in order to
> produce the ECC bytes and put them at the right location in cache.
> 
> And then, when you actually send the "program" command, the NAND cells
> actually get programmed based on what has been loaded in cache.

Thanks for this details! Very interesting!


> 
> Thanks,
> Miquèl
Arseniy Krasnov April 18, 2023, 3:07 p.m. UTC | #34
On 18.04.2023 17:57, Arseniy Krasnov wrote:
> 
> 
> On 18.04.2023 16:25, Miquel Raynal wrote:
>> Hi Arseniy,
>>
>>>>> Hello again @Liang @Miquel!
>>>>>
>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>> callbacks:
>>>>>
>>>>>     nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>     nand->ecc.write_oob = nand_write_oob_std;
>>>>>
>>>>>
>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>> following (1KB ECC):
>>>>>
>>>>> 0x000: [         1 KB of data        ]
>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>> 0x820: [        32B unused           ]
>>>>>
>>>>>
>>>>>
>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>> page will look like this:
>>>>>
>>>>> 0x000: [             0xFF            ]
>>>>> 0x400: [           ........          ]
>>>>> 0x7f0: [             0xFF            ]
>>>>> 0x800: [ 00 .......................  ]
>>>>> 0x830: [ ........................ 3f ]
>>>>>
>>>>> Here we have two problems:
>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>     invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>     is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>
>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>    OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>    OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>  
>>>> Hi Arseniy,
>>>>
>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).  
>>>
>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>> written data (we can't update bits from 0 to 1).
>>>
>>>>
>>>>   
>>>>>
>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>     state (I mean was written at least once) and NAND controller tries to use
>>>>>     ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus  
>>>>
>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().  
>>>
>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
> 
> @Liang, I'll describe current test case in details:
> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>    at 0x400 and 0x810 are updated. All other bytes still 0xff.
> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>    happens because from controller point of view ECC codes are invalid for current
>    data (all ECCs are 0xff). Is this behaviour is ok?
> 5) Ok, don't care on these ECC errors, let's go further.
> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>    4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).
> 
> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
> codes which also covers data.
> 
> Thanks, Arseniy

For example steps 1-4 happens during work with JFFS2, it tries to write OOB, then, sometimes
later read this page in ECC mode - and get ECC errors.

Thanks, Arseniy

>    
> 
>>
>> A page is written after two steps:
>> - loading the data into the NAND chip cache (that's when you use the
>>   bus)
>> - programming the NAND array with the data loaded in cache (that's when
>>   you wait)
>>
>> In theory it does not matter where you write in the cache, it's regular
>> DRAM, you can make random writes there with the appropriate NAND
>> commands. Of course when using embedded hardware ECC engines, the
>> controllers usually expect to be fed in a certain way in order to
>> produce the ECC bytes and put them at the right location in cache.
>>
>> And then, when you actually send the "program" command, the NAND cells
>> actually get programmed based on what has been loaded in cache.
> 
> Thanks for this details! Very interesting!
> 
> 
>>
>> Thanks,
>> Miquèl
Liang Yang April 19, 2023, 3:05 a.m. UTC | #35
Hi Arseniy,

On 2023/4/18 22:57, Arseniy Krasnov wrote:
> [ EXTERNAL EMAIL ]
> 
> 
> 
> On 18.04.2023 16:25, Miquel Raynal wrote:
>> Hi Arseniy,
>>
>>>>> Hello again @Liang @Miquel!
>>>>>
>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>> callbacks:
>>>>>
>>>>>      nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>      nand->ecc.write_oob = nand_write_oob_std;
>>>>>
>>>>>
>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>> following (1KB ECC):
>>>>>
>>>>> 0x000: [         1 KB of data        ]
>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>> 0x820: [        32B unused           ]
>>>>>
>>>>>
>>>>>
>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>> page will look like this:
>>>>>
>>>>> 0x000: [             0xFF            ]
>>>>> 0x400: [           ........          ]
>>>>> 0x7f0: [             0xFF            ]
>>>>> 0x800: [ 00 .......................  ]
>>>>> 0x830: [ ........................ 3f ]
>>>>>
>>>>> Here we have two problems:
>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>      invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>      is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>
>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>     OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>   
>>>> Hi Arseniy,
>>>>
>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).
>>>
>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>> written data (we can't update bits from 0 to 1).
>>>
>>>>
>>>>    
>>>>>
>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>      state (I mean was written at least once) and NAND controller tries to use
>>>>>      ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
>>>>
>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().
>>>
>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
> 
> @Liang, I'll describe current test case in details:
> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>     at 0x400 and 0x810 are updated. All other bytes still 0xff.
> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>     happens because from controller point of view ECC codes are invalid for current
>     data (all ECCs are 0xff). Is this behaviour is ok?

Yes, it is exactly reported ECC errors.

> 5) Ok, don't care on these ECC errors, let's go further.
> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>     4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).

If step 4 has excuted "program" command at the page 
(nand_write_oob_std() does), it can't be written again before erasing 
the page(block). so we have to read the whole page in the ddr and change 
the content, erase block, write it again.

I don't think Jffs2 has the same steps (1-6) as you said above. are you 
sure that happes on Jffs2 or just an example?

> 
> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
> codes which also covers data.
> 
> Thanks, Arseniy
>     
> 
>>
>> A page is written after two steps:
>> - loading the data into the NAND chip cache (that's when you use the
>>    bus)
>> - programming the NAND array with the data loaded in cache (that's when
>>    you wait)
>>
>> In theory it does not matter where you write in the cache, it's regular
>> DRAM, you can make random writes there with the appropriate NAND
>> commands. Of course when using embedded hardware ECC engines, the
>> controllers usually expect to be fed in a certain way in order to
>> produce the ECC bytes and put them at the right location in cache.
>>
>> And then, when you actually send the "program" command, the NAND cells
>> actually get programmed based on what has been loaded in cache.
> 
> Thanks for this details! Very interesting!
> 
> 
>>
>> Thanks,
>> Miquèl
>
Arseniy Krasnov April 19, 2023, 6:41 a.m. UTC | #36
On 19.04.2023 06:05, Liang Yang wrote:
> Hi Arseniy,
> 
> On 2023/4/18 22:57, Arseniy Krasnov wrote:
>> [ EXTERNAL EMAIL ]
>>
>>
>>
>> On 18.04.2023 16:25, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>>>>> Hello again @Liang @Miquel!
>>>>>>
>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>> callbacks:
>>>>>>
>>>>>>      nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>      nand->ecc.write_oob = nand_write_oob_std;
>>>>>>
>>>>>>
>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>> following (1KB ECC):
>>>>>>
>>>>>> 0x000: [         1 KB of data        ]
>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>> 0x820: [        32B unused           ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>> page will look like this:
>>>>>>
>>>>>> 0x000: [             0xFF            ]
>>>>>> 0x400: [           ........          ]
>>>>>> 0x7f0: [             0xFF            ]
>>>>>> 0x800: [ 00 .......................  ]
>>>>>> 0x830: [ ........................ 3f ]
>>>>>>
>>>>>> Here we have two problems:
>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>      invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>      is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>
>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>     OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>   
>>>>> Hi Arseniy,
>>>>>
>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).
>>>>
>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>> written data (we can't update bits from 0 to 1).
>>>>
>>>>>
>>>>>   
>>>>>>
>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>      state (I mean was written at least once) and NAND controller tries to use
>>>>>>      ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
>>>>>
>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().
>>>>
>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
>>
>> @Liang, I'll describe current test case in details:
>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>     at 0x400 and 0x810 are updated. All other bytes still 0xff.
>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>     happens because from controller point of view ECC codes are invalid for current
>>     data (all ECCs are 0xff). Is this behaviour is ok?
> 
> Yes, it is exactly reported ECC errors.

I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the
only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
again.

> 
>> 5) Ok, don't care on these ECC errors, let's go further.
>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>     4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).
> 
> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
> 
> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?

I just checked JFFS2 mount/umount again, here is what i see:
0) First attempt to mount JFFS2.
1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
2) Umount JFFS2. Done.
3) Second attempt to mount JFFS2.
4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
5) It reads page N in ECC mode, and i get:
    jffs2: mtd->read(0x100 bytes from N) returned ECC error
6) Mount failed.

We already had problem which looks like this on another device. Solution was to use OOB area which is
not covered by ECC for JFFS2 cleanmarkers.

Thanks, Arseniy

> 
>>
>> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
>> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
>> codes which also covers data.
>>
>> Thanks, Arseniy
>>    
>>>
>>> A page is written after two steps:
>>> - loading the data into the NAND chip cache (that's when you use the
>>>    bus)
>>> - programming the NAND array with the data loaded in cache (that's when
>>>    you wait)
>>>
>>> In theory it does not matter where you write in the cache, it's regular
>>> DRAM, you can make random writes there with the appropriate NAND
>>> commands. Of course when using embedded hardware ECC engines, the
>>> controllers usually expect to be fed in a certain way in order to
>>> produce the ECC bytes and put them at the right location in cache.
>>>
>>> And then, when you actually send the "program" command, the NAND cells
>>> actually get programmed based on what has been loaded in cache.
>>
>> Thanks for this details! Very interesting!
>>
>>
>>>
>>> Thanks,
>>> Miquèl
>>
Arseniy Krasnov April 20, 2023, 9:37 a.m. UTC | #37
On 19.04.2023 09:41, Arseniy Krasnov wrote:
> 
> 
> On 19.04.2023 06:05, Liang Yang wrote:
>> Hi Arseniy,
>>
>> On 2023/4/18 22:57, Arseniy Krasnov wrote:
>>> [ EXTERNAL EMAIL ]
>>>
>>>
>>>
>>> On 18.04.2023 16:25, Miquel Raynal wrote:
>>>> Hi Arseniy,
>>>>
>>>>>>> Hello again @Liang @Miquel!
>>>>>>>
>>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>>> callbacks:
>>>>>>>
>>>>>>>      nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>>      nand->ecc.write_oob = nand_write_oob_std;
>>>>>>>
>>>>>>>
>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>>> following (1KB ECC):
>>>>>>>
>>>>>>> 0x000: [         1 KB of data        ]
>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>>> 0x820: [        32B unused           ]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>>> page will look like this:
>>>>>>>
>>>>>>> 0x000: [             0xFF            ]
>>>>>>> 0x400: [           ........          ]
>>>>>>> 0x7f0: [             0xFF            ]
>>>>>>> 0x800: [ 00 .......................  ]
>>>>>>> 0x830: [ ........................ 3f ]
>>>>>>>
>>>>>>> Here we have two problems:
>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>>      invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>>      is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>>
>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>     OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>   
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).
>>>>>
>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>>> written data (we can't update bits from 0 to 1).
>>>>>
>>>>>>
>>>>>>   
>>>>>>>
>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>>      state (I mean was written at least once) and NAND controller tries to use
>>>>>>>      ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
>>>>>>
>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().
>>>>>
>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
>>>
>>> @Liang, I'll describe current test case in details:
>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>>     at 0x400 and 0x810 are updated. All other bytes still 0xff.
>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>>     happens because from controller point of view ECC codes are invalid for current
>>>     data (all ECCs are 0xff). Is this behaviour is ok?
>>
>> Yes, it is exactly reported ECC errors.
> 
> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the
> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
> again.
> 
>>
>>> 5) Ok, don't care on these ECC errors, let's go further.
>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>>     4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).
>>
>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
>>
>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?


> 
> I just checked JFFS2 mount/umount again, here is what i see:
> 0) First attempt to mount JFFS2.
> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> 2) Umount JFFS2. Done.
> 3) Second attempt to mount JFFS2.
> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> 5) It reads page N in ECC mode, and i get:
>     jffs2: mtd->read(0x100 bytes from N) returned ECC error
> 6) Mount failed.
> 
> We already had problem which looks like this on another device. Solution was to use OOB area which is
> not covered by ECC for JFFS2 cleanmarkers.
> 
> Thanks, Arseniy
> 

@Liang,

Small addition, if i'm trying to implement OOB read/write in ECC mode, then step 5) will success,
but later, JFFS2 tries to write this page (in ECC mode of course), and in this case ECC codes will
be broken, because we can't update them properly without erasing whole page.

Please take a look at this patch from my colleagues:
https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru
It is related with "We already had problem which looks like this on another device" from above:
in 'f50l1g41lb_ooblayout_free()' we reserve 2 bytes in non-ECC area for bad block markers.

Thanks, Arseniy


>>
>>>
>>> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
>>> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
>>> codes which also covers data.
>>>
>>> Thanks, Arseniy
>>>    
>>>>
>>>> A page is written after two steps:
>>>> - loading the data into the NAND chip cache (that's when you use the
>>>>    bus)
>>>> - programming the NAND array with the data loaded in cache (that's when
>>>>    you wait)
>>>>
>>>> In theory it does not matter where you write in the cache, it's regular
>>>> DRAM, you can make random writes there with the appropriate NAND
>>>> commands. Of course when using embedded hardware ECC engines, the
>>>> controllers usually expect to be fed in a certain way in order to
>>>> produce the ECC bytes and put them at the right location in cache.
>>>>
>>>> And then, when you actually send the "program" command, the NAND cells
>>>> actually get programmed based on what has been loaded in cache.
>>>
>>> Thanks for this details! Very interesting!
>>>
>>>
>>>>
>>>> Thanks,
>>>> Miquèl
>>>
Liang Yang April 26, 2023, 1:51 p.m. UTC | #38
Hi Arseniy,

On 2023/4/20 17:37, Arseniy Krasnov wrote:
> [ EXTERNAL EMAIL ]
> 
> On 19.04.2023 09:41, Arseniy Krasnov wrote:
>>
>>
>> On 19.04.2023 06:05, Liang Yang wrote:
>>> Hi Arseniy,
>>>
>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:
>>>> [ EXTERNAL EMAIL ]
>>>>
>>>>
>>>>
>>>> On 18.04.2023 16:25, Miquel Raynal wrote:
>>>>> Hi Arseniy,
>>>>>
>>>>>>>> Hello again @Liang @Miquel!
>>>>>>>>
>>>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>>>> callbacks:
>>>>>>>>
>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
>>>>>>>>
>>>>>>>>
>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>>>> following (1KB ECC):
>>>>>>>>
>>>>>>>> 0x000: [         1 KB of data        ]
>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>>>> 0x820: [        32B unused           ]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>>>> page will look like this:
>>>>>>>>
>>>>>>>> 0x000: [             0xFF            ]
>>>>>>>> 0x400: [           ........          ]
>>>>>>>> 0x7f0: [             0xFF            ]
>>>>>>>> 0x800: [ 00 .......................  ]
>>>>>>>> 0x830: [ ........................ 3f ]
>>>>>>>>
>>>>>>>> Here we have two problems:
>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>>>
>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>
>>>>>>> Hi Arseniy,
>>>>>>>
>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).
>>>>>>
>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>>>> written data (we can't update bits from 0 to 1).
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
>>>>>>>
>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().
>>>>>>
>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
>>>>
>>>> @Liang, I'll describe current test case in details:
>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>>>      happens because from controller point of view ECC codes are invalid for current
>>>>      data (all ECCs are 0xff). Is this behaviour is ok?
>>>
>>> Yes, it is exactly reported ECC errors.
>>
>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the

Of course, there is no ECC parity bytes in it; or raw write the data 
with the ECC parity bytes per the layout (A) you describe above.

>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
>> again.

yes, you are right.

>>
>>>
>>>> 5) Ok, don't care on these ECC errors, let's go further.
>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).
>>>
>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
>>>
>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?
> 
> 
>>
>> I just checked JFFS2 mount/umount again, here is what i see:
>> 0) First attempt to mount JFFS2.
>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>> 2) Umount JFFS2. Done.
>> 3) Second attempt to mount JFFS2.
>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>> 5) It reads page N in ECC mode, and i get:
>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>> 6) Mount failed.
>>
>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>> not covered by ECC for JFFS2 cleanmarkers.

ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
does it have to use raw write/read on step 1) and 4)?

>>
>> Thanks, Arseniy
>>
> 
> @Liang,
> 
> Small addition, if i'm trying to implement OOB read/write in ECC mode, then step 5) will success,
> but later, JFFS2 tries to write this page (in ECC mode of course), and in this case ECC codes will
> be broken, because we can't update them properly without erasing whole page.
> 
> Please take a look at this patch from my colleagues:
> https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru
> It is related with "We already had problem which looks like this on another device" from above:
> in 'f50l1g41lb_ooblayout_free()' we reserve 2 bytes in non-ECC area for bad block markers.

It is about the SPI NAND and use another controller called spifc.
but in meson nfc, it is heavily depended on the pre-created BBT in NAND 
device.


> 
> Thanks, Arseniy
> 
> 
>>>
>>>>
>>>> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
>>>> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
>>>> codes which also covers data.
>>>>
>>>> Thanks, Arseniy
>>>>
>>>>>
>>>>> A page is written after two steps:
>>>>> - loading the data into the NAND chip cache (that's when you use the
>>>>>     bus)
>>>>> - programming the NAND array with the data loaded in cache (that's when
>>>>>     you wait)
>>>>>
>>>>> In theory it does not matter where you write in the cache, it's regular
>>>>> DRAM, you can make random writes there with the appropriate NAND
>>>>> commands. Of course when using embedded hardware ECC engines, the
>>>>> controllers usually expect to be fed in a certain way in order to
>>>>> produce the ECC bytes and put them at the right location in cache.
>>>>>
>>>>> And then, when you actually send the "program" command, the NAND cells
>>>>> actually get programmed based on what has been loaded in cache.
>>>>
>>>> Thanks for this details! Very interesting!
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Miquèl
>>>>
Arseniy Krasnov April 26, 2023, 2:46 p.m. UTC | #39
On 26.04.2023 16:51, Liang Yang wrote:
> Hi Arseniy,
> 
> On 2023/4/20 17:37, Arseniy Krasnov wrote:
>> [ EXTERNAL EMAIL ]
>>
>> On 19.04.2023 09:41, Arseniy Krasnov wrote:
>>>
>>>
>>> On 19.04.2023 06:05, Liang Yang wrote:
>>>> Hi Arseniy,
>>>>
>>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:
>>>>> [ EXTERNAL EMAIL ]
>>>>>
>>>>>
>>>>>
>>>>> On 18.04.2023 16:25, Miquel Raynal wrote:
>>>>>> Hi Arseniy,
>>>>>>
>>>>>>>>> Hello again @Liang @Miquel!
>>>>>>>>>
>>>>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>>>>> callbacks:
>>>>>>>>>
>>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>>>>> following (1KB ECC):
>>>>>>>>>
>>>>>>>>> 0x000: [         1 KB of data        ]
>>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>>>>> 0x820: [        32B unused           ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>>>>> page will look like this:
>>>>>>>>>
>>>>>>>>> 0x000: [             0xFF            ]
>>>>>>>>> 0x400: [           ........          ]
>>>>>>>>> 0x7f0: [             0xFF            ]
>>>>>>>>> 0x800: [ 00 .......................  ]
>>>>>>>>> 0x830: [ ........................ 3f ]
>>>>>>>>>
>>>>>>>>> Here we have two problems:
>>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>>>>
>>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>
>>>>>>>> Hi Arseniy,
>>>>>>>>
>>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).
>>>>>>>
>>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>>>>> written data (we can't update bits from 0 to 1).
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
>>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus
>>>>>>>>
>>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().
>>>>>>>
>>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.
>>>>>
>>>>> @Liang, I'll describe current test case in details:
>>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
>>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>>>>      happens because from controller point of view ECC codes are invalid for current
>>>>>      data (all ECCs are 0xff). Is this behaviour is ok?
>>>>
>>>> Yes, it is exactly reported ECC errors.
>>>
>>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the
> 
> Of course, there is no ECC parity bytes in it; or raw write the data with the ECC parity bytes per the layout (A) you describe above.
> 

But don't it looks like strange? Just writing OOB makes page unreadable? May be it is better to move OOB data
out of ECC area as I suggested in v2?

>>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
>>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
>>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
>>> again.
> 
> yes, you are right.
> 
>>>
>>>>
>>>>> 5) Ok, don't care on these ECC errors, let's go further.
>>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).
>>>>
>>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
>>>>
>>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?
>>
>>
>>>
>>> I just checked JFFS2 mount/umount again, here is what i see:
>>> 0) First attempt to mount JFFS2.
>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>> 2) Umount JFFS2. Done.
>>> 3) Second attempt to mount JFFS2.
>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>> 5) It reads page N in ECC mode, and i get:
>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>> 6) Mount failed.
>>>
>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>> not covered by ECC for JFFS2 cleanmarkers.
> 
> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> does it have to use raw write/read on step 1) and 4)?
> 

If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
writes to such pages later) - we can't update ECC codes properly without erasing whole page.
Write operation will be done without problem, but read will trigger ECC errors due to broken
ECC codes.

In general problem that we discuss is that in current implementation data and OOB conflicts
with each other by sharing same ECC codes, these ECC codes could be written only once (without
erasing), while data and OOB has different callbacks to access and thus supposed to work
separately.

What do You think?

Thanks, Arseniy

>>>
>>> Thanks, Arseniy
>>>
>>
>> @Liang,
>>
>> Small addition, if i'm trying to implement OOB read/write in ECC mode, then step 5) will success,
>> but later, JFFS2 tries to write this page (in ECC mode of course), and in this case ECC codes will
>> be broken, because we can't update them properly without erasing whole page.
>>
>> Please take a look at this patch from my colleagues:
>> https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru
>> It is related with "We already had problem which looks like this on another device" from above:
>> in 'f50l1g41lb_ooblayout_free()' we reserve 2 bytes in non-ECC area for bad block markers.
> 
> It is about the SPI NAND and use another controller called spifc.
> but in meson nfc, it is heavily depended on the pre-created BBT in NAND device.
> 
> 
>>
>> Thanks, Arseniy
>>
>>
>>>>
>>>>>
>>>>> That's why I asked Your opinion about moving OOB data to nonprotected by ECC area (and
>>>>> leave user's bytes untouched). In this case OOB access is free and not linked with ECC
>>>>> codes which also covers data.
>>>>>
>>>>> Thanks, Arseniy
>>>>>
>>>>>>
>>>>>> A page is written after two steps:
>>>>>> - loading the data into the NAND chip cache (that's when you use the
>>>>>>     bus)
>>>>>> - programming the NAND array with the data loaded in cache (that's when
>>>>>>     you wait)
>>>>>>
>>>>>> In theory it does not matter where you write in the cache, it's regular
>>>>>> DRAM, you can make random writes there with the appropriate NAND
>>>>>> commands. Of course when using embedded hardware ECC engines, the
>>>>>> controllers usually expect to be fed in a certain way in order to
>>>>>> produce the ECC bytes and put them at the right location in cache.
>>>>>>
>>>>>> And then, when you actually send the "program" command, the NAND cells
>>>>>> actually get programmed based on what has been loaded in cache.
>>>>>
>>>>> Thanks for this details! Very interesting!
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Miquèl
>>>>>
Miquel Raynal May 2, 2023, 9:59 a.m. UTC | #40
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Wed, 26 Apr 2023 17:46:19 +0300:

> On 26.04.2023 16:51, Liang Yang wrote:
> > Hi Arseniy,
> > 
> > On 2023/4/20 17:37, Arseniy Krasnov wrote:  
> >> [ EXTERNAL EMAIL ]
> >>
> >> On 19.04.2023 09:41, Arseniy Krasnov wrote:  
> >>>
> >>>
> >>> On 19.04.2023 06:05, Liang Yang wrote:  
> >>>> Hi Arseniy,
> >>>>
> >>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:  
> >>>>> [ EXTERNAL EMAIL ]
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 18.04.2023 16:25, Miquel Raynal wrote:  
> >>>>>> Hi Arseniy,
> >>>>>>  
> >>>>>>>>> Hello again @Liang @Miquel!
> >>>>>>>>>
> >>>>>>>>> One more question about OOB access, as I can see current driver uses the following
> >>>>>>>>> callbacks:
> >>>>>>>>>
> >>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
> >>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
> >>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
> >>>>>>>>> following (1KB ECC):
> >>>>>>>>>
> >>>>>>>>> 0x000: [         1 KB of data        ]
> >>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
> >>>>>>>>> 0x410: [         1 KB of data        ]    (A)
> >>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
> >>>>>>>>> 0x820: [        32B unused           ]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
> >>>>>>>>> page will look like this:
> >>>>>>>>>
> >>>>>>>>> 0x000: [             0xFF            ]
> >>>>>>>>> 0x400: [           ........          ]
> >>>>>>>>> 0x7f0: [             0xFF            ]
> >>>>>>>>> 0x800: [ 00 .......................  ]
> >>>>>>>>> 0x830: [ ........................ 3f ]
> >>>>>>>>>
> >>>>>>>>> Here we have two problems:
> >>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
> >>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
> >>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
> >>>>>>>>>
> >>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
> >>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
> >>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>  
> >>>>>>>> Hi Arseniy,
> >>>>>>>>
> >>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).  
> >>>>>>>
> >>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
> >>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
> >>>>>>> written data (we can't update bits from 0 to 1).
> >>>>>>>  
> >>>>>>>>
> >>>>>>>>  
> >>>>>>>>>
> >>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
> >>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
> >>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus  
> >>>>>>>>
> >>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
> >>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().  
> >>>>>>>
> >>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
> >>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.  
> >>>>>
> >>>>> @Liang, I'll describe current test case in details:
> >>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
> >>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
> >>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
> >>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
> >>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
> >>>>>      happens because from controller point of view ECC codes are invalid for current
> >>>>>      data (all ECCs are 0xff). Is this behaviour is ok?  
> >>>>
> >>>> Yes, it is exactly reported ECC errors.  
> >>>
> >>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the  
> > 
> > Of course, there is no ECC parity bytes in it; or raw write the data with the ECC parity bytes per the layout (A) you describe above.
> >   
> 
> But don't it looks like strange? Just writing OOB makes page unreadable? May be it is better to move OOB data
> out of ECC area as I suggested in v2?
> 
> >>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
> >>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
> >>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
> >>> again.  
> > 
> > yes, you are right.
> >   
> >>>  
> >>>>  
> >>>>> 5) Ok, don't care on these ECC errors, let's go further.
> >>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
> >>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).  
> >>>>
> >>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
> >>>>
> >>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?  
> >>
> >>  
> >>>
> >>> I just checked JFFS2 mount/umount again, here is what i see:
> >>> 0) First attempt to mount JFFS2.
> >>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>> 2) Umount JFFS2. Done.
> >>> 3) Second attempt to mount JFFS2.
> >>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>> 5) It reads page N in ECC mode, and i get:
> >>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>> 6) Mount failed.
> >>>
> >>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>> not covered by ECC for JFFS2 cleanmarkers.  
> > 
> > ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> > does it have to use raw write/read on step 1) and 4)?
> >   
> 
> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> Write operation will be done without problem, but read will trigger ECC errors due to broken
> ECC codes.
> 
> In general problem that we discuss is that in current implementation data and OOB conflicts
> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> erasing), while data and OOB has different callbacks to access and thus supposed to work
> separately.

The fact that there might be helpers just for writing OOB areas or just
in-band areas are optimizations. NAND pages are meant to be written a
single time, no matter what portion you write. In some cases, it is
possible to perform subpage writes if the chip supports it. Pages may
be split into several areas which cover a partial in-band area *and* a
partial OOB area. If you write into the in-band *or* out-of-band areas
of a given subpage, you *cannot* write the other part later without
erasing.

Thanks,
Miquèl
Arseniy Krasnov May 2, 2023, 10:11 a.m. UTC | #41
On 02.05.2023 12:59, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Wed, 26 Apr 2023 17:46:19 +0300:
> 
>> On 26.04.2023 16:51, Liang Yang wrote:
>>> Hi Arseniy,
>>>
>>> On 2023/4/20 17:37, Arseniy Krasnov wrote:  
>>>> [ EXTERNAL EMAIL ]
>>>>
>>>> On 19.04.2023 09:41, Arseniy Krasnov wrote:  
>>>>>
>>>>>
>>>>> On 19.04.2023 06:05, Liang Yang wrote:  
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:  
>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 18.04.2023 16:25, Miquel Raynal wrote:  
>>>>>>>> Hi Arseniy,
>>>>>>>>  
>>>>>>>>>>> Hello again @Liang @Miquel!
>>>>>>>>>>>
>>>>>>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>>>>>>> callbacks:
>>>>>>>>>>>
>>>>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>>>>>>> following (1KB ECC):
>>>>>>>>>>>
>>>>>>>>>>> 0x000: [         1 KB of data        ]
>>>>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>>>>>>> 0x820: [        32B unused           ]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>>>>>>> page will look like this:
>>>>>>>>>>>
>>>>>>>>>>> 0x000: [             0xFF            ]
>>>>>>>>>>> 0x400: [           ........          ]
>>>>>>>>>>> 0x7f0: [             0xFF            ]
>>>>>>>>>>> 0x800: [ 00 .......................  ]
>>>>>>>>>>> 0x830: [ ........................ 3f ]
>>>>>>>>>>>
>>>>>>>>>>> Here we have two problems:
>>>>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>>>>>>
>>>>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>  
>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>
>>>>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).  
>>>>>>>>>
>>>>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>>>>>>> written data (we can't update bits from 0 to 1).
>>>>>>>>>  
>>>>>>>>>>
>>>>>>>>>>  
>>>>>>>>>>>
>>>>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
>>>>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus  
>>>>>>>>>>
>>>>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().  
>>>>>>>>>
>>>>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.  
>>>>>>>
>>>>>>> @Liang, I'll describe current test case in details:
>>>>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>>>>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>>>>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>>>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
>>>>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>>>>>>      happens because from controller point of view ECC codes are invalid for current
>>>>>>>      data (all ECCs are 0xff). Is this behaviour is ok?  
>>>>>>
>>>>>> Yes, it is exactly reported ECC errors.  
>>>>>
>>>>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the  
>>>
>>> Of course, there is no ECC parity bytes in it; or raw write the data with the ECC parity bytes per the layout (A) you describe above.
>>>   
>>
>> But don't it looks like strange? Just writing OOB makes page unreadable? May be it is better to move OOB data
>> out of ECC area as I suggested in v2?
>>
>>>>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
>>>>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
>>>>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
>>>>> again.  
>>>
>>> yes, you are right.
>>>   
>>>>>  
>>>>>>  
>>>>>>> 5) Ok, don't care on these ECC errors, let's go further.
>>>>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>>>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).  
>>>>>>
>>>>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
>>>>>>
>>>>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?  
>>>>
>>>>  
>>>>>
>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>> 0) First attempt to mount JFFS2.
>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>> 2) Umount JFFS2. Done.
>>>>> 3) Second attempt to mount JFFS2.
>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>> 6) Mount failed.
>>>>>
>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>> not covered by ECC for JFFS2 cleanmarkers.  
>>>
>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>> does it have to use raw write/read on step 1) and 4)?
>>>   
>>
>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>> ECC codes.
>>
>> In general problem that we discuss is that in current implementation data and OOB conflicts
>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>> separately.
> 
> The fact that there might be helpers just for writing OOB areas or just
> in-band areas are optimizations. NAND pages are meant to be written a
> single time, no matter what portion you write. In some cases, it is
> possible to perform subpage writes if the chip supports it. Pages may
> be split into several areas which cover a partial in-band area *and* a
> partial OOB area. If you write into the in-band *or* out-of-band areas
> of a given subpage, you *cannot* write the other part later without

Thanks for details! So in case of JFFS2 it looks like strange, that it tries
to write page after writing clean markers to it before? In the old vendor's
driver OOB write callback is suppressed by return 0 always and JFFS2 works
correctly.

Thanks, Arseniy

> erasing.
> 
> Thanks,
> Miquèl
Miquel Raynal May 2, 2023, 11:27 a.m. UTC | #42
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 13:11:38 +0300:

> On 02.05.2023 12:59, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Wed, 26 Apr 2023 17:46:19 +0300:
> >   
> >> On 26.04.2023 16:51, Liang Yang wrote:  
> >>> Hi Arseniy,
> >>>
> >>> On 2023/4/20 17:37, Arseniy Krasnov wrote:    
> >>>> [ EXTERNAL EMAIL ]
> >>>>
> >>>> On 19.04.2023 09:41, Arseniy Krasnov wrote:    
> >>>>>
> >>>>>
> >>>>> On 19.04.2023 06:05, Liang Yang wrote:    
> >>>>>> Hi Arseniy,
> >>>>>>
> >>>>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:    
> >>>>>>> [ EXTERNAL EMAIL ]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 18.04.2023 16:25, Miquel Raynal wrote:    
> >>>>>>>> Hi Arseniy,
> >>>>>>>>    
> >>>>>>>>>>> Hello again @Liang @Miquel!
> >>>>>>>>>>>
> >>>>>>>>>>> One more question about OOB access, as I can see current driver uses the following
> >>>>>>>>>>> callbacks:
> >>>>>>>>>>>
> >>>>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
> >>>>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
> >>>>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
> >>>>>>>>>>> following (1KB ECC):
> >>>>>>>>>>>
> >>>>>>>>>>> 0x000: [         1 KB of data        ]
> >>>>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
> >>>>>>>>>>> 0x410: [         1 KB of data        ]    (A)
> >>>>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
> >>>>>>>>>>> 0x820: [        32B unused           ]
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
> >>>>>>>>>>> page will look like this:
> >>>>>>>>>>>
> >>>>>>>>>>> 0x000: [             0xFF            ]
> >>>>>>>>>>> 0x400: [           ........          ]
> >>>>>>>>>>> 0x7f0: [             0xFF            ]
> >>>>>>>>>>> 0x800: [ 00 .......................  ]
> >>>>>>>>>>> 0x830: [ ........................ 3f ]
> >>>>>>>>>>>
> >>>>>>>>>>> Here we have two problems:
> >>>>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
> >>>>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
> >>>>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
> >>>>>>>>>>>
> >>>>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
> >>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
> >>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
> >>>>>>>>>>>    
> >>>>>>>>>> Hi Arseniy,
> >>>>>>>>>>
> >>>>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).    
> >>>>>>>>>
> >>>>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
> >>>>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
> >>>>>>>>> written data (we can't update bits from 0 to 1).
> >>>>>>>>>    
> >>>>>>>>>>
> >>>>>>>>>>    
> >>>>>>>>>>>
> >>>>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
> >>>>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
> >>>>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus    
> >>>>>>>>>>
> >>>>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
> >>>>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().    
> >>>>>>>>>
> >>>>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
> >>>>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.    
> >>>>>>>
> >>>>>>> @Liang, I'll describe current test case in details:
> >>>>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
> >>>>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
> >>>>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
> >>>>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
> >>>>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
> >>>>>>>      happens because from controller point of view ECC codes are invalid for current
> >>>>>>>      data (all ECCs are 0xff). Is this behaviour is ok?    
> >>>>>>
> >>>>>> Yes, it is exactly reported ECC errors.    
> >>>>>
> >>>>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the    
> >>>
> >>> Of course, there is no ECC parity bytes in it; or raw write the data with the ECC parity bytes per the layout (A) you describe above.
> >>>     
> >>
> >> But don't it looks like strange? Just writing OOB makes page unreadable? May be it is better to move OOB data
> >> out of ECC area as I suggested in v2?
> >>  
> >>>>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
> >>>>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
> >>>>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
> >>>>> again.    
> >>>
> >>> yes, you are right.
> >>>     
> >>>>>    
> >>>>>>    
> >>>>>>> 5) Ok, don't care on these ECC errors, let's go further.
> >>>>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
> >>>>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).    
> >>>>>>
> >>>>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
> >>>>>>
> >>>>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?    
> >>>>
> >>>>    
> >>>>>
> >>>>> I just checked JFFS2 mount/umount again, here is what i see:
> >>>>> 0) First attempt to mount JFFS2.
> >>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>>>> 2) Umount JFFS2. Done.
> >>>>> 3) Second attempt to mount JFFS2.
> >>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>>>> 5) It reads page N in ECC mode, and i get:
> >>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>>>> 6) Mount failed.
> >>>>>
> >>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>>>> not covered by ECC for JFFS2 cleanmarkers.    
> >>>
> >>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> >>> does it have to use raw write/read on step 1) and 4)?
> >>>     
> >>
> >> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> >> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> >> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> >> Write operation will be done without problem, but read will trigger ECC errors due to broken
> >> ECC codes.
> >>
> >> In general problem that we discuss is that in current implementation data and OOB conflicts
> >> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> >> erasing), while data and OOB has different callbacks to access and thus supposed to work
> >> separately.  
> > 
> > The fact that there might be helpers just for writing OOB areas or just
> > in-band areas are optimizations. NAND pages are meant to be written a
> > single time, no matter what portion you write. In some cases, it is
> > possible to perform subpage writes if the chip supports it. Pages may
> > be split into several areas which cover a partial in-band area *and* a
> > partial OOB area. If you write into the in-band *or* out-of-band areas
> > of a given subpage, you *cannot* write the other part later without  
> 
> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
> to write page after writing clean markers to it before? In the old vendor's
> driver OOB write callback is suppressed by return 0 always and JFFS2 works
> correctly.

Can you point the code you're mentioning? (both what JFFS2 which looks
strange to you and the old vendor hack)


Thanks,
Miquèl
Arseniy Krasnov May 2, 2023, 11:32 a.m. UTC | #43
On 02.05.2023 14:27, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 13:11:38 +0300:
> 
>> On 02.05.2023 12:59, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Wed, 26 Apr 2023 17:46:19 +0300:
>>>   
>>>> On 26.04.2023 16:51, Liang Yang wrote:  
>>>>> Hi Arseniy,
>>>>>
>>>>> On 2023/4/20 17:37, Arseniy Krasnov wrote:    
>>>>>> [ EXTERNAL EMAIL ]
>>>>>>
>>>>>> On 19.04.2023 09:41, Arseniy Krasnov wrote:    
>>>>>>>
>>>>>>>
>>>>>>> On 19.04.2023 06:05, Liang Yang wrote:    
>>>>>>>> Hi Arseniy,
>>>>>>>>
>>>>>>>> On 2023/4/18 22:57, Arseniy Krasnov wrote:    
>>>>>>>>> [ EXTERNAL EMAIL ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 18.04.2023 16:25, Miquel Raynal wrote:    
>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>    
>>>>>>>>>>>>> Hello again @Liang @Miquel!
>>>>>>>>>>>>>
>>>>>>>>>>>>> One more question about OOB access, as I can see current driver uses the following
>>>>>>>>>>>>> callbacks:
>>>>>>>>>>>>>
>>>>>>>>>>>>>       nand->ecc.write_oob_raw = nand_write_oob_std;
>>>>>>>>>>>>>       nand->ecc.write_oob = nand_write_oob_std;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Function 'nand_write_oob_std()' writes data to the end of the page. But as I
>>>>>>>>>>>>> can see by dumping 'data_buf' during read, physical layout of each page is the
>>>>>>>>>>>>> following (1KB ECC):
>>>>>>>>>>>>>
>>>>>>>>>>>>> 0x000: [         1 KB of data        ]
>>>>>>>>>>>>> 0x400: [ 2B user data] [ 14B ECC code]
>>>>>>>>>>>>> 0x410: [         1 KB of data        ]    (A)
>>>>>>>>>>>>> 0x810: [ 2B user data] [ 14B ECC code]
>>>>>>>>>>>>> 0x820: [        32B unused           ]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, after 'nand_write_oob_std()' (let data be sequence from [0x0 ... 0x3f]),
>>>>>>>>>>>>> page will look like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 0x000: [             0xFF            ]
>>>>>>>>>>>>> 0x400: [           ........          ]
>>>>>>>>>>>>> 0x7f0: [             0xFF            ]
>>>>>>>>>>>>> 0x800: [ 00 .......................  ]
>>>>>>>>>>>>> 0x830: [ ........................ 3f ]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here we have two problems:
>>>>>>>>>>>>> 1) Attempt to display raw data by 'nanddump' utility produces a little bit
>>>>>>>>>>>>>       invalid output, as driver relies on layout (A) from above. E.g. OOB data
>>>>>>>>>>>>>       is at 0x400 and 0x810. Here is an example (attempt to write 0x11 0x22 0x33 0x44):
>>>>>>>>>>>>>
>>>>>>>>>>>>> 0x000007f0: 11 22 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |."..............|
>>>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>>>      OOB Data: 33 44 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |3D..............|
>>>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>>>      OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  |................|
>>>>>>>>>>>>>    
>>>>>>>>>>>> Hi Arseniy,
>>>>>>>>>>>>
>>>>>>>>>>>> I realized the write_oob_raw() and write_oob() are wrong in meson_nand.c. I suggest both of them should be reworked and follow the format of meson nand controller. i.e. firstly format the data in Layout (A) and then write. reading is firstly reading the data of layout (A) and then compost the layout (B).    
>>>>>>>>>>>
>>>>>>>>>>> IIUC after such writing only OOB (e.g. user bytes) according layout (A), hw will also write ECC codes, so
>>>>>>>>>>> it will be impossible to write data to this page later, because we cannot update ECC codes properly for the newly
>>>>>>>>>>> written data (we can't update bits from 0 to 1).
>>>>>>>>>>>    
>>>>>>>>>>>>
>>>>>>>>>>>>    
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Attempt to read data in ECC mode will fail, because IIUC page is in dirty
>>>>>>>>>>>>>       state (I mean was written at least once) and NAND controller tries to use
>>>>>>>>>>>>>       ECC codes at 0x400 and 0x810, which are obviously broken in this case. Thus    
>>>>>>>>>>>>
>>>>>>>>>>>> As i said above, write_oob_raw() and write_oob() should be reworked.
>>>>>>>>>>>> i don't know what do you mean page was written at least once. anyway the page should be written once, even just write_oob_raw().    
>>>>>>>>>>>
>>>>>>>>>>> Sorry, You mean that after OOB write, we cannot write to the data area (e.g. 0x0 .. 0x810) until page will be erased? For example
>>>>>>>>>>> JFFS2 writes to OOB own markers, then it tries to write to the data area of such page.    
>>>>>>>>>
>>>>>>>>> @Liang, I'll describe current test case in details:
>>>>>>>>> 1) I have erased page, I can read it in  both raw and ecc modes - no problem (it is full of 0xFF).
>>>>>>>>> 2) I (JFFS2 for example) want to write only OOB - let it be clean markers.
>>>>>>>>> 3) I use raw write to the needed page (please correct me if i'm wrong). Four bytes
>>>>>>>>>      at 0x400 and 0x810 are updated. All other bytes still 0xff.
>>>>>>>>> 4) Now, when i'm trying to read this page in ECC mode, I get ECC errors: IIUC this
>>>>>>>>>      happens because from controller point of view ECC codes are invalid for current
>>>>>>>>>      data (all ECCs are 0xff). Is this behaviour is ok?    
>>>>>>>>
>>>>>>>> Yes, it is exactly reported ECC errors.    
>>>>>>>
>>>>>>> I see, so if we write OOB (e.g. using raw mode), there is no way to read this page in ECC mode later? And the    
>>>>>
>>>>> Of course, there is no ECC parity bytes in it; or raw write the data with the ECC parity bytes per the layout (A) you describe above.
>>>>>     
>>>>
>>>> But don't it looks like strange? Just writing OOB makes page unreadable? May be it is better to move OOB data
>>>> out of ECC area as I suggested in v2?
>>>>  
>>>>>>> only way to make it readable is to write it in ECC mode, but before this write, we need to read it's
>>>>>>> user's byte (from previous OOB write) in raw mode, put it to info buf (as user's bytes) and write this page. In this
>>>>>>> case NAND controller will generate ECC codes including user's byte and page become readable in ECC mode
>>>>>>> again.    
>>>>>
>>>>> yes, you are right.
>>>>>     
>>>>>>>    
>>>>>>>>    
>>>>>>>>> 5) Ok, don't care on these ECC errors, let's go further.
>>>>>>>>> 6) I'm going to write same page in ECC mode - how to do it correctly? There is already
>>>>>>>>>      4 OOB bytes, considered to be covered by ECC (but in fact now - ECC area is FFed).    
>>>>>>>>
>>>>>>>> If step 4 has excuted "program" command at the page (nand_write_oob_std() does), it can't be written again before erasing the page(block). so we have to read the whole page in the ddr and change the content, erase block, write it again.
>>>>>>>>
>>>>>>>> I don't think Jffs2 has the same steps (1-6) as you said above. are you sure that happes on Jffs2 or just an example?    
>>>>>>
>>>>>>    
>>>>>>>
>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>> 2) Umount JFFS2. Done.
>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>> 6) Mount failed.
>>>>>>>
>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>> not covered by ECC for JFFS2 cleanmarkers.    
>>>>>
>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>     
>>>>
>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>> ECC codes.
>>>>
>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>> separately.  
>>>
>>> The fact that there might be helpers just for writing OOB areas or just
>>> in-band areas are optimizations. NAND pages are meant to be written a
>>> single time, no matter what portion you write. In some cases, it is
>>> possible to perform subpage writes if the chip supports it. Pages may
>>> be split into several areas which cover a partial in-band area *and* a
>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>> of a given subpage, you *cannot* write the other part later without  
>>
>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>> to write page after writing clean markers to it before? In the old vendor's
>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>> correctly.
> 
> Can you point the code you're mentioning? (both what JFFS2 which looks
> strange to you and the old vendor hack)

Here is version of the old vendor's driver:

https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260

In my version there is no BUG() there, but it is same driver for the same chip.

About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
may be it is unexpected behaviour of JFFS2?

Thanks, Arseniy

> 
> 
> Thanks,
> Miquèl
Miquel Raynal May 2, 2023, 12:17 p.m. UTC | #44
Hi Arseniy,

Richard, your input is welcome below :-)

> >>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
> >>>>>>> 0) First attempt to mount JFFS2.
> >>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>>>>>> 2) Umount JFFS2. Done.
> >>>>>>> 3) Second attempt to mount JFFS2.
> >>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>>>>>> 5) It reads page N in ECC mode, and i get:
> >>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>>>>>> 6) Mount failed.
> >>>>>>>
> >>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>>>>>> not covered by ECC for JFFS2 cleanmarkers.      
> >>>>>
> >>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> >>>>> does it have to use raw write/read on step 1) and 4)?
> >>>>>       
> >>>>
> >>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> >>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> >>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> >>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
> >>>> ECC codes.
> >>>>
> >>>> In general problem that we discuss is that in current implementation data and OOB conflicts
> >>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> >>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
> >>>> separately.    
> >>>
> >>> The fact that there might be helpers just for writing OOB areas or just
> >>> in-band areas are optimizations. NAND pages are meant to be written a
> >>> single time, no matter what portion you write. In some cases, it is
> >>> possible to perform subpage writes if the chip supports it. Pages may
> >>> be split into several areas which cover a partial in-band area *and* a
> >>> partial OOB area. If you write into the in-band *or* out-of-band areas
> >>> of a given subpage, you *cannot* write the other part later without    
> >>
> >> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
> >> to write page after writing clean markers to it before? In the old vendor's
> >> driver OOB write callback is suppressed by return 0 always and JFFS2 works
> >> correctly.  
> > 
> > Can you point the code you're mentioning? (both what JFFS2 which looks
> > strange to you and the old vendor hack)  
> 
> Here is version of the old vendor's driver:
> 
> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
> 
> In my version there is no BUG() there, but it is same driver for the same chip.
> 
> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
> may be it is unexpected behaviour of JFFS2?

TBH I am not knowledgeable about JFFS2, maybe Richard can help here.

Are you sure you flash is recognized by JFFS2 as being a NAND device?
Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
cleanmarker seem to be discarded when using a NAND device, and
recognizing the device as a NAND device requires the above option to be
set apparently.

Thanks,
Miquèl
Arseniy Krasnov May 2, 2023, 12:24 p.m. UTC | #45
On 02.05.2023 15:17, Miquel Raynal wrote:
> Hi Arseniy,
> 
> Richard, your input is welcome below :-)
> 
>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>>>> 2) Umount JFFS2. Done.
>>>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>>>> 6) Mount failed.
>>>>>>>>>
>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.      
>>>>>>>
>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>>>       
>>>>>>
>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>>>> ECC codes.
>>>>>>
>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>>>> separately.    
>>>>>
>>>>> The fact that there might be helpers just for writing OOB areas or just
>>>>> in-band areas are optimizations. NAND pages are meant to be written a
>>>>> single time, no matter what portion you write. In some cases, it is
>>>>> possible to perform subpage writes if the chip supports it. Pages may
>>>>> be split into several areas which cover a partial in-band area *and* a
>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>>>> of a given subpage, you *cannot* write the other part later without    
>>>>
>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>>>> to write page after writing clean markers to it before? In the old vendor's
>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>>>> correctly.  
>>>
>>> Can you point the code you're mentioning? (both what JFFS2 which looks
>>> strange to you and the old vendor hack)  
>>
>> Here is version of the old vendor's driver:
>>
>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
>>
>> In my version there is no BUG() there, but it is same driver for the same chip.
>>
>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
>> may be it is unexpected behaviour of JFFS2?
> 
> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
> 
> Are you sure you flash is recognized by JFFS2 as being a NAND device?
> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
> cleanmarker seem to be discarded when using a NAND device, and
> recognizing the device as a NAND device requires the above option to be
> set apparently.

Yes, I have

CONFIG_JFFS2_FS_WRITEBUFFER=y

And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
cleanmarkers are not discarded with NAND device. 

Thanks, Arseniy

> 
> Thanks,
> Miquèl
Miquel Raynal May 2, 2023, 1:05 p.m. UTC | #46
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:

> On 02.05.2023 15:17, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > Richard, your input is welcome below :-)
> >   
> >>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
> >>>>>>>>> 0) First attempt to mount JFFS2.
> >>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>>>>>>>> 2) Umount JFFS2. Done.
> >>>>>>>>> 3) Second attempt to mount JFFS2.
> >>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>>>>>>>> 5) It reads page N in ECC mode, and i get:
> >>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>>>>>>>> 6) Mount failed.
> >>>>>>>>>
> >>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.        
> >>>>>>>
> >>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> >>>>>>> does it have to use raw write/read on step 1) and 4)?
> >>>>>>>         
> >>>>>>
> >>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> >>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> >>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> >>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
> >>>>>> ECC codes.
> >>>>>>
> >>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
> >>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> >>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
> >>>>>> separately.      
> >>>>>
> >>>>> The fact that there might be helpers just for writing OOB areas or just
> >>>>> in-band areas are optimizations. NAND pages are meant to be written a
> >>>>> single time, no matter what portion you write. In some cases, it is
> >>>>> possible to perform subpage writes if the chip supports it. Pages may
> >>>>> be split into several areas which cover a partial in-band area *and* a
> >>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
> >>>>> of a given subpage, you *cannot* write the other part later without      
> >>>>
> >>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
> >>>> to write page after writing clean markers to it before? In the old vendor's
> >>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
> >>>> correctly.    
> >>>
> >>> Can you point the code you're mentioning? (both what JFFS2 which looks
> >>> strange to you and the old vendor hack)    
> >>
> >> Here is version of the old vendor's driver:
> >>
> >> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
> >>
> >> In my version there is no BUG() there, but it is same driver for the same chip.
> >>
> >> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
> >> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
> >> may be it is unexpected behaviour of JFFS2?  
> > 
> > TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
> > 
> > Are you sure you flash is recognized by JFFS2 as being a NAND device?
> > Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
> > cleanmarker seem to be discarded when using a NAND device, and
> > recognizing the device as a NAND device requires the above option to be
> > set apparently.  
> 
> Yes, I have
> 
> CONFIG_JFFS2_FS_WRITEBUFFER=y
> 
> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
> cleanmarkers are not discarded with NAND device. 

Excellent. So when cleanmarker_size == 0, it means there is no
cleanmarker. But if it is a NAND device, we write the marker anyway.

Well I guess it used to work on old controllers using a Hamming ECC
engine not protecting any user OOB bytes, so writing the clean markers
would simply not lead to ECC bytes being produced/written. Or it might
have worked as well on controller drivers not enabling the ECC engine
when performing OOB-only writes. It also requires the chip to be old
enough to support multiple writes on the same (sub)page as long as the
written bits do not overlap?

Perhaps that's what the hack in the old driver is for. But that's
IMHO broken in case of unexpected reboot :-)

Miquèl
Arseniy Krasnov May 2, 2023, 4:13 p.m. UTC | #47
On 02.05.2023 16:05, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
> 
>> On 02.05.2023 15:17, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> Richard, your input is welcome below :-)
>>>   
>>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>>>>>> 2) Umount JFFS2. Done.
>>>>>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>>>>>> 6) Mount failed.
>>>>>>>>>>>
>>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.        
>>>>>>>>>
>>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>>>>>         
>>>>>>>>
>>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>>>>>> ECC codes.
>>>>>>>>
>>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>>>>>> separately.      
>>>>>>>
>>>>>>> The fact that there might be helpers just for writing OOB areas or just
>>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
>>>>>>> single time, no matter what portion you write. In some cases, it is
>>>>>>> possible to perform subpage writes if the chip supports it. Pages may
>>>>>>> be split into several areas which cover a partial in-band area *and* a
>>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>>>>>> of a given subpage, you *cannot* write the other part later without      
>>>>>>
>>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>>>>>> to write page after writing clean markers to it before? In the old vendor's
>>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>>>>>> correctly.    
>>>>>
>>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
>>>>> strange to you and the old vendor hack)    
>>>>
>>>> Here is version of the old vendor's driver:
>>>>
>>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
>>>>
>>>> In my version there is no BUG() there, but it is same driver for the same chip.
>>>>
>>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
>>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
>>>> may be it is unexpected behaviour of JFFS2?  
>>>
>>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
>>>
>>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
>>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
>>> cleanmarker seem to be discarded when using a NAND device, and
>>> recognizing the device as a NAND device requires the above option to be
>>> set apparently.  
>>
>> Yes, I have
>>
>> CONFIG_JFFS2_FS_WRITEBUFFER=y
>>
>> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
>> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
>> cleanmarkers are not discarded with NAND device. 
> 
> Excellent. So when cleanmarker_size == 0, it means there is no
> cleanmarker. But if it is a NAND device, we write the marker anyway.
> 
> Well I guess it used to work on old controllers using a Hamming ECC
> engine not protecting any user OOB bytes, so writing the clean markers
> would simply not lead to ECC bytes being produced/written. Or it might
> have worked as well on controller drivers not enabling the ECC engine
> when performing OOB-only writes. It also requires the chip to be old
> enough to support multiple writes on the same (sub)page as long as the
> written bits do not overlap?

Yes, with controller which supports such modes there will be no problem here!
What i see, is that this controller doesn't support multiple writes to the
same page in ECC mode(e.g. it can't update ECC correctly). So in v2 i've added
patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.
So for me main question here is:

How JFFS2 should work with controllers where we can't update data and OOB
independently? Driver of this filesystem knows nothing about this features of
the controller.

Or JFFS2 works incorrectly in my case when it tries to call write page callback
after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).

Or may be it is better to suppress OOB write callback (or set it to NULL) in this
driver as in vendor's driver?

Thanks, Arseniy

> 
> Perhaps that's what the hack in the old driver is for. But that's
> IMHO broken in case of unexpected reboot :-)
> 
> Miquèl
Miquel Raynal May 3, 2023, 8:03 a.m. UTC | #48
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 19:13:38 +0300:

> On 02.05.2023 16:05, Miquel Raynal wrote:
> > Hi Arseniy,
> > 
> > avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
> >   
> >> On 02.05.2023 15:17, Miquel Raynal wrote:  
> >>> Hi Arseniy,
> >>>
> >>> Richard, your input is welcome below :-)
> >>>     
> >>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
> >>>>>>>>>>> 0) First attempt to mount JFFS2.
> >>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>>>>>>>>>> 2) Umount JFFS2. Done.
> >>>>>>>>>>> 3) Second attempt to mount JFFS2.
> >>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
> >>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>>>>>>>>>> 6) Mount failed.
> >>>>>>>>>>>
> >>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.          
> >>>>>>>>>
> >>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> >>>>>>>>> does it have to use raw write/read on step 1) and 4)?
> >>>>>>>>>           
> >>>>>>>>
> >>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> >>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> >>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> >>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
> >>>>>>>> ECC codes.
> >>>>>>>>
> >>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
> >>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> >>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
> >>>>>>>> separately.        
> >>>>>>>
> >>>>>>> The fact that there might be helpers just for writing OOB areas or just
> >>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
> >>>>>>> single time, no matter what portion you write. In some cases, it is
> >>>>>>> possible to perform subpage writes if the chip supports it. Pages may
> >>>>>>> be split into several areas which cover a partial in-band area *and* a
> >>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
> >>>>>>> of a given subpage, you *cannot* write the other part later without        
> >>>>>>
> >>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
> >>>>>> to write page after writing clean markers to it before? In the old vendor's
> >>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
> >>>>>> correctly.      
> >>>>>
> >>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
> >>>>> strange to you and the old vendor hack)      
> >>>>
> >>>> Here is version of the old vendor's driver:
> >>>>
> >>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
> >>>>
> >>>> In my version there is no BUG() there, but it is same driver for the same chip.
> >>>>
> >>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
> >>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
> >>>> may be it is unexpected behaviour of JFFS2?    
> >>>
> >>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
> >>>
> >>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
> >>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
> >>> cleanmarker seem to be discarded when using a NAND device, and
> >>> recognizing the device as a NAND device requires the above option to be
> >>> set apparently.    
> >>
> >> Yes, I have
> >>
> >> CONFIG_JFFS2_FS_WRITEBUFFER=y
> >>
> >> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
> >> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
> >> cleanmarkers are not discarded with NAND device.   
> > 
> > Excellent. So when cleanmarker_size == 0, it means there is no
> > cleanmarker. But if it is a NAND device, we write the marker anyway.
> > 
> > Well I guess it used to work on old controllers using a Hamming ECC
> > engine not protecting any user OOB bytes, so writing the clean markers
> > would simply not lead to ECC bytes being produced/written. Or it might
> > have worked as well on controller drivers not enabling the ECC engine
> > when performing OOB-only writes. It also requires the chip to be old
> > enough to support multiple writes on the same (sub)page as long as the
> > written bits do not overlap?  
> 
> Yes, with controller which supports such modes there will be no problem here!
> What i see, is that this controller doesn't support multiple writes to the
> same page in ECC mode(e.g. it can't update ECC correctly).

I don't think this is a controller limitation. The NAND chip cannot
write ECC bytes a first time and then overwrite other ECC bytes, that
cannot work. The fact that we write ECC bytes in the first place is
because the ECC engine covers the free OOB bytes used by JFFS2 to write
its cleanmarkers.

> So in v2 i've added
> patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.

I am sorry but the above sentence is not clear to me. I believe you
meant the free OOB bytes are moved outside of the area protected by the
ECC engine. In this case I guess it should be fine.

> So for me main question here is:
> 
> How JFFS2 should work with controllers where we can't update data and OOB
> independently? Driver of this filesystem knows nothing about this features of
> the controller.
> 
> Or JFFS2 works incorrectly in my case when it tries to call write page callback
> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
> 
> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
> driver as in vendor's driver?

I would assume using the unprotected free OOB bytes to store the
cleanmarkers should work. But that's a bit fragile and very filesystem
oriented. I don't like this much. But on the other side JFFS2 is
legacy, you should use UBI (which does not play with OOB areas) :-)

Thanks,
Miquèl

> 
> Thanks, Arseniy
> 
> > 
> > Perhaps that's what the hack in the old driver is for. But that's
> > IMHO broken in case of unexpected reboot :-)
> > 
> > Miquèl
Arseniy Krasnov May 3, 2023, 10:23 a.m. UTC | #49
On 03.05.2023 11:03, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 19:13:38 +0300:
> 
>> On 02.05.2023 16:05, Miquel Raynal wrote:
>>> Hi Arseniy,
>>>
>>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
>>>   
>>>> On 02.05.2023 15:17, Miquel Raynal wrote:  
>>>>> Hi Arseniy,
>>>>>
>>>>> Richard, your input is welcome below :-)
>>>>>     
>>>>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>>>>>>>> 2) Umount JFFS2. Done.
>>>>>>>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>>>>>>>> 6) Mount failed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.          
>>>>>>>>>>>
>>>>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>>>>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>>>>>>>           
>>>>>>>>>>
>>>>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>>>>>>>> ECC codes.
>>>>>>>>>>
>>>>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>>>>>>>> separately.        
>>>>>>>>>
>>>>>>>>> The fact that there might be helpers just for writing OOB areas or just
>>>>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
>>>>>>>>> single time, no matter what portion you write. In some cases, it is
>>>>>>>>> possible to perform subpage writes if the chip supports it. Pages may
>>>>>>>>> be split into several areas which cover a partial in-band area *and* a
>>>>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>>>>>>>> of a given subpage, you *cannot* write the other part later without        
>>>>>>>>
>>>>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>>>>>>>> to write page after writing clean markers to it before? In the old vendor's
>>>>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>>>>>>>> correctly.      
>>>>>>>
>>>>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
>>>>>>> strange to you and the old vendor hack)      
>>>>>>
>>>>>> Here is version of the old vendor's driver:
>>>>>>
>>>>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
>>>>>>
>>>>>> In my version there is no BUG() there, but it is same driver for the same chip.
>>>>>>
>>>>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
>>>>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
>>>>>> may be it is unexpected behaviour of JFFS2?    
>>>>>
>>>>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
>>>>>
>>>>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
>>>>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
>>>>> cleanmarker seem to be discarded when using a NAND device, and
>>>>> recognizing the device as a NAND device requires the above option to be
>>>>> set apparently.    
>>>>
>>>> Yes, I have
>>>>
>>>> CONFIG_JFFS2_FS_WRITEBUFFER=y
>>>>
>>>> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
>>>> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
>>>> cleanmarkers are not discarded with NAND device.   
>>>
>>> Excellent. So when cleanmarker_size == 0, it means there is no
>>> cleanmarker. But if it is a NAND device, we write the marker anyway.
>>>
>>> Well I guess it used to work on old controllers using a Hamming ECC
>>> engine not protecting any user OOB bytes, so writing the clean markers
>>> would simply not lead to ECC bytes being produced/written. Or it might
>>> have worked as well on controller drivers not enabling the ECC engine
>>> when performing OOB-only writes. It also requires the chip to be old
>>> enough to support multiple writes on the same (sub)page as long as the
>>> written bits do not overlap?  
>>
>> Yes, with controller which supports such modes there will be no problem here!
>> What i see, is that this controller doesn't support multiple writes to the
>> same page in ECC mode(e.g. it can't update ECC correctly).
> 
> I don't think this is a controller limitation. The NAND chip cannot
> write ECC bytes a first time and then overwrite other ECC bytes, that
> cannot work. The fact that we write ECC bytes in the first place is
> because the ECC engine covers the free OOB bytes used by JFFS2 to write
> its cleanmarkers.
> 
>> So in v2 i've added
>> patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.
> 
> I am sorry but the above sentence is not clear to me. I believe you
> meant the free OOB bytes are moved outside of the area protected by the
> ECC engine. In this case I guess it should be fine.

Exactly, free bytes which are reported by OOB layout callbacks were moved out of
ECC area.

> 
>> So for me main question here is:
>>
>> How JFFS2 should work with controllers where we can't update data and OOB
>> independently? Driver of this filesystem knows nothing about this features of
>> the controller.
>>
>> Or JFFS2 works incorrectly in my case when it tries to call write page callback
>> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
>>
>> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
>> driver as in vendor's driver?
> 
> I would assume using the unprotected free OOB bytes to store the
> cleanmarkers should work. But that's a bit fragile and very filesystem
> oriented. I don't like this much. But on the other side JFFS2 is
> legacy, you should use UBI (which does not play with OOB areas) :-)

Problem here is that we can't use UBI in this case, because it does not support
small fs images. So the only way to make JFFS2 work is to move free OOB bytes to
non protected area. Otherwise i think we have strange situation that JFFS2 can't
work correctly on specific type on NAND controller. We already had same problem
on another NAND controller, and solution was to move OOB free bytes no non-protected
area:

https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru/

Thanks, Arseniy

> 
> Thanks,
> Miquèl
> 
>>
>> Thanks, Arseniy
>>
>>>
>>> Perhaps that's what the hack in the old driver is for. But that's
>>> IMHO broken in case of unexpected reboot :-)
>>>
>>> Miquèl  
>
Richard Weinberger May 3, 2023, 7:48 p.m. UTC | #50
Sorry for joining the party late.

On Wed, May 3, 2023 at 10:07 AM Miquel Raynal <miquel.raynal@bootlin.com> wrote:
> > So for me main question here is:
> >
> > How JFFS2 should work with controllers where we can't update data and OOB
> > independently? Driver of this filesystem knows nothing about this features of
> > the controller.
> >
> > Or JFFS2 works incorrectly in my case when it tries to call write page callback
> > after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
> >
> > Or may be it is better to suppress OOB write callback (or set it to NULL) in this
> > driver as in vendor's driver?
>
> I would assume using the unprotected free OOB bytes to store the
> cleanmarkers should work. But that's a bit fragile and very filesystem
> oriented. I don't like this much. But on the other side JFFS2 is
> legacy, you should use UBI (which does not play with OOB areas) :-)

Please note that JFFS2's main use case is NOR flash. Support for NAND
flash was never
considered as fully complete/stable. That's why UBIFS (JFFS3 in the
very beginning)
was invented.

Thanks,
//richard
Arseniy Krasnov May 4, 2023, 11:37 a.m. UTC | #51
On 03.05.2023 13:23, Arseniy Krasnov wrote:
> 
> 
> On 03.05.2023 11:03, Miquel Raynal wrote:
>> Hi Arseniy,
>>
>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 19:13:38 +0300:
>>
>>> On 02.05.2023 16:05, Miquel Raynal wrote:
>>>> Hi Arseniy,
>>>>
>>>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
>>>>   
>>>>> On 02.05.2023 15:17, Miquel Raynal wrote:  
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> Richard, your input is welcome below :-)
>>>>>>     
>>>>>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>>>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>>>>>>>>> 2) Umount JFFS2. Done.
>>>>>>>>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>>>>>>>>> 6) Mount failed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.          
>>>>>>>>>>>>
>>>>>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>>>>>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>>>>>>>>           
>>>>>>>>>>>
>>>>>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>>>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>>>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>>>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>>>>>>>>> ECC codes.
>>>>>>>>>>>
>>>>>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>>>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>>>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>>>>>>>>> separately.        
>>>>>>>>>>
>>>>>>>>>> The fact that there might be helpers just for writing OOB areas or just
>>>>>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
>>>>>>>>>> single time, no matter what portion you write. In some cases, it is
>>>>>>>>>> possible to perform subpage writes if the chip supports it. Pages may
>>>>>>>>>> be split into several areas which cover a partial in-band area *and* a
>>>>>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>>>>>>>>> of a given subpage, you *cannot* write the other part later without        
>>>>>>>>>
>>>>>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>>>>>>>>> to write page after writing clean markers to it before? In the old vendor's
>>>>>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>>>>>>>>> correctly.      
>>>>>>>>
>>>>>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
>>>>>>>> strange to you and the old vendor hack)      
>>>>>>>
>>>>>>> Here is version of the old vendor's driver:
>>>>>>>
>>>>>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
>>>>>>>
>>>>>>> In my version there is no BUG() there, but it is same driver for the same chip.
>>>>>>>
>>>>>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
>>>>>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
>>>>>>> may be it is unexpected behaviour of JFFS2?    
>>>>>>
>>>>>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
>>>>>>
>>>>>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
>>>>>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
>>>>>> cleanmarker seem to be discarded when using a NAND device, and
>>>>>> recognizing the device as a NAND device requires the above option to be
>>>>>> set apparently.    
>>>>>
>>>>> Yes, I have
>>>>>
>>>>> CONFIG_JFFS2_FS_WRITEBUFFER=y
>>>>>
>>>>> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
>>>>> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
>>>>> cleanmarkers are not discarded with NAND device.   
>>>>
>>>> Excellent. So when cleanmarker_size == 0, it means there is no
>>>> cleanmarker. But if it is a NAND device, we write the marker anyway.
>>>>
>>>> Well I guess it used to work on old controllers using a Hamming ECC
>>>> engine not protecting any user OOB bytes, so writing the clean markers
>>>> would simply not lead to ECC bytes being produced/written. Or it might
>>>> have worked as well on controller drivers not enabling the ECC engine
>>>> when performing OOB-only writes. It also requires the chip to be old
>>>> enough to support multiple writes on the same (sub)page as long as the
>>>> written bits do not overlap?  
>>>
>>> Yes, with controller which supports such modes there will be no problem here!
>>> What i see, is that this controller doesn't support multiple writes to the
>>> same page in ECC mode(e.g. it can't update ECC correctly).
>>
>> I don't think this is a controller limitation. The NAND chip cannot
>> write ECC bytes a first time and then overwrite other ECC bytes, that
>> cannot work. The fact that we write ECC bytes in the first place is
>> because the ECC engine covers the free OOB bytes used by JFFS2 to write
>> its cleanmarkers.
>>
>>> So in v2 i've added
>>> patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.
>>
>> I am sorry but the above sentence is not clear to me. I believe you
>> meant the free OOB bytes are moved outside of the area protected by the
>> ECC engine. In this case I guess it should be fine.
> 
> Exactly, free bytes which are reported by OOB layout callbacks were moved out of
> ECC area.
> 
>>
>>> So for me main question here is:
>>>
>>> How JFFS2 should work with controllers where we can't update data and OOB
>>> independently? Driver of this filesystem knows nothing about this features of
>>> the controller.
>>>
>>> Or JFFS2 works incorrectly in my case when it tries to call write page callback
>>> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
>>>
>>> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
>>> driver as in vendor's driver?
>>
>> I would assume using the unprotected free OOB bytes to store the
>> cleanmarkers should work. But that's a bit fragile and very filesystem
>> oriented. I don't like this much. But on the other side JFFS2 is
>> legacy, you should use UBI (which does not play with OOB areas) :-)
> 
> Problem here is that we can't use UBI in this case, because it does not support
> small fs images. So the only way to make JFFS2 work is to move free OOB bytes to
> non protected area. Otherwise i think we have strange situation that JFFS2 can't
> work correctly on specific type on NAND controller. We already had same problem
> on another NAND controller, and solution was to move OOB free bytes no non-protected
> area:
> 
> https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru/
> 
> Thanks, Arseniy

Upd: may be i can add option for this driver, which makes JFFS2 work correctly on this chip.
This feature suppresses OOB writes as in old driver. By default it is disabled and OOB is
ECC protected(current behaviour), if enabled - it prints WARN_ONCE() and always returns 0.
What do You think?

Or may be add an option, which moves free bytes of OOB to ECC non-protected area and it is disabled
by default.

Thanks, Arseniy

> 
>>
>> Thanks,
>> Miquèl
>>
>>>
>>> Thanks, Arseniy
>>>
>>>>
>>>> Perhaps that's what the hack in the old driver is for. But that's
>>>> IMHO broken in case of unexpected reboot :-)
>>>>
>>>> Miquèl  
>>
Arseniy Krasnov May 4, 2023, 11:40 a.m. UTC | #52
On 03.05.2023 22:48, Richard Weinberger wrote:
> Sorry for joining the party late.
> 
> On Wed, May 3, 2023 at 10:07 AM Miquel Raynal <miquel.raynal@bootlin.com> wrote:
>>> So for me main question here is:
>>>
>>> How JFFS2 should work with controllers where we can't update data and OOB
>>> independently? Driver of this filesystem knows nothing about this features of
>>> the controller.
>>>
>>> Or JFFS2 works incorrectly in my case when it tries to call write page callback
>>> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
>>>
>>> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
>>> driver as in vendor's driver?
>>
>> I would assume using the unprotected free OOB bytes to store the
>> cleanmarkers should work. But that's a bit fragile and very filesystem
>> oriented. I don't like this much. But on the other side JFFS2 is
>> legacy, you should use UBI (which does not play with OOB areas) :-)
> 
> Please note that JFFS2's main use case is NOR flash. Support for NAND
> flash was never
> considered as fully complete/stable. That's why UBIFS (JFFS3 in the
> very beginning)
> was invented.

Hello Richard!

Thanks for this details! But I think it will be useful to make JFFS2 work correctly
on this chip :) Anyway i think it is not rare/exotic case to use JFFS2 on NAND.

Thanks, Arseniy

> 
> Thanks,
> //richard
Miquel Raynal May 4, 2023, 12:17 p.m. UTC | #53
Hi Arseniy,

avkrasnov@sberdevices.ru wrote on Thu, 4 May 2023 14:37:45 +0300:

> On 03.05.2023 13:23, Arseniy Krasnov wrote:
> > 
> > 
> > On 03.05.2023 11:03, Miquel Raynal wrote:  
> >> Hi Arseniy,
> >>
> >> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 19:13:38 +0300:
> >>  
> >>> On 02.05.2023 16:05, Miquel Raynal wrote:  
> >>>> Hi Arseniy,
> >>>>
> >>>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
> >>>>     
> >>>>> On 02.05.2023 15:17, Miquel Raynal wrote:    
> >>>>>> Hi Arseniy,
> >>>>>>
> >>>>>> Richard, your input is welcome below :-)
> >>>>>>       
> >>>>>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
> >>>>>>>>>>>>>> 0) First attempt to mount JFFS2.
> >>>>>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
> >>>>>>>>>>>>>> 2) Umount JFFS2. Done.
> >>>>>>>>>>>>>> 3) Second attempt to mount JFFS2.
> >>>>>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
> >>>>>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
> >>>>>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
> >>>>>>>>>>>>>> 6) Mount failed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
> >>>>>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.            
> >>>>>>>>>>>>
> >>>>>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
> >>>>>>>>>>>> does it have to use raw write/read on step 1) and 4)?
> >>>>>>>>>>>>             
> >>>>>>>>>>>
> >>>>>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
> >>>>>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
> >>>>>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
> >>>>>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
> >>>>>>>>>>> ECC codes.
> >>>>>>>>>>>
> >>>>>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
> >>>>>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
> >>>>>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
> >>>>>>>>>>> separately.          
> >>>>>>>>>>
> >>>>>>>>>> The fact that there might be helpers just for writing OOB areas or just
> >>>>>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
> >>>>>>>>>> single time, no matter what portion you write. In some cases, it is
> >>>>>>>>>> possible to perform subpage writes if the chip supports it. Pages may
> >>>>>>>>>> be split into several areas which cover a partial in-band area *and* a
> >>>>>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
> >>>>>>>>>> of a given subpage, you *cannot* write the other part later without          
> >>>>>>>>>
> >>>>>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
> >>>>>>>>> to write page after writing clean markers to it before? In the old vendor's
> >>>>>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
> >>>>>>>>> correctly.        
> >>>>>>>>
> >>>>>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
> >>>>>>>> strange to you and the old vendor hack)        
> >>>>>>>
> >>>>>>> Here is version of the old vendor's driver:
> >>>>>>>
> >>>>>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
> >>>>>>>
> >>>>>>> In my version there is no BUG() there, but it is same driver for the same chip.
> >>>>>>>
> >>>>>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
> >>>>>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
> >>>>>>> may be it is unexpected behaviour of JFFS2?      
> >>>>>>
> >>>>>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
> >>>>>>
> >>>>>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
> >>>>>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
> >>>>>> cleanmarker seem to be discarded when using a NAND device, and
> >>>>>> recognizing the device as a NAND device requires the above option to be
> >>>>>> set apparently.      
> >>>>>
> >>>>> Yes, I have
> >>>>>
> >>>>> CONFIG_JFFS2_FS_WRITEBUFFER=y
> >>>>>
> >>>>> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
> >>>>> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
> >>>>> cleanmarkers are not discarded with NAND device.     
> >>>>
> >>>> Excellent. So when cleanmarker_size == 0, it means there is no
> >>>> cleanmarker. But if it is a NAND device, we write the marker anyway.
> >>>>
> >>>> Well I guess it used to work on old controllers using a Hamming ECC
> >>>> engine not protecting any user OOB bytes, so writing the clean markers
> >>>> would simply not lead to ECC bytes being produced/written. Or it might
> >>>> have worked as well on controller drivers not enabling the ECC engine
> >>>> when performing OOB-only writes. It also requires the chip to be old
> >>>> enough to support multiple writes on the same (sub)page as long as the
> >>>> written bits do not overlap?    
> >>>
> >>> Yes, with controller which supports such modes there will be no problem here!
> >>> What i see, is that this controller doesn't support multiple writes to the
> >>> same page in ECC mode(e.g. it can't update ECC correctly).  
> >>
> >> I don't think this is a controller limitation. The NAND chip cannot
> >> write ECC bytes a first time and then overwrite other ECC bytes, that
> >> cannot work. The fact that we write ECC bytes in the first place is
> >> because the ECC engine covers the free OOB bytes used by JFFS2 to write
> >> its cleanmarkers.
> >>  
> >>> So in v2 i've added
> >>> patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.  
> >>
> >> I am sorry but the above sentence is not clear to me. I believe you
> >> meant the free OOB bytes are moved outside of the area protected by the
> >> ECC engine. In this case I guess it should be fine.  
> > 
> > Exactly, free bytes which are reported by OOB layout callbacks were moved out of
> > ECC area.
> >   
> >>  
> >>> So for me main question here is:
> >>>
> >>> How JFFS2 should work with controllers where we can't update data and OOB
> >>> independently? Driver of this filesystem knows nothing about this features of
> >>> the controller.
> >>>
> >>> Or JFFS2 works incorrectly in my case when it tries to call write page callback
> >>> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
> >>>
> >>> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
> >>> driver as in vendor's driver?  
> >>
> >> I would assume using the unprotected free OOB bytes to store the
> >> cleanmarkers should work. But that's a bit fragile and very filesystem
> >> oriented. I don't like this much. But on the other side JFFS2 is
> >> legacy, you should use UBI (which does not play with OOB areas) :-)  
> > 
> > Problem here is that we can't use UBI in this case, because it does not support
> > small fs images. So the only way to make JFFS2 work is to move free OOB bytes to
> > non protected area. Otherwise i think we have strange situation that JFFS2 can't
> > work correctly on specific type on NAND controller. We already had same problem
> > on another NAND controller, and solution was to move OOB free bytes no non-protected
> > area:
> > 
> > https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru/
> > 
> > Thanks, Arseniy  
> 
> Upd: may be i can add option for this driver, which makes JFFS2 work correctly on this chip.
> This feature suppresses OOB writes as in old driver. By default it is disabled and OOB is
> ECC protected(current behaviour), if enabled - it prints WARN_ONCE() and always returns 0.
> What do You think?
> 
> Or may be add an option, which moves free bytes of OOB to ECC non-protected area and it is disabled
> by default.

I prefer having a single ooblayout where we expose unprotected user OOB
bytes only. As of today, the only upstream user of user OOB bytes is
JFFS2 anyway.

Thanks,
Miquèl
Arseniy Krasnov May 4, 2023, 12:31 p.m. UTC | #54
On 04.05.2023 15:17, Miquel Raynal wrote:
> Hi Arseniy,
> 
> avkrasnov@sberdevices.ru wrote on Thu, 4 May 2023 14:37:45 +0300:
> 
>> On 03.05.2023 13:23, Arseniy Krasnov wrote:
>>>
>>>
>>> On 03.05.2023 11:03, Miquel Raynal wrote:  
>>>> Hi Arseniy,
>>>>
>>>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 19:13:38 +0300:
>>>>  
>>>>> On 02.05.2023 16:05, Miquel Raynal wrote:  
>>>>>> Hi Arseniy,
>>>>>>
>>>>>> avkrasnov@sberdevices.ru wrote on Tue, 2 May 2023 15:24:09 +0300:
>>>>>>     
>>>>>>> On 02.05.2023 15:17, Miquel Raynal wrote:    
>>>>>>>> Hi Arseniy,
>>>>>>>>
>>>>>>>> Richard, your input is welcome below :-)
>>>>>>>>       
>>>>>>>>>>>>>>>> I just checked JFFS2 mount/umount again, here is what i see:
>>>>>>>>>>>>>>>> 0) First attempt to mount JFFS2.
>>>>>>>>>>>>>>>> 1) It writes OOB to page N (i'm using raw write). It is cleanmarker value 0x85 0x19 0x03 0x20. Mount is done.
>>>>>>>>>>>>>>>> 2) Umount JFFS2. Done.
>>>>>>>>>>>>>>>> 3) Second attempt to mount JFFS2.
>>>>>>>>>>>>>>>> 4) It reads OOB from page N (i'm using raw read). Value is 0x85 0x19 0x03 0x20. Done.
>>>>>>>>>>>>>>>> 5) It reads page N in ECC mode, and i get:
>>>>>>>>>>>>>>>>      jffs2: mtd->read(0x100 bytes from N) returned ECC error
>>>>>>>>>>>>>>>> 6) Mount failed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We already had problem which looks like this on another device. Solution was to use OOB area which is
>>>>>>>>>>>>>>>> not covered by ECC for JFFS2 cleanmarkers.            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ok, so there is not ECC parity bytes and mtd->read() returns ECC error.
>>>>>>>>>>>>>> does it have to use raw write/read on step 1) and 4)?
>>>>>>>>>>>>>>             
>>>>>>>>>>>>>
>>>>>>>>>>>>> If i'm using non raw access to OOB, for example write OOB (user bytes) in ECC mode, then
>>>>>>>>>>>>> steps 1) and 4) and 5) passes ok, but write to this page will be impossible (for example JFFS2
>>>>>>>>>>>>> writes to such pages later) - we can't update ECC codes properly without erasing whole page.
>>>>>>>>>>>>> Write operation will be done without problem, but read will trigger ECC errors due to broken
>>>>>>>>>>>>> ECC codes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In general problem that we discuss is that in current implementation data and OOB conflicts
>>>>>>>>>>>>> with each other by sharing same ECC codes, these ECC codes could be written only once (without
>>>>>>>>>>>>> erasing), while data and OOB has different callbacks to access and thus supposed to work
>>>>>>>>>>>>> separately.          
>>>>>>>>>>>>
>>>>>>>>>>>> The fact that there might be helpers just for writing OOB areas or just
>>>>>>>>>>>> in-band areas are optimizations. NAND pages are meant to be written a
>>>>>>>>>>>> single time, no matter what portion you write. In some cases, it is
>>>>>>>>>>>> possible to perform subpage writes if the chip supports it. Pages may
>>>>>>>>>>>> be split into several areas which cover a partial in-band area *and* a
>>>>>>>>>>>> partial OOB area. If you write into the in-band *or* out-of-band areas
>>>>>>>>>>>> of a given subpage, you *cannot* write the other part later without          
>>>>>>>>>>>
>>>>>>>>>>> Thanks for details! So in case of JFFS2 it looks like strange, that it tries
>>>>>>>>>>> to write page after writing clean markers to it before? In the old vendor's
>>>>>>>>>>> driver OOB write callback is suppressed by return 0 always and JFFS2 works
>>>>>>>>>>> correctly.        
>>>>>>>>>>
>>>>>>>>>> Can you point the code you're mentioning? (both what JFFS2 which looks
>>>>>>>>>> strange to you and the old vendor hack)        
>>>>>>>>>
>>>>>>>>> Here is version of the old vendor's driver:
>>>>>>>>>
>>>>>>>>> https://github.com/kszaq/linux-amlogic/blob/master_new_amports/drivers/amlogic/nand/nand/aml_nand.c#L3260
>>>>>>>>>
>>>>>>>>> In my version there is no BUG() there, but it is same driver for the same chip.
>>>>>>>>>
>>>>>>>>> About JFFS2 - i didn't check its source code, but what I can see using printk(), is that it first
>>>>>>>>> tries to write cleanmarker using OOB write callback. Then later it tries to write to this page, so
>>>>>>>>> may be it is unexpected behaviour of JFFS2?      
>>>>>>>>
>>>>>>>> TBH I am not knowledgeable about JFFS2, maybe Richard can help here.
>>>>>>>>
>>>>>>>> Are you sure you flash is recognized by JFFS2 as being a NAND device?
>>>>>>>> Did you enable CONFIG_JFFS2_FS_WRITEBUFFER correctly? Because
>>>>>>>> cleanmarker seem to be discarded when using a NAND device, and
>>>>>>>> recognizing the device as a NAND device requires the above option to be
>>>>>>>> set apparently.      
>>>>>>>
>>>>>>> Yes, I have
>>>>>>>
>>>>>>> CONFIG_JFFS2_FS_WRITEBUFFER=y
>>>>>>>
>>>>>>> And i see, that jffs2_mark_erased_block() calls jffs2_cleanmarker_oob() which checks that we have MTD_NANDFLASH. This
>>>>>>> check is true, so then jffs2_write_nand_cleanmarker() is called and there is OOB write in it. So I see opposite thing:
>>>>>>> cleanmarkers are not discarded with NAND device.     
>>>>>>
>>>>>> Excellent. So when cleanmarker_size == 0, it means there is no
>>>>>> cleanmarker. But if it is a NAND device, we write the marker anyway.
>>>>>>
>>>>>> Well I guess it used to work on old controllers using a Hamming ECC
>>>>>> engine not protecting any user OOB bytes, so writing the clean markers
>>>>>> would simply not lead to ECC bytes being produced/written. Or it might
>>>>>> have worked as well on controller drivers not enabling the ECC engine
>>>>>> when performing OOB-only writes. It also requires the chip to be old
>>>>>> enough to support multiple writes on the same (sub)page as long as the
>>>>>> written bits do not overlap?    
>>>>>
>>>>> Yes, with controller which supports such modes there will be no problem here!
>>>>> What i see, is that this controller doesn't support multiple writes to the
>>>>> same page in ECC mode(e.g. it can't update ECC correctly).  
>>>>
>>>> I don't think this is a controller limitation. The NAND chip cannot
>>>> write ECC bytes a first time and then overwrite other ECC bytes, that
>>>> cannot work. The fact that we write ECC bytes in the first place is
>>>> because the ECC engine covers the free OOB bytes used by JFFS2 to write
>>>> its cleanmarkers.
>>>>  
>>>>> So in v2 i've added
>>>>> patch which moves OOB out of ECC area, thus JFFS2 driver will work correctly.  
>>>>
>>>> I am sorry but the above sentence is not clear to me. I believe you
>>>> meant the free OOB bytes are moved outside of the area protected by the
>>>> ECC engine. In this case I guess it should be fine.  
>>>
>>> Exactly, free bytes which are reported by OOB layout callbacks were moved out of
>>> ECC area.
>>>   
>>>>  
>>>>> So for me main question here is:
>>>>>
>>>>> How JFFS2 should work with controllers where we can't update data and OOB
>>>>> independently? Driver of this filesystem knows nothing about this features of
>>>>> the controller.
>>>>>
>>>>> Or JFFS2 works incorrectly in my case when it tries to call write page callback
>>>>> after calling write OOB callback (IIUC it is better to ask Richard as You mentioned above).
>>>>>
>>>>> Or may be it is better to suppress OOB write callback (or set it to NULL) in this
>>>>> driver as in vendor's driver?  
>>>>
>>>> I would assume using the unprotected free OOB bytes to store the
>>>> cleanmarkers should work. But that's a bit fragile and very filesystem
>>>> oriented. I don't like this much. But on the other side JFFS2 is
>>>> legacy, you should use UBI (which does not play with OOB areas) :-)  
>>>
>>> Problem here is that we can't use UBI in this case, because it does not support
>>> small fs images. So the only way to make JFFS2 work is to move free OOB bytes to
>>> non protected area. Otherwise i think we have strange situation that JFFS2 can't
>>> work correctly on specific type on NAND controller. We already had same problem
>>> on another NAND controller, and solution was to move OOB free bytes no non-protected
>>> area:
>>>
>>> https://lore.kernel.org/all/20230329114240.378722-1-mmkurbanov@sberdevices.ru/
>>>
>>> Thanks, Arseniy  
>>
>> Upd: may be i can add option for this driver, which makes JFFS2 work correctly on this chip.
>> This feature suppresses OOB writes as in old driver. By default it is disabled and OOB is
>> ECC protected(current behaviour), if enabled - it prints WARN_ONCE() and always returns 0.
>> What do You think?
>>
>> Or may be add an option, which moves free bytes of OOB to ECC non-protected area and it is disabled
>> by default.
> 
> I prefer having a single ooblayout where we expose unprotected user OOB
> bytes only. As of today, the only upstream user of user OOB bytes is
> JFFS2 anyway.

Ok, so i'll prepare v3 on the next week with this update (it is already in v2, but i'm going
to modify another patch with updates from @Liang).

Thanks, Arseniy

> 
> Thanks,
> Miquèl
diff mbox series

Patch

diff --git a/drivers/mtd/nand/raw/meson_nand.c b/drivers/mtd/nand/raw/meson_nand.c
index f84a10238e4d..f2f2472cb511 100644
--- a/drivers/mtd/nand/raw/meson_nand.c
+++ b/drivers/mtd/nand/raw/meson_nand.c
@@ -858,9 +858,12 @@  static int meson_nfc_read_page_sub(struct nand_chip *nand,
 static int meson_nfc_read_page_raw(struct nand_chip *nand, u8 *buf,
 				   int oob_required, int page)
 {
+	struct mtd_info *mtd = nand_to_mtd(nand);
 	u8 *oob_buf = nand->oob_poi;
 	int ret;
 
+	memset(oob_buf, 0, mtd->oobsize);
+
 	ret = meson_nfc_read_page_sub(nand, page, 1);
 	if (ret)
 		return ret;
@@ -881,6 +884,8 @@  static int meson_nfc_read_page_hwecc(struct nand_chip *nand, u8 *buf,
 	u8 *oob_buf = nand->oob_poi;
 	int ret, i;
 
+	memset(oob_buf, 0, mtd->oobsize);
+
 	ret = meson_nfc_read_page_sub(nand, page, 0);
 	if (ret)
 		return ret;