files read corruption issue

68030 + SDRAM + IDE

Moderators: terriblefire, Terriblefire Moderator

Post Reply
richx
Posts: 46
Joined: Sun Aug 11, 2019 5:03 pm

files read corruption issue

Post by richx »

.

** EDIT / SOLUTION: The test image had MaxTransfer set to the default 0xffffff, changing it to 0x1fe00 for the image appears to solve the problem with my image, looks like that is a good value when there are problems.
Enjoy the details and test image for how to trigger some of these issues.

Been looking at a puzzling issue on my default test image (FFS partition) that took a little while to narrow down on my TF536. When a large enough file such as an WHDLoad ADF file, is copied using WB (whole file read at once) to RAM: it gets corrupted, but a shell copy (file is read in chunks) to RAM: does not corrupt the file. This affects WHDLoad as well, as WHDLoad appears to read at once or reads in bigger chunks.

To add complexity, I can only replicate this issue with large files that are extracted and written by LHA ("lha x"). If the extracted file is then copied somewhere else (using shell copy), it no longer shows the issue.

So two things are needed to trigger the issue, the file needs to have been extracted/written by LHA, and AmigaOS needs to read it in bigger chunks. If file is read in smaller chunks, there is no corruption. File is also extracted/written/copied correctly.

Yeah, I know this sounds crazy. I analyzed HDF images with file written by LHA vs when file was simply shell copied, and the LHA written file appears to have larger but fewer chunks of filesystem data interleaved with file data. The LHA written chunks of filesystem data are 0x600-0x800 bytes vs just 0x200-0x400 bytes, but twice as many filesystem chunks exist with the copied file. So theory is there is a difference how the file is written into the filesystem enough to trigger the issue with larger reads in certain scenarios.

How the file read is corrupted is always consistent for given read size, regardless of file content or file size, file just needs to be large enough (and larger different read chunk size has different corruption). Basically, for an 1MB file that is read all at once, the following sections are always corrupted and read from other areas of the same file:

0x44000-0x47fff : data is from 0x24200-0x281ff, offset -0x1fe00, section size 0x4000
0x83000-0x86fff : data is from 0x63200-0x671ff, offset -0x1fe00, section size 0x4000, sections 0x3f000 apart
0xc2000-0xc5fff : data is from 0xa2200-0xa61ff, offset -0x1fe00, section size 0x4000, sections 0x3f000 apart

This issue does not occur in emulators, or with other adapters. The TF IDE test ADF and programs did not report any issues with my (unbuffered) TF536. CPU Fastrom does not matter. Tried with different CPUs, different machines, different cables, different IDE adapters, different CompactFlash and SD cards, same issue across all. Could not replicate issue on an image with PFS3 partition.

To make things easy to reproduce, I have created the attached 10MB HDF image. It contains 4 large 1MB test files in files dir, two "bad" files that were extracted/written by LHA and show issue, two "good" files were copied and don't show issue. I wrote a program (source included) to read files in either smaller chunks or all at once to mimic the WB vs shell copy, and to calculate the MD5SUM of the read file and to find the duplicated sections.

I would be grateful if people could try this image on their TF536s and report back findings. Simply boot up with the image, run the "cf" script with or without read size specified:

# reads files all at once (large chunk), should show consistent issue on "bad" files and no issues on "good" files
1> cf

# reads files in smaller chunks, all files should show no issues
1> cf 100000

MD5SUM for test files is b83d6e88ce83da7a4ea06e103cb2c70e.

check_files_output.jpg
check_files_output.jpg (86.88 KiB) Viewed 3035 times
Attachments
10MB_forum.hdf.7z
(87.44 KiB) Downloaded 98 times
terriblefire
Moderator Team
Moderator Team
Posts: 5368
Joined: Mon Aug 28, 2017 10:56 pm
Location: Glasgow, UK

Re: files read corruption issue

Post by terriblefire »

lets get that max transfer checked. Thats usually the culprit.
———
"It is not necessarily a supply voltage at no load, but the amount of current it can provide when touched that
indicates how much hurting you shall receive."
richx
Posts: 46
Joined: Sun Aug 11, 2019 5:03 pm

Re: files read corruption issue

Post by richx »

The MaxTransfer was set to 0xffffff, changing it to 0x1fe00 solves the read corruption with the 10MB test image! Thanks! Did not know this, didn't have any issues with the default 0xffffff before.
User avatar
GadgetUK164
Posts: 430
Joined: Fri Jan 04, 2019 2:26 pm

Re: files read corruption issue

Post by GadgetUK164 »

EDIT: I see your 536 is unbuffered - might be worth adding those buffers!!! How long is your IDE cable, and what CF card / drive are you using?
My YouTube Channel - www.youtube.com/GadgetUK164
richx
Posts: 46
Joined: Sun Aug 11, 2019 5:03 pm

Re: files read corruption issue

Post by richx »

Yeah might add the buffers in the future. The IDE cable is one of those 2 inch ones, tried the eBay 8GB blue/white card, Sandisk 8GB etc. Looks like the 0x1fe00 MaxTransfer solves it, still curious if others get the same issues with the default 0xffffff.
go0se
Posts: 403
Joined: Sun Nov 25, 2018 7:55 pm

Re: files read corruption issue

Post by go0se »

richx wrote: Sat Mar 13, 2021 7:12 pm The MaxTransfer was set to 0xffffff, changing it to 0x1fe00 solves the read corruption with the 10MB test image! Thanks! Did not know this, didn't have any issues with the default 0xffffff before.
Damn! You mean I've just set up an A500 and wrote a 10MB SD card image for nothing!?! ;)

I can confirm that I get the same output as on your screenshot when testing with your 10MB image on a (buffered) TF536 here.

If altering max transfer fixes it then result!
terriblefire
Moderator Team
Moderator Team
Posts: 5368
Joined: Mon Aug 28, 2017 10:56 pm
Location: Glasgow, UK

Re: files read corruption issue

Post by terriblefire »

MaxTransfer is the amiga groundhog day issue. Just what you need during lockdown.

Its insidious because it doesnt happen on winuae.
———
"It is not necessarily a supply voltage at no load, but the amount of current it can provide when touched that
indicates how much hurting you shall receive."
richx
Posts: 46
Joined: Sun Aug 11, 2019 5:03 pm

Re: files read corruption issue

Post by richx »

@go0se Thanks for confirming!

Elusive, interesting that most large files worked and had to be written by LHA to trigger.
Post Reply

Return to “TF536”