mtools (mcopy) acting weird?

[Octacone]

I thought I was going crazy but apparently I wasn't:
Every time I build my bootloader something crazy happens. There is a file called Stage_2.sys that gets copied on my virtual HDD using mcopy. So where is the problem?
"cluster_low" of that file(inside the root directory (of FAT 32)) increases by 1! So far it has reached 386!
What is going on, there is no logical explanation??? It is like the ONLY file on the entire HDD, there is free space everywhere, FAT is practically empty and mcopy decides to increment cluster_low by 1 each time and thus moving the entire file around...
Anybody has an idea of what could be going on? This is all of course happening on Linux.
Here are commands I use:

Code: Select all

sudo mcopy -o Output/Stage_2.sys c:
# this is how c: is defined:
drive c: file="/pathTo/HDD.img" partition=1

[iansjack]

I suspect that this behaviour is by design. It increases the chances of recovery for deleted files as the data is not overwritten until there are no free clusters that haven't be used before. If you want the file to be written in the same place you need to format the disk before rewiting it.

[Octacone]

I suspect that this behaviour is by design. It increases the chances of recovery for deleted files as the data is not overwritten until there are no free clusters that haven't be used before. If you want the file to be written in the same place you need to format the disk before rewiting it.

Is there a way to prevent this permanently? I tired removing that file and then putting it back again, but doesn't work.
What do you mean by format the disk before rewriting it? Format in what way? It is already formatted as FAT 32.

[iansjack]

No. I don't think you can alter this behaviour, assuming that it is part of the design of the FAT32 filesystem (other than writing your own handler for the filesystem). It shouldn't matter to a well-behaved program where a file is located on the disk.

By formatting the disk I mean that, if you want the file to occupy the same clusters on the disk you need to return it to exactly the same state as it was before you originally wrote the file. Deleting a file doesn't do this (for example, it leaves the directory entry in place with a marker indicating that the file has been deleted). So you need to format the disk again each time you want to write the file to that location. The real answer is that your program shouldn't depend on the absolute position of a file; it should parse the directory entry and file allocation table to determine the clusters used by the file.

[Octacone]

No. I don't think you can alter this behaviour, assuming that it is part of the design of the FAT32 filesystem (other than writing your own handler for the filesystem). It shouldn't matter to a well-behaved program where a file is located on the disk.

By formatting the disk I mean that, if you want the file to occupy the same clusters on the disk you need to return it to exactly the same state as it was before you originally wrote the file. Deleting a file doesn't do this (for example, it leaves the directory entry in place with a marker indicating that the file has been deleted). So you need to format the disk again each time you want to write the file to that location. The real answer is that your program shouldn't depend on the absolute position of a file; it should parse the directory entry and file allocation table to determine the clusters used by the file.

Even if I delete its directory entry the same thing happens. By design just doesn't sound correct. I took some time and had a look at other "real" FAT 32 formatted storage mediums. They all seem perfectly fine, FAT seems to be filled contiguously and first file has a cluster number of 3 (not 391 while having 390 free entries in front of it!). I am trying to make my bootloader as flexible as possible so there aren't much assumptions really, that is not a problem. What matters is that every time I press F5 (Build and Run) my file gets moved by 1 in the file allocation table leaving N - 1 free spaces behind it. Eventually FAT will collapse. Another thing I noticed is that FAT[3] contains F8 FF FF 0F (little endian) without having a file that corresponds to it.

And on last thing, after heavy experimenting and trial and error I managed to break everything

Code: Select all

Fat problem while decoding 2 0
Streamcache allocation problem:C 2

Edit:
Adding F8 FF FF 0F back to FAT[3] somehow fixed to issue...
Don't know how it was related cause it wasn't linked to any file, it looks like it was put there by mkdosfs.

[iansjack]

That FAT[3] entry is, I presume, for the root directory. Deleting it would certainly render the filesystem unusable.

Again, I emphasise, your programs should not rely on a file occupying a particular cluster. You need to obtain the position from its directory entry.

Note that the file information sector in FAT32 contains a field detailing the number of the last allocated cluster. You might consider what this field is used for.

[Octacone]

That FAT[3] entry is, I presume, for the root directory. Deleting it would certainly render the filesystem unusable.

Again, I emphasise, your programs should not rely on a file occupying a particular cluster. You need to obtain the position from its directory entry.

Note that the file information sector in FAT32 contains a field detailing the number of the last allocated cluster. You might consider what this field is used for.

Actually the root directory has its cluster number set to 0. While inside the BPB it is set to 2.
My code doesn't assume anything, it actually reads the cluster_low and cluster_high number and stores them for needed computation.

NICE CATCH: "FSInfo Sector" contains a field called "FSI_Nxt_Free", and guess what, it contains a value of 395. So that is where those numbers were coming from.
Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.

[iansjack]

Actually the root directory has its cluster number set to 0.

No. Cluster 0 isn't a valid data cluster and, in FAT32, the root directory is just another file.

Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.

I've already explained why FAT32 would write to unused clusters rather than previously used ones. It makes data recovery more likely. In addition, I would expect that it eases the task of finding a set of contiguous clusters large enough to hold a new file. Once all clusters have been used it will have to start reusing old ones - starting from the beginning again, these will be the clusters for the oldest deleted files.

If your code assumes nothing, why does this behaviour bother you?

[Octacone]

Actually the root directory has its cluster number set to 0.

No. Cluster 0 isn't a valid data cluster and, in FAT32, the root directory is just another file.

I am aware of that but:
Root_Directory_Cluster value inside the BPB = 2 (it is actually okay since arrays are zero based, forgot about that, I wasn't talking about FAT[3] but FAT[2])
Root_Directory_Entry itself cluster value = 0 (I think it should be zero actually, no clusters linked = 0 for standard directories (files) and 0 because it entry no.1)

Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.

I've already explained why FAT32 would write to unused clusters rather than previously used ones. It makes data recovery more likely. In addition, I would expect that it eases the task of finding a set of contiguous clusters large enough to hold a new file. Once all clusters have been used it will have to start reusing old ones - starting from the beginning again, these will be the clusters for the oldest deleted files.

Just tested this with an actual USB drive. I created an empty FAT 32 partition, put a file inside it, checked the value and it was 3, then I added another file and replaced the first one, 3 became 4 and newly create file had a value of 5. True!

If your code assumes nothing, why does this behaviour bother you?

That adds a whole new layer of complexity, like a lot. I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

[Octocontrabass]

I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

You could always use another sector for additional code, like how Microsoft does.

[Octacone]

I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

You could always use another sector for additional code, like how Microsoft does.

Where am I allowed to put it? I know there are some reserved sectors, but Microsoft says they should be left unused for future expandability (that is likely to never happen in case of FAT).
I did notice some code in unusual places when inspecting a FAT 32 formatted USB doe.

[Octocontrabass]

DOS-based Windows places the extra bootloader code in the third sector of the partition (LSN 2), right after the FSInfo sector. Like this. Since it's part of the boot record, there's a backup copy six sectors later (LSN .

Windows NT places the extra bootloader code in the thirteenth sector (LSN 12). Like this. I have no idea why they changed it.

[Octacone]

DOS-based Windows places the extra bootloader code in the third sector of the partition (LSN 2), right after the FSInfo sector. Like this. Since it's part of the boot record, there's a backup copy six sectors later (LSN .

Windows NT places the extra bootloader code in the thirteenth sector (LSN 12). Like this. I have no idea why they changed it.

Thanks for the links! Knowing this will be very helpful.