Merge functionality gives errors on restore and seems to recover unrelated files

[reasv]

I tried using the new merge functionality on the 5 partitions I referenced in issue #18 , however the recovery wasn't successful.
Here is the output:
https://pastebin.com/sns4t84A
Notice first how the root doesn't contain any files. In the individual partitions I had recovered before, some of them had files in the root directory, therefore I would assume that at least those files would end up in the root once I merged these partitions.

As for the LostFiles, it has very few files and none of them were part of the original partition.
In fact from the bodyfile, I noticed that their timestamps were from before the original partition was even created.

Here are the exact commands I executed:

merge 312 451
merge 312 283
merge 312 248
merge 312 1007

Then I tried to restore partition #312.
After the merge, it was listed like this when issuing the recoverable command:

Partition #312 -> Partition (NTFS, ??? b, 576538 files, Recoverable, Offset: 264192, Offset (b): 135266304, Sec/Clus: 8, MFT offset: 4287141256, MFT mirror offset: None)

[Lazza]

Write command ("help" for details):
> restore 312 5
INFO:root:Restoring #5 Root

Did you run tree or any other command on 312 after merge but before restore? If you did not (which is fine, of course) then it should show something like:

Write command ("help" for details):
> restore 312 5
Rebuilding partition...
Done
INFO:root:Restoring #5 Root

[Lazza]

Never mind. I found an issue with the merge, however I checked that merging generates another problem in which file names get screwed in some cases (due to a double rebuild).

I am going to check if there is a good way to fix this.

[Lazza]

Please try this one. There could still be issues if you run merge after having rebuilt a partition, for instance you could incur into useless files left in LostFiles.

I suggest you run merge before doing anything else, however feel free to try also after listing them separately if you want to debug a bit of corner cases. 😉

[reasv]

Thank you.
I'm going to test this ASAP, but I would also like to point out that there's a small bug that has persisted from when you first added merge.
In ntfs.py, you do
from ..utils import merge
Then later on in the code you do
utils.merge()
This is an issue since utils itself was not imported, and the program crashes.
I manually fixed it in my local copy, and I considered doing a pull request but it seems like too small of an issue for that.
Just change utils.merge() into merge()

[Lazza]

Woops! 😅 OK done, thanks.

[reasv]

I tried it again, and sadly it gives me a similar result.
I did merge immediately after listing recoverable.
Then I saved the bodyfile and csv of partition 312 (the one I merged the others into)

But I'm getting a few unrelated random files again, and nothing from my old partition.

Despite 312 being listed as having lots of files:

Partition #312 -> Partition (NTFS, ??? b, 584691 files, Recoverable, Offset: 264192, Offset (b): 135266304, Sec/Clus: 8, MFT offset: 4287141256, MFT mirror offset: None)

I didn't do any operations except merge with 312 on any other partitions.

[Lazza]

BTW why are you merging everything into 312? The main piece is the first one, in your case 248. I am not sure if this makes any difference but you should try merging all the secondary parts into the main one.

[reasv]

I tried that and it seems to be working.
I'm gonna report once I've copied and examined the files.

[reasv]

Ok, I merged them in order by partition #, starting from 248, 283, 312, 451, 1007.

Then I restored all files by restoring 248.

At first it seemed like it was restoring them exactly like it did when I restored the 5 partitions individually. The same folders were in the Root folder, the majority of files were in the LostFiles folders, and everything seemed similar. As if I had restored the 5 partitions one by one and just merged the output folders together. Unfortunately by this I also mean that I found no corrupted files that got "repaired". Most files were corrupted.

Then I noticed something very odd. The size of the output folder was 1/3 the size of the output folder of the 5 partitions I restored individually.
Then I realized that a huge folder that was in the Root folder of restored partition 451 was missing from the output of the merged partition. The output of restoring partition 451 by itself is bigger than the total I got from restoring the merged 5 partitions.
So there was definitely something missing.

I thought that maybe I forgot to include 451
I restarted the program and did it again.
And this time I was careful to merge them all into 248, and in order.
It gave me the same result (predictably, since I am sure I merged them all the first time too)

Also when I list recoverable, partition 248 has less files than the sum of the other partitions'
In fact it has less files than even just partition 312 itself:

Partition #248 -> Partition (NTFS, ??? b, 307804 files, Recoverable, Offset: 264192, Offset (b): 135266304, Sec/Clus: 8, MFT offset: 2698650792, MFT mirror offset: None)

This is different from when I merged starting from 312, where you could see there were 500k+ files.

[Lazza]

Unfortunately by this I also mean that I found no corrupted files that got "repaired". Most files were corrupted.

Well, this is not really something we can do much about. If the files were partially overwritten those parts cannot be recovered.

I did not notice that the partitions were "out of order". Try merging them in the MFT order, they should be like this if I am correct:

Partition  #451 -> Partition (MFT offset:  614360032, MFT mirror offset: None)
Partition #1007 -> Partition (MFT offset: 2592906944, MFT mirror offset: None)
Partition  #248 -> Partition (MFT offset: 2698650792, MFT mirror offset: None)
Partition  #283 -> Partition (MFT offset: 3418165840, MFT mirror offset: None)
Partition  #312 -> Partition (MFT offset: 4287141256, MFT mirror offset: None)

This ordering thingie might be automated if we see that it works but I would like to try a bit before pushing other modifications. Also, could you please share privately with me all those CSV files (merged and not merged)? Thanks.

[reasv]

Hi, sorry for the late response.
I tried it in that order and only got a small amount of random files, which I don't think were in the partition.
As for the CSV, ok I'll send them to you ASAP