Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MCsL / cast lists #9

Closed
MrBrax opened this issue Jul 6, 2017 · 33 comments
Closed

MCsL / cast lists #9

MrBrax opened this issue Jul 6, 2017 · 33 comments

Comments

@MrBrax
Copy link
Contributor

MrBrax commented Jul 6, 2017

Due to this: https://github.com/Brian151/OpenShockwave/blob/master/docu/FormatNotes.txt#L334
and me getting some kind of success reading the format, is the spec there correct, or is it just guessing?

Here's some progress I've made:
1499341843023

(albeit it breaks if casts are external and uses paths)

But i can't figure out how it assigns the cast members to the cast library, i think this is where you people are stuck too, yeah?

One interesting note is that on some of the CAS* entries, the first integer more often than not matches up with what cast number the first member has.

@tomysshadow
Copy link
Collaborator

tomysshadow commented Jul 6, 2017

Nope, we already figured that out - or rather, it was already implemented in Schockabsorber. That's all done in the KEY* section. If I recall correctly, if the ID in the key section is greater than 1024 (since, I believe, the maximum number of casts is 1000) then the ID instead refers to which cast library the item is owned by. So the KEY* section takes on two purposes in that sense.

Also, haven't had a chance to document this but I should add: the IDs in MCsL refer to the CAS* chunks, based on their index in mmap. The CAS* are a table of the IDs of CASt chunks in the mmap table, iirc, and the chunks those CASt rely on are found using KEY*. So it goes MCsL > CAS* > mmap > CASt >KEY* > Data (such as BITD, snd , etc.) note that multiple chunks can be linked to one CASt in KEY*, for example the sound headers can be seperate of the sound itself using sndH and sndS

This is one of my more recent realizations and it is with this that I finally begin to understand the structure somewhat.

Also, note that LCsM isn't present in older Director movies as not all versions of Director have cast libraries.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

Wow, that's a weird implementation. I don't really see how it would add up though, i'm testing out with a game dxr here (same as on the screenshot) that has 4 internal cast libraries, does it just modulo 1024 then?

@tomysshadow
Copy link
Collaborator

No; take a look at the KEY* section. If an item belongs to the first cast library it'll have an ID of 0x0400. If it belongs to the second it'll have an ID of 0x0401. If it belongs to the third it'll have an ID of 0x0402. I assume it was done this way to maintain some kind of compatibility with Director 4. If the file is little-endian, it'll be in reverse and be 0x0004, 0x0104, 0x0204 etc. so it IS one ID, not two numbers.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

But how does it retain the cast member numbers in the other libraries then? Just appending everything? Pretty confused.

@tomysshadow
Copy link
Collaborator

I don't know what you're asking. CAS* refers to IDs in the mmap, not KEY*. The purpose of KEY* is to establish a relationship of being linked together. As in, "this CASt is linked to this BITD" or "this CASt is linked to this snd " or "this member is linked to this library"

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

yeah i'm super confused, those aren't the kind of values i'm getting out of the key entry

1499344747530

@tomysshadow
Copy link
Collaborator

tomysshadow commented Jul 6, 2017

Let me demonstrate with some images. I forgot the IDs are four bytes long, but I mostly remembered correct.

keyone
Here you can see that ID 0x00000014, which is the index of a BITD in the mmap, is being linked to 0x00000013, the index of its associated CASt in the mmap.
keytwo
Here, we can see that ID 0x000000FA, the index of a THUM in the mmap, is being linked to the same CASt. This makes sense - the BITD contains the image, and THUM contains the image's thumbnail. Therefore, they are both linked to the same CASt.
keythree
Same story here. We can see a snd chunk being linked to a CASt chunk.
keyfour
Now this one is different. This is saying this CAS* chunk is linked to the first cast library. This is because the ID to link to is greater than or equal to, in this case, 1024, therefore meaning it refers to a cast library, in this case: the first cast library. This movie doesn't have an MCsL chunk, but if it did, it would say what the name of that first cast library is (usually "Internal.") Therefore, all the CASt chunk IDs listed in that CAS* chunk belong to the first cast library or "Internal."
keyfive
Furthermore, this Lctx chunk is also linked to the first cast library, as it is being linked to a number greater than or equal to 1024.

0x00000401 would link to cast library one. 0x00000402 would link to cast library two. etc.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

So it links the entries to the cast library itself and not a CASt? How would it then know what member to link to?

@tomysshadow
Copy link
Collaborator

tomysshadow commented Jul 6, 2017

CAS* contains the IDs of CASt chunks in the mmap. So in examples one, two, and three above, you can see how you'd go from CASt to BITD for example. And CAS* contains IDs of CASt chunks. And you can tell which cast library a CAS* represents from looking at the KEY* chunk, like in example four. Then you can find the name of that cast library from MCsL.

Sorry if I'm doing a poor job of explaining it but frankly I find it confusing myself.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

yeah this is way too confusing for me, and i just remembered there are two key* entries so i got those confused too

i think i'm starting to understand it though, data entries and cast entries are different and links aren't really static

but the thing i wanted with all of this is just getting a list of all the cast members in their correct libraries, and i ran into so much other stuff i can barely focus anymore

@tomysshadow
Copy link
Collaborator

OK. The CAS* chunks are a list of IDs of CASt chunks. In other words, CAS* = list of things in a single cast library. The KEY* chunk says which cast library that is. First, second third... and it does this using IDs greater or equal to 1024.

The KEY* section also has another purpose, which is to establish a link between CASt chunks and the actual data they refer to, and this is done with just normal IDs. So once you have your list of CASt chunks, from the CAS* chunks - that's how you get to the data.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

oh so it's a specific case when it's cas*? well alright

i still don't understand how all this ties together though, but i have several diagnoses so i guess it's not that strange ¯\(ツ)

@tomysshadow
Copy link
Collaborator

No, it's not specific to CAS*. For example an Lctx can also be linked to a cast library. But for your purposes you probably care about CAS* the most. The thing that makes it a link to a cast library is the ID being greater than or equal to 1024.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

i think i'm getting it now, but what about the link id being like 1500? that's on other files but still

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

Got it all working, thanks for the help.

But for the record, cast libraries can contain more than 1000 items:

1499354911091

@Brian151
Copy link
Owner

Brian151 commented Jul 6, 2017

holy....................

All of my notes on MCsL came from my own research, aided by a friend, since i couldn't run director...

So some MOVIES lack it, also... great...
We seriously need to locate, pull, and read version info from these things soon...

I didn't know how these link-up to CAS* was yet discovered...
I also potentially had found one more thing for the brief amount of time I was able to use my own computer. But, not sure I wrote it down, and said computer won't stay running long enough for me to extract my work, anyways...

@tomysshadow
Don't you mean if the first 2 bytes of the ID are 1024, it's a link to the cast library?

I will ignore pretty much, everything else that was said here

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

149936419188294

Here's another mindbender ;)

@Brian151
Copy link
Owner

Brian151 commented Jul 6, 2017

It might help if you showed the actual hex view, also?

I will try to look at MCsL again very soon

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 7, 2017

@Brian151
4C 73 43 4D A1 00 00 00 00 00 00 0C 00 00 00 04 00 04 00 00 00 11 00 00 00 00 00 00 00 00 00 00 00 0A 00 00 00 0A 00 00 00 0C 00 00 00 14 00 00 00 1E 00 00 00 1E 00 00 00 20 00 00 00 28 00 00 00 2E 00 00 00 2E 00 00 00 30 00 00 00 38 00 00 00 41 00 00 00 41 00 00 00 43 00 00 00 4B 08 49 6E 74 65 72 6E 61 6C 00 00 00 00 01 00 29 00 00 04 00 08 47 72 61 70 68 69 63 73 00 00 00 00 02 00 8E 00 00 04 01 04 4C 6A 75 64 00 00 00 00 01 00 5A 00 00 04 02 07 50 65 74 74 46 69 6E 00 00 00 00 01 00 F3 00 00 04 03 00

I checked out MBARN.DXR and it's completely different from what i've been working with, must have been a big overhaul in director between those games

@Brian151
Copy link
Owner

Brian151 commented Jul 7, 2017

k thx

@Brian151
Copy link
Owner

Brian151 commented Jul 7, 2017

@MrBrax
You got some strange things going on with this, at least, compared to what I've seen so far...

[analysis of two MCsL, @MrBrax 's example, and the one for Spybotics: The Nightfall Incident]

//from spybotics

//offsets?
00 00 00 00  00 00 00 00  00 00 00 0A  00 00 00 0A 
00 00 00 0C  00 00 00 14  00 00 00 1C  00 00 00 1C 
00 00 00 1E  00 00 00 26  00 00 00 2D  00 00 00 2D 
00 00 00 2F  00 00 00 37  00 00 00 3D  00 00 00 3D 
00 00 00 3F  00 00 00 47  00 00 00 4E  00 00 00 4E 
00 00 00 50  00 00 00 58  00 00 00 62  00 00 00 62 
00 00 00 64  00 00 00 6C  00 00 00 77  00 00 00 77 
00 00 00 79  00 00 00 81  00 00 00 8C  00 00 00 8C 
00 00 00 8E  00 00 00 96  00 00 00 A5  00 00 00 A6 
00 00 00 A8  00 00 00 B0  00 00 00 BF  00 00 00 C0 
00 00 00 C2  00 00 00 CA  00 00 00 D9  00 00 00 DA 
00 00 00 DC  00 00 00 E4  00 00 00 F3  00 00 00 F4 
00 00 00 F6  00 00 00 FE  00 00 01 0D  00 00 01 0E 
00 00 01 10  00 00 01 18  00 00 01 22  00 00 01 22 
00 00 01 24  00 00 01 2C  00 00 01 38  00 00 01 38 00 00 01 3A

//dataLength
00 00 01 42

//data
{
	08 49 6E 74 65 72 6E 61 6C // "Internal"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 95 // 149 members
	00 01 // cast #1
	04 00 // 1024
}
{
	06 76 69 73 75 61 6C // "visual"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	01 C1 // 449 members
	00 02 // cast #2
	04 00 // 1024
}
{
	05 74 69 6C 65 73 // "tiles"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	02 45 // 581 members
	00 03 // cast #3
	04 00 // 1024
}
{
	04 6D 69 73 63 // "misc"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 2E // 46 members
	00 04 // cast #4
	04 00 // 1024
}
{
	05 73 6F 75 6E 64 // "sound"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 53 // 83 members
	00 05 // cast #5
	04 00 // 1024
}
{
	08 77 6F 72 6C 64 4D 61 70 // "worldMap"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 98 // 152 members
	00 06 // cast #6
	04 00 // 1024
}
{
	09 74 65 78 74 20 64 61 74 61 // "textdata"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 6F // 111 members
	00 07 // cast #7
	04 00 // 1024
}
{
	09 6C 65 76 65 6C 44 61 74 61 // "levelData"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 27 // 39 members
	00 08 // cast #8
	04 00 // 1024
}
{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 31 // "sound_level_1"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}
{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 32 // "sound_level_2"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}
{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 33 // "sound_level_3"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}
{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 34 // "sound_level_4"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}
{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 35 // "sound_level_5"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}
{
	08 74 75 74 6F 72 69 61 6C // "tutorial"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 39 // 57 members
	00 09 // cast #9
	04 00 // 1024
}
{
	0A 61 6E 69 6D 61 74 69 6F 6E 73 // "animations"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 28 // 40 members
	00 0A // cast #10
	04 00 // 1024
}

//MrBrax's sample

//offsets
00 00 00 00  00 00 00 00  00 00 00 0A  00 00 00 0A 
00 00 00 0C  00 00 00 14  00 00 00 1E  00 00 00 1E 
00 00 00 20  00 00 00 28  00 00 00 2E  00 00 00 2E 
00 00 00 30  00 00 00 38  00 00 00 41  00 00 00 41  00 00 00 43

//dataLength
00 00 00 4B

//data
{
	08 49 6E 74 65 72 6E 61 6C // "Internal"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 29 // 41 members
	00 00 // cast # ???
	04 00 // 1024
}
{
	08 47 72 61 70 68 69 63 73 // "Graphics"
	00    // null
	00 00 // preload when needed
	00 02 // stored how???
	00 8E // 142 members
	00 00 // cast # ???
	04 01 // 1025
}
{
	04 4C 6A 75 64 // "Ljud"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 5A // 90 members
	00 00 // cast # ???
	04 02 // 1026
}
{
	07 50 65 74 74 46 69 6E // "PettFin"
	00    // null
	00 00 // preload when needed
	00 01 // stored internally
	00 F3 // 243 members
	00 00 // cast # ???
	04 03 // 1027
}

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 7, 2017

Here's one for a DXR with external casts:
4C 73 43 4D 42 01 00 00 00 00 00 0C 00 00 00 05 00 04 00 00 00 15 00 00 00 00 00 00 00 00 00 00 00 0A 00 00 00 0A 00 00 00 0C 00 00 00 14 00 00 00 18 00 00 00 36 00 00 00 38 00 00 00 40 00 00 00 46 00 00 00 66 00 00 00 68 00 00 00 70 00 00 00 78 00 00 00 9A 00 00 00 9C 00 00 00 A4 00 00 00 AE 00 00 00 D2 00 00 00 D4 00 00 00 DC 08 49 6E 74 65 72 6E 61 6C 00 00 00 00 01 01 33 00 00 04 00 02 30 30 00 1C 43 3A 5C 64 6F 6B 5C 4D 75 6C 6C 65 5C 4D 76 4E 6F 50 72 6F 74 5C 30 30 2E 63 73 74 00 00 00 00 01 01 ED 00 00 04 03 04 64 61 74 61 00 1E 43 3A 5C 64 6F 6B 5C 4D 75 6C 6C 65 5C 4D 76 4E 6F 50 72 6F 74 5C 64 61 74 61 2E 63 73 74 00 00 00 00 01 00 0B 00 00 04 05 06 43 44 64 61 74 61 00 20 43 3A 5C 64 6F 6B 5C 4D 75 6C 6C 65 5C 4D 76 4E 6F 50 72 6F 74 5C 43 44 64 61 74 61 2E 63 73 74 00 00 00 00 01 05 6E 00 00 04 07 08 74 65 6D 70 50 6C 75 67 00 22 43 3A 5C 64 6F 6B 5C 4D 75 6C 6C 65 5C 4D 76 4E 6F 50 72 6F 74 5C 74 65 6D 70 50 6C 75 67 2E 63 73 74 00 00 00 00 00 00 00 00 00 04 08

@Brian151
Copy link
Owner

Brian151 commented Jul 7, 2017

Will analyze tomorrow...
Must sleep soon

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 7, 2017

Some more discoveries:

Strings in older versions (e.g. MBARN.DXR) doesn't have a null byte at the end.
Version can be read in DRCF (https://gist.github.com/putnamhill/2921806)
Path for casts in MCsL is right after the name, and if it exists skip 2 more bytes

Checking out dirapi.dll, found some proper definitions:
1 Bitmap
2 Film Loop
3 Field
4 Palette
5 PICT
6 Sound
7 Button
8 Shape
9 Movie
10 Digital Video
11 Script
12 Text
13 OLE
14 Transition

@Brian151
Copy link
Owner

Brian151 commented Jul 10, 2017

I think I see what's going on with the offset tables. I suspected they were like CASt, I was right (mostly), what threw me off is this:
Let's assume each entry needs four offsets to point to its data
k...

divide the length of ANY offset table by four, you will ALWAYS have
someNumber.25

Now, what might cause this?
One of the offsets in the table is either a pointer to the data, itself (presumably relative to the end of the offsets table), or a junk entry added in for whatever reason.

I will have to revise my notes
That extra byte* in external casts still exists, but based on this new research, and your recent example:
THIS:

{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 31 // "sound_level_1"
	00    // null
	00 00 // preload when needed
	00 00 // stored externally
	01    // unknown
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}

may actually be THIS:

{
	0D 73 6F 75 6E 64 5F 6C 65 76 65 6C 5F 31 // "sound_level_1"
	00      // null
	00      // empty String filePath
        00 00 // preload when needed
        00 01 // huh...
	00 00 // ??? members
	00 00 // cast # ???
	04 00 // 1024
}

fun stuff...

I will try to get the latest stuff up, but that might not happen till tomorrow
Ignoring that a certain value's purpose is now put into question, and we have yet to find any examples where... OTHER information is in MCsL, I'd say this section is almost fully cracked!

extra byte is not just a byte, it's another string, the external .cst file path

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 10, 2017

yeah the file path is most likely correct

1499676297883

all this seems right, it's just those offsets/values in the beginning now it feels like

@tomysshadow
Copy link
Collaborator

tomysshadow commented Jul 10, 2017 via email

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 10, 2017

i've had the "cast fields" array thing work on every cast so far, remember that director versions differ a lot

@Brian151
Copy link
Owner

i'll reply later... busy

@Brian151
Copy link
Owner

right, so...

@tomysshadow
in regards to mmap generation:
This is the most logical way to generate it, bear in mind that mmap is an offset table, and one appended before the data in a director file. (by default) It needs to be generated last. It's kinda hard to explain... I've contemplated how I might save the files, and arrived on this conclusion.

CASt types:
they have a 'standardized' make-up, at least from my observations. The kicker is it's a section full of optional fields, and so far, the only way to know if a field is there is to examine the length and contents of the offset table. I also have yet to fully identify the fields, either. There's also an optional specificData at the end, this is where the serious variation should begin, and each specificData is in itself, a smaller file format. However, as @MrBrax said, there could be version discrepancies. Remember it's already been confirmed the protected variants work ever so slightly different, but many of the concepts are effectively, the same.

STXT:
according to @MrBrax, he found the full list of types...
3: field
7: button
11: script
I have already positively confirmed a link between CASt type 11 and STXT sections, hence my assumption S[cript]T[e]XT being the intended meaning of STXT's id. However, the full format and purpose is not yet understood. fields and buttons, iirc, both are highly compatible with the idea of being scripted?
Ultimately, more research is required into this whole matter.

@MrBrax
Your screenshot and my notes do not fully seem to match? Also, the storageType value is now brought into question, as external casts CAN have this value set to 1... Let's also not forget that it's been observed as 2. Perhaps the presence of even an empty String filePath is what differentiates internally and externally stored casts? For now, we can't be fully certain what's going on... One thing that I really do not get is how the entries are structured...
we have:
a field for the name
a field for the file path
a field for just the preload settings, and it's comprised of one single u16
a field for all other attributes, comprised of u16's
Why is the preload settings an entirely separate thing from the other attributes???!

That's all

Also, again... let's keep issues to their primary topic.
Let's open a new one if CASt is going to be a focus, since the intended scope of this was the MCsL, and it could get chaotic if we go beyond that...

That, or we can create a generic "sections" issue?
(ok, that's probably a very bad idea...)

But, let's try not to have multiple and fairly separate ideas be discussed on same thread, and especially if it should happen on multiple threads.

Much easier to keep track of

Hope everyone understands?

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 11, 2017

For the file/note differences:

1499760082150

These are the game files I'm comparing and traversing, so i can't really remember which ones I've sent to you or what's currently open in my hex editor :)

@Brian151
Copy link
Owner

let's see, I tend to have easily up to 10 unnamed 'files' when I'm doing stuff with this format. Or, I have 10-20 that I just click-dragged... So, I can relate to forgetting these things...

So, is it safe to close this issue, now?

@Brian151
Copy link
Owner

closing, because this matter is mostly resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants