Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8-bit BITD issues #5

Open
MrBrax opened this issue May 18, 2017 · 45 comments
Open

8-bit BITD issues #5

MrBrax opened this issue May 18, 2017 · 45 comments

Comments

@MrBrax
Copy link
Contributor

MrBrax commented May 18, 2017

So the converter doesn't work properly for the files i have, it skips a third of the height, and the length of the files does not correspond to (width*height) so there has to be some kind of compression (RLE?).

I've almost completed a python script that can extract text, WAV & AIFF, and BITD is the one i'm stuck on, been working on this thing almost every wake hour for a couple of weeks now, and just found this project. Really interesting stuff!

Since the converter only has 24 & 32 depth selectable, I'm assuming the format hasn't been deciphered yet?

oh and also, exported bmp's from director all have the same palette (in-game images always have palette set to -1) so i'm thinking the bytes aren't corresponding to colors but palette positions?

@MrBrax
Copy link
Contributor Author

MrBrax commented May 18, 2017

I almost have a working 8-bit extractor now, just gotta fix it leaving out some pixels at the end

@tomysshadow
Copy link
Collaborator

I haven't had a chance to look at a lower bitdepth BITD yet, that's all. However from what Brian tells me the lower bitdepth BITDs use colour palettes in the form of Photoshop's CLUT format, which are in different chunks. Multiple can exist in one file, and I'm not sure yet how those BITDs are linked with their CLUT. Either way, I'll add it in at some point - the BITD Reader is still quite new.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 18, 2017

It's quite an interesting format. It uses linear & rle encoding in the same way, but with palette entries instead of rgb yeah.

My script can put out proper bitmaps of most bitd files, but some get distorted like so:
1495106008672

Setting the X coordinate to (length - j) in the loop fixes it:
1495106280780

But breaks all the other files. Must be a value somewhere defining this.

@tomysshadow
Copy link
Collaborator

tomysshadow commented May 18, 2017

That distorted image looks to me like the width is just one pixel off. I'm going to have to try implementing 16-bit images, to see if I get similar problems. It may be something implicit. Perhaps the rightmost pixel is getting cut off because it's all the same colour. If you hadn't already noticed, alpha values are sometimes reversed if alpha isn't used in a 32-bit image.

I am aware of how the format works. I figured it out in a hex editor in order to write SeismoGRAPH.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 18, 2017

Figured it out with a tip from a friend. If the width is even, go to x -1 instead of 0 on the line change!
149513607901146
Works great, using the palette from a BMP extracted with Director. Transparency is gonna be a pain though, since it's on the sprite itself and not the bitmap.

@tomysshadow
Copy link
Collaborator

tomysshadow commented May 18, 2017

So you're suggesting the width is always even and if it isn't an extra column of pixels is added? I'll have to add that in. I wonder what your code looks like in comparison to mine, it's probably more advanced considering it is actually inspecting the entire file. We're currently in the process of switching over to using Haxe instead, since with JavaScript which we've been using thus far we won't be able to load Xtras. That's why SeismoGRAPH is disconnected from everything else.

If the image is 32-bit then alpha transparency is just another channel before the red, green and blue. If it's using JPEG compression, which is completely different of course, it's in a separate ALFA chunk.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 18, 2017

It's difficult to explain. I uploaded my script here: https://gist.github.com/MrBrax/1f3ae06c9320863f1d7b79b988c03e60

@Brian151
Copy link
Owner

palettes are generally linked by their cast member ID. However, there's an issue I've had with specifying a palette from another cast. (fun fact: this seems to crash Director when attempting to edit said bitmap) The linkage is done in the bitmap 'specificData' block of a bitmap member. the CLUT chunks should be linked to their respective palette cast members like everything else? The linking of CASt header chunks (i won't call them meta-data because the INTENDED meaning of this is "extra data not needed by program", CASt are quite important, and specify the type of information typically found in a header) to the actual data chunks is something I need to start working on again...

As for the bitmap data itself, my efforts can confirm that at least the 8 bit BITD CAN be uncompressed. I have yet to explain how it differentiates between the two, or what would cause it to use either. One of my sample files (I imported MineCraft textures) compressed the 8-bit bitd. I later tried to create an 8-bit bitmap from something that actually would benefit from compression. No such luck. Director outright REFUSED to compress the bitmap. I tried many times and got raw indexed 8BPP data in my BITD. On my compressed sample, I tried finding any indicator how that worked, and no particular luck. And then my desktop started acting up. I suspect hardware failure is really the cause, at this point. Anyways, I'm back on my parent's computer. No Director (or a lot of my latest progress/samples), but Git works, at least.

All I'll say about BITD's is I HATE them... I just can't come to understand them, and I'm glad you two can. embedded resources are easy, W3D will be a challenge to parse/display, and the CLUT things, well... only thing I don't understand is the second value. But, I feel confident I can ignore it for the time being.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

It's the worst! I've been trying to extract all the CXT/DXR resources from a game and it's coming along pretty good, except now some images come up corrupted (png are converted, ignore):
1495200532786

And just as i'm writing this, i'm noticing that this: https://github.com/Brian151/OpenShockwave/blob/master/docu/FormatNotes.txt#L235 could infact indicate if the image has alpha somehow, i dunno really.

There's also the "Sprite" object thing, is that bound to the CAST, or is it a separate object somewhere? Would maybe explain the "Ink" feature and more.
scratch that, they use the "score" list thing

@Brian151
Copy link
Owner

alright

This observation is among the progress I have lost. Sadly, I once again can't actually check that kind of stuff out in Director, so...

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

Can't you install the Director trial? But i guess if it's not your PC then it's not very appropriate.

I think there's something wrong with the Director UI or dirOpener's conversion of the files, because except for a few bitmaps, it says it's 8-bit and no alpha, but because i can't convert those files like i can with all the other ones, there has to be something different.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

My god, it was this line all along, that i used: https://github.com/Brian151/OpenShockwave/blob/master/tools/BitdReader/js/BitdReader.js#L72

I changed it to 0x100 and now it works.

@tomysshadow
Copy link
Collaborator

It shouldn't? I specifically had to use 0x101. Does it work in all cases?

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

Yes, but it might be for 8-bit only. 1-bit bitmaps are still corrupted (which i haven't gotten to yet).

@Brian151
Copy link
Owner

It's a matter of disk space and CPU/RAM that I won't... Pretty sure this computer couldn't handle it. As is, my computer kinda struggled running it.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

Really? It's a pretty simple program in my opinion, i use MX 2004 (10.1) and it's like 80MB installed

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

Anyway, here's something fun:

161: 13x16, 1bpp

00 E0 07 20 38 20 20 10 10 10 10 10 10 08 20 38 20 40 40 40 40 80 80 80 81 00 61 00 19 00 06 00

and

181: 16x13, 1bpp

00 0C 00 32 39 C2 26 02 20 01 40 01 40 01 80 0E 80 30 80 C0 71 00 0D 00 03 00

149521471985838

Is it using every byte multiple times or something like that? I can't see how it would work otherwise.

@tomysshadow
Copy link
Collaborator

For such a simple image I don't see any semblance of a pattern. What happens if you change the bytes to slightly different values? Experimentation is the easiest way to figure this out. Just make a backup of the original.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

So from what i understand it now, every byte corresponds to a different pattern in 8px wide chunks, that's more of a clue.

@tomysshadow
Copy link
Collaborator

Each byte is 8px wide? Is it a bitmask then? That's what first comes to mind.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

yeah i suppose, i haven't gotten that deep in binary stuff yet, so i barely know how that works haha

@tomysshadow
Copy link
Collaborator

Every byte has eight bits, it's likely that one bit represents one pixel. That'd make sense for 1bpp. That's super easy to implement, so I should add it in right away.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

149522341238426
There we go, with some help of bitstring!

Director itself can't export 1-bit images, quality stuff.

@Brian151
Copy link
Owner

The 1BPP most likely is 8 pixels to a byte, this would be the ideal way to implement 1BPP
IDK if it'd use compression or not. It technically could, but most likely only if a whole byte can be repeated

As for exporting 1BPP, I'm pretty sure anything lower than 8 BPP is fairly non-standard in graphics. About the only machines that natively support things like 4BPP and lower are older computers, and more specifically, game consoles, like the NES or SNES. I may be mistaken, though. Most PCs use 24 and 32-bit, and ofc, there's now... HDR. That said, there's no practical advantage, or at least, no perceived one, in Director exporting something lower than 8BPP. None of the 'standard' programs would be able to read it.

@Brian151
Copy link
Owner

on director installation:
30 GB disk capacity, usually less than 1 GB free
The CPU and RAM are most likely, totally insufficient. My desktop was upgraded, and Director still is a resource-intensive program to run, especially with larger files.
I don't feel safe risking that. As-is, Firefox uses on average, 70-80% of the CPU, and at least half of the RAM.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 19, 2017

To note, the CAST entries for the 1-bit images are different than the rest, the data ends after the regx/regy values, so it does not have that unknown byte and bit depth.

Ouch, that's bad.

@tomysshadow
Copy link
Collaborator

tomysshadow commented May 20, 2017

Well, if anything that at least gives us a clue that the unknown byte has something to do with something 1bpp images don't have.

I implemented 1bpp images but I'm having trouble pushing the commit, I'll try again later.

@Brian151
Copy link
Owner

It would make sense for 1BPP images to lack the palette entry, as they have no concept of color.
The lack of the bit depth is kinda odd, though

Well, good to know. And overall, I do not feel this makes the BITD's too much harder to deal with. Have 1BPP images been encountered in any color besides black? From my understanding, the typical process for colorizing a 1BPP would be the programmer specifies a color for it to display as when it's in use. Technically, it can have two colors: FG and BG, but the BG is usually the stage BG, or made to appear transparent. Not sure how Director handles this, I can't find any information about how Director deals with lower BPP besides it can import them.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 20, 2017

149527328804320
Well, the "foreColor" sprite attribute can be used to color it.

149527337780637

And with the "ink" attribute you can make it transparent.

@Brian151
Copy link
Owner

I'll have to look at that stuff later

@MrBrax
Copy link
Contributor Author

MrBrax commented May 20, 2017

149528365587614

damn, some bitmaps still come out as corrupted. changing it back to 0x101 fixes it by a tiny bit though

should i put my python script in the tools folder maybe?

@Brian151
Copy link
Owner

We'll figure it out soon I guess

You could?
How well is it commented/documented?

@MrBrax
Copy link
Contributor Author

MrBrax commented May 20, 2017

Not very well, but i think the code is pretty self-explanatory. More for sharing my interpretation of the file format than being a proper tool.

@tomysshadow
Copy link
Collaborator

tomysshadow commented May 21, 2017

If you look at ProjectorRays you can see I'm aiming for something a bit more advanced when it comes to reading in the file format. It might be of assistance though. A lot of it will eventually be pretty straightforward since stuff like sounds are just using pre-existing standards. I say go for it.

Oh, and I should mention that our tools are under the Apache License 2.0, so be sure to specify otherwise in the whitelist if you want.

@MrBrax
Copy link
Contributor Author

MrBrax commented May 21, 2017

i don't mind at all what happens to my stuff, as long as it's not proprietary and obfuscated by obscurity, like corporate stuff :)

@Brian151
Copy link
Owner

proprietary code and obfuscation has definitely done immense damage to the overall environment of computing. The only place I feel maybe code should be closed is in matters of security. But even then, there's openSSL, so...

When you think of it, proprietary code is what's led us where we are, now. Trying to pick-up the pieces of another failed example of corporate software.

I'll review your pull request later, have some rather bothersome things on my mind...

@MrBrax
Copy link
Contributor Author

MrBrax commented May 23, 2017

1495537159107

So i'm back at it again with this corrupting issue. There has to be a value somewhere that defines if it's linear/run length encoding, because it's either fixed or at least less corrupted if i force linear on it:

1495540667954

These follow a completely different pattern, throwing the "0x101" stuff off.
1495540704051

Scratch that, it's just palette values, no length descriptors

1495541003509

@tomysshadow
Copy link
Collaborator

MrBrax, please refer to chunks such as Lnam by their big-endian names, thanks

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 5, 2017

@tomysshadow i'm just following the styling of the other entries in that file, Lnam should be the correct name

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 6, 2017

149937014956201

Reworked the python script massively and it now outputs cast libraries. Got palette output, but haven't found how the cast selects it, so it's just referring to the palette cast in the code. Fun stuff!

@Brian151
Copy link
Owner

Brian151 commented Jul 7, 2017

https://github.com/Brian151/OpenShockwave/blob/master/docu/FormatNotes.txt#L238

Aside from some edge cases*, palettes are linked via their cast member ID in whatever cast they belong to
From what I gather, using the KEY* section, you can associate a given palette's CASt with its respective CLUT section.

you may also want to look at:
bitmapCastCompare.txt
bitmaps002.txt
both are in the examples directory

I might need to review the specificData thing, as information about palette linking might have gotten compromised by an edit, somewhere...

  1. Some palettes are not stored in the movie, namely, the system defaults
  2. I have had some issues when referencing a palette from another cast. It's possibly not intended, but it does seem Director is somehow storing that information (or trying)... However, I haven't figured-out where it's storing that information.

That said, we already have pretty much everything we need to link bitmap cast members to the palettes they use.

@MrBrax
Copy link
Contributor Author

MrBrax commented Jul 7, 2017

@Brian151

I have had some issues when referencing a palette from another cast

this is exactly what i'm trying to figure out, does it support that at all? because yes i know about the palette value and it seems to match the cast id, but if it can use palettes from other libraries then there's gotta be a lookup table somewhere, right? and what about the built in palettes? i can't really find the value where it's defined if used

@Brian151
Copy link
Owner

Brian151 commented Jul 7, 2017

It allows a palette from another cast to be specified...
No idea where it puts the information to use that palette.

As I said, Director crashed any time I tried editing a bitmap where a palette had been specified like that...

The most logical way (which I'm guessing Macromedia/Adobe didn't do...) would be:
Uint16 castID
Uint16 palMemberID

From all my observations, the only thing that changed was the 0XFFFF before the member ID
But, it only ever changed to 0x0000
Again, I don't have those examples right now... >.<

Perhaps in this case, the information is stored in the main part of the CASt?
IDK...

Anyways, I'm going to shower, then I'm off to bed
Only up now cuz I got caught-up browsing things

@akelagercrantz
Copy link

For anyone stumbling on this thread as I did, I made some updates to the script from @MrBrax and published it as a Gist (since this repo seems to have been deprecated for a set of other repos that no longer contain the script).

@MrBrax
Copy link
Contributor Author

MrBrax commented Nov 3, 2023

@akelagercrantz I worked on that script a bit more after I posted it, here's an even newer one: https://gist.github.com/MrBrax/75a23fe8d3b404df5a8041364d5774d8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants