Original compression algorithm leaving parts uncompressed?

[grenoult]

Hi there,

Thanks for writing the specifications.
I'm playing with a personal project of mine: it's a Sim City 2000 file editor (demo: https://sc2k-save-editor.pages.dev/).
My goal is to be able to open, edit and save any sc2 file but I'm struggling at the saving part. It seems the compression algorithm doesn't compress some parts. Details are below:

Summary

• Something missing in the compression algorithm.
• Tested on FLARANCE.sc2 MISC chunk, appearing in June property tax

Details

I’m trying to uncompress and compress data for FLARANCE city, especially the MISC chunk.

I wrote a unit test to compress the uncompressed data and assert the result should be identical, but it’s failing.

• Tests results details

  ➜  sc2k-save-editor git:(master) ✗ npm test -- RleCompression.test.js -t
  
  > [email protected] test
  > jest RleCompression.test.js -t
  
   FAIL  tests/functional/RleCompression.test.js
    ● Test compression › test compression on MISC chunk (FLARANGE) 213
  
      expect(received).toEqual(expected) // deep equality
  
      - Expected  - 13
      + Received  +  9
  
      @@ -1363,13 +1363,13 @@
          1,
          235,
          208,
          132,
          0,
      -   3,
      +   1,
          1,
      -   234,
      +   129,
          234,
          132,
          0,
          3,
          1,
      @@ -3169,15 +3169,15 @@
          0,
          1,
          35,
          130,
          0,
      -   5,
      +   1,
          4,
      -   255,
      +   130,
          255,
      -   255,
      +   1,
          211,
          129,
          0,
          2,
          2,
      @@ -3248,28 +3248,24 @@
          2,
          250,
          10,
          130,
          0,
      -   5,
      +   1,
          1,
      -   255,
      -   255,
      -   255,
      +   131,
          255,
          129,
          0,
          2,
          3,
          140,
          130,
          0,
      -   5,
      +   1,
          1,
      -   255,
      -   255,
      -   255,
      +   131,
          255,
          138,
          0,
          1,
          1,
  
        25 |     it('test compression on MISC chunk (FLARANGE) 213', () => {
        26 |         const result = rleCompression.compress(uncompressed);
      > 27 |         expect(result).toEqual(compressed);
           |                        ^
        28 |     });
        29 | });
        30 |
  
        at Object.toEqual (tests/functional/RleCompression.test.js:27:24)
  
   PASS  tests/unit/RleCompression.test.js
  
  Test Suites: 1 failed, 1 passed, 2 total
  Tests:       1 failed, 15 passed, 16 total
  Snapshots:   0 total
  Time:        0.829 s, estimated 1 s
  Ran all test suites matching /RleCompression.test.js/i.
  

I only investigated the first case and I could find the following:

1. when compressing [1, 234 ,234]
2. the result is [1, 1, 129, 134]
3. but the expected result (in the original saved file) is [3, 1, 234, 234].

Which doesn’t match the specifications (and the consequent algorithm):

1. [1, 234 ,234] is a series of 2 tuples:
   1. [1] which would be compressed as [1, 1] (1 + n byte version)
   2. [234, 234] which would be compressed as [129, 234] (2 byte version)
2. however is it processed as one block only, instead of 2 tuples.
3. it seems like parts of the data was simply left uncompressed which puzzles me.

Any help would be appreciated, thanks!

[golimarrrr]

But does the game read it? Maybe it can read compressed or uncompressed

[dfloer]

From my testing, the MISC chunk is indeed expected to be compressed, perhaps you've got a bug in your implementation?

I may also have a bug in the spec, but I'm pretry sure the MISC chunk should be compressed.

[dfloer]

So I took another close look at this, and it maybe appears that there seems to be cases where single bytes may not be compressed. Compressing them does not appear to matter to the game though, as it'll happily decompress them.

I'll try to see if I can figure out what exactly is going on, and how the algorithm signals this.