New Command 2055 - Process JSON Code

[jetrotal]

This command allows you to Get or Set values inside a JSON string.
(Requires Maniacs Patch)

The syntax always comes in pairs. The first parameter of each pair indicates whether you are using a direct value or a variable/indirect variable, through ValueOrVariable().

---

Syntax (TPC):

@raw 2055,
      "",          // Path to the target value within the JSON data
      0, 0,        // Operation (0-1) -  0: Get (extract value), 1: Set (update value)
      0, 5,        // Source string variable ID containing the JSON data
      0, 2,        // Target variable Type (0-2) - 0: Switch, 1: Variable, 2: StringVariable
      0, 1,        // Target variable ID to receive data (for Get) or contain new value (for Set)
      0, 100,      // Usage of string variable instead of the path string.

Get Example:

• Extract the value of "d" from the JSON data {"a":{"b":{"c":[{"d":"result"},"not_result"]}}} and store it in the string variable with ID 1.

@raw 2055,
      "a.b.c[0].d", // Path to the target value
      0, 0,         // Operation: Get
      0, 5,         // Source variable ID containing the JSON data
      0, 2,        // Target variable is a StringVar
      0, 1,         // Target variable ID to receive the extracted value
      0, 0          // Use literal path string

Set Example:

• Update the value of "d" in the JSON data stored in the string variable with ID 5, using the value from the string variable with ID 2.

@raw 2055,
      "a.b.c[0].d", // Path to the target value
      1, 0,         // Operation: Set
      0, 5,         // Source variable ID containing the JSON data that will be modified
      0, 2,        // Target variable is a StringVar
      0, 2,         // Target variable ID containing the new value for "d"
      0, 0          // Use literal path string

---

This command works nice until you deal with a json string with more than 2mb:

Every time I call it, the engine freezes for some frames.
That may happen due to how I keep parsing and deleting entire JSON objects every time I call it. Maybe there's a way to keep it stored in memory for some time, to deal with multiple calls in sequence.

Here's a JSON file I used for testing bigger strings:
JSONdatabase.txt

[florianessl]

Hm... neat idea. I was just thinking about how it would be nice to have a special storage type for user defined structs.
My current draft for ScopedVars rewrites the storage for Switches & Variables (possibly also strings - not tried yet) to share a common base class and the first thing that came to mind after this was done was: how about support for more complex data types? :D
Haven't thought about JSON, but this could work..

So, a possible solution for the freezes would be of course, to subclass this base storage for a new JSON type and use it to keep the data in memory. This would have the upside of already having access to all the common mechanisms (Get, Set, Range operations, Validation, Load&Save, special operations for "scoped" variables...) for the different data types without any extra implementation.
I'd have to share my current code of course. I think I can finally also create a PR for my little experiments...

[jetrotal]

@florianessl huh, thats cool!
My idea was ducktaped like this due to being easy to implement without creating issues in savegame.
my main needs was having a json file that I could import/export and a way to call objects by name.

Your idea seems to be more solid, though I don't have the knowledge to make it work.
Feel free to adopt it instead of my current solution, it's kinda messy as it is.

[Ghabry]

I like this idea. But I think this needs an extra argument to reason about the "data type".

So the code knows whether to read/write from an int variable or a string variable. Is currently a bit ambiguous (and sometimes you want that a number lands in a string directly).

---

You almost implemented JSON Path for the lookup. Was this intentional? https://goessner.net/articles/JsonPath/

JSON path is supported by certain libraries, so no need to implement this manually :).

I'm not sure if picojson supports this. Picojson actually only exists because it is a single header and is only used in emscripten. For any more serious work we can switch to any other JSON library. For tools we e.g. use nlohmann json.

---

For performance we could do some simple caching of a "json object" in memory in the String class to make this faster the first time this command is used.

[jetrotal]

@Ghabry right now my outputs are always string. There are ways on strvars to parse them as int later. But reducing the amount of steps as you suggested may be the better approach.

PicoJson was kinda rejected on the stuff I researched. I kept it because it was already used.

I'm leaning towards florianessi suggestions. Seems a proper evolution to what I proposed.

[Ghabry]

Okay then I will mark this as a draft first as it depends on work of @florianessl . Before we do the same work twice. ^^

[jetrotal]

OK @Ghabry, I guess this doesn't have PR dependencies, since I won't work directly with Florian's code by now.

---

I'm having a problem compiling this in some platforms.

Looks like some versions of easyRPG supports this by default:
#include <nlohmann/json.hpp>

Others (android, web, ubuntu, etc) can't find it.

How can I proceed?

[Ghabry]

This needs a detection in the build system because not all platforms have this library by default. I will add a detection for you in ~1 day.

[Ghabry]

FYI: Because this is an optional dependency to make this work after #3228 is merged you must edit your files (see e.g. filesystem_lzh.cpp/.h where the same logic is used):

Header:

LICENSE

#ifndef
#define

#include "system.h" <- NEW

#ifdef HAVE_NLOHMANN_JSON <- NEW

[...] all the code here

#endif <- NEW

#endif

Source:

LICENSE

#include "json_helper.h"

#ifdef HAVE_NLOHMANN_JSON <- NEW

[...] all the other includes and code here

#endif <- NEW

In game_interpreter.cpp you must guard the event command:

// Inside JsonCommand

#ifndef HAVE_NLOHMANN_JSON
Output::Warning("CommandProcessJson: JSON not supported on this platform");
return true;
#else
All the normal event code here
#endif

[jetrotal]

Rebased and no longer "Draft".

[Ghabry]

btw nlohmann json has a functionality called "json pointers" https://json.nlohmann.me/features/json_pointer/ which is like XPath for JSON.

The only downside I see with them is that I do not see a way to insert new elements with them :/. When I can get this working for insert it can replace your GetObjectAtPath and is even more flexible :)

[Ghabry]

json pointers really work for writing <3

auto j = json::parse(R"({
    "array": ["A", "B", "C"],
    "nested": {
        "one": 1,
        "two": 2,
        "three": [true, false]
    }
})");

j["/nested/four"_json_pointer] = 42;

j["/nested/five/six"_json_pointer] = 21;

j["/new_array/3"_json_pointer] = "3rd item";

std::cout << j.dump(4) << "\n";

gives

{
    "array": [
        "A",
        "B",
        "C"
    ],
    "nested": {
        "five": {
            "six": 21
        },
        "four": 42,
        "one": 1,
        "three": [
            true,
            false
        ],
        "two": 2
    },
    "new_array": [
        null,
        null,
        null,
        "3rd item"
    ]
}

so will replace your custom path parser with this one...

[jetrotal]

sure, I guess that would be faster than what I'm currently using.

btw, have you testet it with a huge file?
I uploaded an database clone here: https://github.com/EasyRPG/Player/files/15196282/JSONdatabase.txt

I've seen a lot of people in bad blood with nlohmann json library, due to its performance and compilation speed.

print from: https://github.com/jart/json.cpp

[Ghabry]

Does not really matter that nlohmann is slow. Is fast enough when reading when caching the parsed JSON file.

You can also make it faster by extracting a subtree from the JSON, so e.g. reading /LDB/Database/actors/Actor into another string, then modifying it by e.g. setting /name and more random stuff and then writing it back at the original location.

---

Implemented the JSON Pointer stuff. The command order is compatible with the one you defined:

@> 2055: /LDB/Database/actors/Actor/name, 0, 0, 0, 1, 0, 2, 0, 2

gives Zack

@> 2055: /LDB/Database/actors/Actor/title, 0, 1, 0, 1, 0, 2, 0, 2

writes to the JSON

---

Parsing the big JSON file you provided lags the game for a moment so is not feasible to always parse the JSON when doing anything with it.

I added caching to make this significantly faster:

The first time a string is accessed with the JSON command the parsed JSON is cached.

On subsequent reads you get the JSON from the cache. (FAST)

A write to the String (including setting JSON values) invalidates the cache. This means the JSON will be parsed again when accessing it the next time. (I think this can be also optimized a bit but fast reads are imo more important than fast writes xD).