Measurement Method Is Not Accurate

[Alois-xx]

I was wondering why in your tests Revenj is so much better. Especially since I know Jil is near the optimum what can be done based on my measurements here:

• https://github.com/Alois-xx/SerializerTests
  I add Revenj 1.5.1 Nuget package and tried it vs Jil.

D:\Source\git\SerializerTests\bin\Release\net472>SerializerTests.exe -test combined -N 3000000 -serializer Jil,Revenj
Serializer      Objects "Time to serialize in s"        "Time to deserialize in s"      "Size in bytes" FileVersion     Framework       ProjectHome     DataFormat      FormatDetails   Supports Versioning
Revenj  3000000 2.8786  4.409   111777835       1.5.1.0 .NET Framework 4.7.3416.0       https://github.com/ngs-doo/revenj       Text    Json    Yes
Jil     3000000 0.6261  1.325   111777803       2.17.0.0        .NET Framework 4.7.3416.0       https://github.com/kevin-montrose/Jil   Text    Json    Yes

According to my tests JIL is at least 4 times faster. Then I did take a look at your test suite and found that you use factory delegates although at a first look everything points towards to the same factory

Func<int, SmallObjects.Message> factory1 = i => Models.Small.Message.Factory<SmallObjects.Message>(i);
Func<int, Models.Small.Message> factory1 = i => Models.Small.Message.Factory<Models.Small.Message>(i);

This factory for uses dynamic for some reason which looks strange but ok:

		public static T Factory<T>(int i) where T : new()
		{
			dynamic instance = new T();
			instance.message = "some message " + i;
			instance.version = i;
			return instance;
		}

After pulling out the heavy stuff like Intels VTune I was checking if your micro benchmark shows differences in Cache Level behavior or other exotic things. It turns out it is much simpler. You are creating entirely different objects which allocate a different amount of data because Models.Small.Message is the type which contains the factory which is used for JIL and other things but SmallObjects.Message is a different object defined in Revenj.Serialization.dll which contains generated code.
That is a little cheating here because you will win every startup measurement with pregenerated code which is not the most fair comparison. But anyway you are faster that is ok.

Now I did take the liberty to add to your test Revenj 1.5 from nuget and tested your serializer with the same data object and for serialize Jil is indeed nearly two times faster than RevenJ.

D:\>D:\Source\git\json-benchmark\Benchmark\bin\Release\JsonBenchmark.exe RevenjJsonMinimal Small Serialization 5000000
duration = 6660
size = 257777780
invalid deserialization = 0

D:\>D:\Source\git\json-benchmark\Benchmark\bin\Release\JsonBenchmark.exe Jil Small Serialization 5000000
duration = 3621
size = 257777780
invalid deserialization = 0

If I include Serialize and Deseralize then RevenJ is over two times slower if the same data object is used and not some pregenerated code in conjunction with some serializer which has no Nuget package is used.

D:\>D:\Source\git\json-benchmark\Benchmark\bin\Release\JsonBenchmark.exe Jil Small Both 5000000
duration = 6914
size = 257777780
invalid deserialization = 0

D:\>D:\Source\git\json-benchmark\Benchmark\bin\Release\JsonBenchmark.exe RevenjJsonMinimal Small Both 5000000
duration = 16297
size = 257777780
invalid deserialization = 0

I can fully support your claim on your main page https://github.com/ngs-doo/json-benchmark

• Almost everyone claims to be THE FASTEST

This also includes you.

Please update your test suite with a fair comparison of different serializers which leads to reproducible results. By the way Utf8Json is even faster also with your test suite.

[zapov]

The difference in Revenj speed comes from two main sources:

1. using the Revenj ChunkedMemoryStream
2. using the generated code

If you try to use Revenj with some other streams it won't be able to leverage various optimizations.
If you try to use Revenj on some non DSL class it will fallback to Json.NET

I wrote two separate classes for the same model as one is DSL managed and the other is what people usually write. Thus the custom factory which is able to create and initialize both instances.

This is really old benchmark and I did not evolve Revenj C# since than. I did the Java version which at the time worked like Revenj, but today is like any other serializer.

So all in all, your analysis is flawed and if you are really interested in it, you should redo it with the new information I explained here.

[Alois-xx]

Thanks for the additional information. I have checked further because this still does not explain why I see different amounts of CPU for the object factories.

Line # | Process (Name) | Stack | Weight (in view) (ms) | % Weight
-- | -- | -- | -- | --
20 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- Revenj.Serialization.ni.dll!Revenj.Serialization.JsonSerialization.Serialize(System.Object, System.IO.Stream, Boolean) | 1,358.497700 | 1.31
21 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Post::Factory 0x0 | 492.890800 | 0.47
22 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Complex::Factory 0x0 | 337.749600 | 0.32
23 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- mscorlib.ni.dll!System.IO.StreamWriter.Flush(Boolean, Boolean) | 334.720300 | 0.32
24 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Message::Factory 0x0 | 324.155600 | 0.31


Line # | Process (Name) | Stack | Weight (in view) (ms) | % Weight
-- | -- | -- | -- | --
20 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.LibrarySetup+<>c::<SetupJil>b__4_0 0x0 | 2,917.550300 | 2.80
21 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- Jil.ni.dll!Jil.JSON.Serialize[System.__Canon](System.__Canon, System.IO.TextWriter, Jil.Options) | 2,893.752900 | 2.78
22 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.LibrarySetup+<>c::<SetupJil>b__4_0 0x0<itself> | 17.009100 | 0.02
23 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- ?!? | 3.914800 | 0.00
24 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|    \|- Revenj.Utility.ni.dll!Revenj.Utility.ChunkedMemoryStream.GetWriter() | 2.873500 | 0.00
25 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Post::Factory 0x0 | 654.558300 | 0.63
26 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Message::Factory 0x0 | 397.872400 | 0.38
27 |   | \|    \|    \|    \|    \|    \|    \|    \|    \|- JsonBenchmark.exe!JsonBenchmark.Models.Small.Complex::Factory 0x0 | 377.970100 | 0.36

I have skipped serialization to measure the object creation overhead only and the numbers did normalize. This looks like a CPU cache artefact. I have also moved object creation out of the measure loop and created the objects beforehand and then randomize array access to get random memory access. The numbers did change a bit but not much. It looks like you are 60% up to two times faster than JIl with your pregenerated code. The only pitty is that your serializer will not create the highly efficient code for random objects and you never did implement that.

I cannot reproduce your published numbers with your AMD CPU. I have an i7-4770K CPU @ 3.50GHz where the differences are not so big. So yes good work but in the meantime we have with Utf8Json a serializer which is for small objects even faster and works with pretty much any object.

[zapov]

The latest bench was done on Windows/Intel with various updates to the libraries.
AMD was done before that. I would not be surprised there are various Spectre/Meltdown side-effects which affect micro benchmarks like this.

Java DSL version is much closer to the real limit. Not sure how fast Utf8Json is, but if it's noticeably slower than that there is still a lot of room for improvements.
After you get rid of allocations (Revenj currently) and start working on byte level (Utf8Json and Java DSL) the only thing which remains is how good are your type converters. And there is often room for improvement there (especially if you are calling into standard libraries).

I was rather surprised when I originally wrote this how most libraries fail to work with anything non-trivial. And people blamed me/this benchmark that their favorite library is buggy instead of accepting that maybe, just maybe they are drinking some kool-aid and jumping to conclusions.