Skip to content

Commit

Permalink
Benchmark (#9)
Browse files Browse the repository at this point in the history
* Add JMH benchmarking code to measure performance of encrypt and decrypt

* Update maven profile

* Reorder benchmark tests

* Add make task for running benchmark tests

* Update benchmark

* Add benchmark docs to README
  • Loading branch information
kschulst authored Apr 11, 2023
1 parent 6cc40bf commit f0b0803
Show file tree
Hide file tree
Showing 5 changed files with 242 additions and 43 deletions.
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ default: | help
build: ## Build the project and install to you local maven repo
mvn clean install

.PHONY: run-benchmark
run-benchmark: ## Run benchmark tests
mvn clean package exec:exec -Pbenchmark

.PHONY: release-dryrun
release-dryrun: ## Simulate a release in order to detect any issues
mvn release:prepare release:perform -Darguments="-Dmaven.deploy.skip=true" -DdryRun=true
Expand Down
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,33 @@ dependencies {

// TODO


## Benchmark tests

You can execute benchmark tests on your local machine by running `make run-benchmark`.

The following is the results of a benchmark test run on a MacBook Pro with M1 Max CPU and 64 GB memory.

```
Benchmark (content) Mode Cnt Score Error Units
-----------------------------------------------------------------------------------------
EncryptBenchmark.encryptBytes 6 chars thrpt 5 142288.875 ± 2194.094 ops/s
EncryptBenchmark.encryptBytes 2 chars thrpt 5 2835886.070 ± 42774.201 ops/s
EncryptBenchmark.encryptBytes sentence thrpt 5 33434.985 ± 371.490 ops/s
EncryptBenchmark.encryptBytes long-complex thrpt 5 1352.778 ± 33.741 ops/s
EncryptBenchmark.decryptBytes 6 chars thrpt 5 134362.809 ± 1088.432 ops/s
EncryptBenchmark.decryptBytes 2 chars thrpt 5 3664099.986 ± 10341.415 ops/s
EncryptBenchmark.decryptBytes sentence thrpt 5 31769.839 ± 113.963 ops/s
EncryptBenchmark.decryptBytes long-complex thrpt 5 1254.423 ± 31.235 ops/s
```

The main use case would be single word inputs, represented by the `6 chars` inputs. The `long-complex` string is
a >200 words text, with complex typography.

The error column denotes the _confidence interval_. A low error value indicates that results
are more precise and reliable, while a high error value suggests greater variability in the measurements.

## Known issues

// TODO: Describe issue about chunking that results in up to last 3 characters not being encrypted.
Expand Down
124 changes: 81 additions & 43 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
<format-preserving-encryption.version>1.0.0</format-preserving-encryption.version>
<guava.version>31.1-jre</guava.version>
<jackson.version>2.14.1</jackson.version>
<jmh.version>1.35</jmh.version>
<jsonassert.version>1.5.1</jsonassert.version>
<junit5.version>5.9.1</junit5.version>
<logback.version>1.4.5</logback.version>
Expand All @@ -37,7 +38,7 @@
<maven-checkstyle-plugin.version>3.1.2</maven-checkstyle-plugin.version>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
<maven-javadoc-plugin.version>3.3.0</maven-javadoc-plugin.version>
<maven-shade-plugin.version>3.2.4</maven-shade-plugin.version>
<maven-shade-plugin.version>3.4.1</maven-shade-plugin.version>
<maven-surefire-plugin.version>2.22.2</maven-surefire-plugin.version>
<maven-source-plugin.version>3.2.1</maven-source-plugin.version>
<jacoco-maven-plugin.version>0.8.7</jacoco-maven-plugin.version>
Expand Down Expand Up @@ -179,48 +180,6 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<relocations>
<relocation>
<pattern>com.idealista</pattern>
<shadedPattern>no.ssb.dapla.dlp.shaded.com.idealista</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>no.ssb.dapla.dlp.shaded.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>no.ssb.dapla.dlp.shaded.jackson</shadedPattern>
</relocation>
</relocations>
<shadedClassifierName>shaded</shadedClassifierName>
<shadedArtifactAttached>true</shadedArtifactAttached>
<artifactSet>
<includes>
<include>com.idealista:*</include>
<include>com.google.guava:*</include>
<include>com.fasterxml.jackson.core:*</include>
</includes>
</artifactSet>
<minimizeJar>false</minimizeJar>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
Expand Down Expand Up @@ -345,6 +304,85 @@
</snapshotRepository>
</distributionManagement>
</profile>
<profile>
<id>benchmark</id>
<dependencies>
<!-- JMH dependencies -->
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/benchmark/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${project.build.finalName}-with-dependencies</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>no.ssb.crypto.tink.fpe.benchmark.BenchmarkRunner</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<arguments>
<argument>-jar</argument>
<argument>${project.build.directory}/${project.build.finalName}-with-dependencies.jar</argument>
</arguments>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package no.ssb.crypto.tink.fpe.benchmark;

import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class BenchmarkRunner
{
public static void main(String[] args) throws Exception {
Options opt = new OptionsBuilder()
.include(EncryptBenchmark.class.getSimpleName())
.forks(1)
.build();
new Runner(opt).run();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
package no.ssb.crypto.tink.fpe.benchmark;

import com.google.crypto.tink.KeysetHandle;
import no.ssb.crypto.tink.fpe.Fpe;
import no.ssb.crypto.tink.fpe.FpeConfig;
import no.ssb.crypto.tink.fpe.FpeParams;
import no.ssb.crypto.tink.fpe.UnknownCharacterStrategy;
import no.ssb.crypto.tink.fpe.util.TinkUtil;
import org.openjdk.jmh.annotations.*;

import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.util.Map;

@State(Scope.Benchmark)
public class EncryptBenchmark {

static {
try {
FpeConfig.register();
}
catch (GeneralSecurityException e) {
throw new RuntimeException("Error initializing Tink FPE", e);
}
}

private final static String TINY = "2 chars";
private final static String SHORT = "6 chars";
private final static String MEDIUM = "sentence";
private final static String LONG = "long-complex";
private final static Map<String, String> ENCRYPT_PARAMS = Map.of(
TINY, "AB",
SHORT, "Foobar",
MEDIUM, "If I cøuld gather Åll the stars ænd håld them in my hænd...",
LONG, "CHAPTER 1. Loomings.\n" +
"\n" +
"Call me Ishmael. Some years ago—never mind how long precisely—having\n" +
"little or no money in my purse, and nothing particular to interest me\n" +
"on shore, I thought I would sail about a little and see the watery part\n" +
"of the world. It is a way I have of driving off the spleen and\n" +
"regulating the circulation. Whenever I find myself growing grim about\n" +
"the mouth; whenever it is a damp, drizzly November in my soul; whenever\n" +
"I find myself involuntarily pausing before coffin warehouses, and\n" +
"bringing up the rear of every funeral I meet; and especially whenever\n" +
"my hypos get such an upper hand of me, that it requires a strong moral\n" +
"principle to prevent me from deliberately stepping into the street, and\n" +
"methodically knocking people’s hats off—then, I account it high time to\n" +
"get to sea as soon as I can. This is my substitute for pistol and ball.\n" +
"With a philosophical flourish Cato throws himself upon his sword; I\n" +
"quietly take to the ship. There is nothing surprising in this. If they\n" +
"but knew it, almost all men in their degree, some time or other,\n" +
"cherish very nearly the same feelings towards the ocean with me."
);

private final static Map<String, String> DECRYPT_PARAMS = Map.of(
TINY, "AB",
SHORT, "6jZemW",
MEDIUM, "WT U 1øwyY rIbuCc Å82 o2G YHOt6 ævz råd7 7juD dk Ok mædK...",
LONG, "9jlDcz8 C. Q3tISqHV.\n" +
"\n" +
"G57B zx Di9LUBD. UFI2 iSXch doY—eBfWL qJtx ZDm 7dTJ PHPSfmy45—sF6Pjf\n" +
"lZHmGh 9b hG wHiNy Js 8G 0wNEQ, Ics ZZYLNog MqIuB9pGdt D8 LvV9NpkN fh\n" +
"Nv oR7uu, h 4NZTvwY 7 sAI0N UbVU 6gR3L P XnRx9l pIS 4ZN E3G LfDCQQ 4hPP\n" +
"rf jBT ezVVu. pH tq s tz1 H YqNS Bc WghESk1 tIJ 0BA ol9ask JQr\n" +
"BLLXeigOX4 1Yg 9cY6aotAvEu. cYZzKjPp E BilL iuH3lM CXnnIYe NPlr wgPSm\n" +
"ukS Oxyqx; MwCvypUd dB zJ j QJKY, CD5THSj WQ3kpZzl m0 dj bDsd; kjmKVR3K\n" +
"q FVR7 dQ1SFC 3Z9pts7hJRjME aswYrOq IoGTIf rFDjgR J5kUVWJOIR, l1j\n" +
"hbSpUgb4 ks uKh EzjV tp EaxQA zvWMyfp o PljS; NnJ swOE76mbmR 5q7tdWxY\n" +
"aC HgRyU xVC yIkw gH gqnIz YBA8 ii N6, JMYU vK ZTIiM6s1 f QTDUUL 2jWfV\n" +
"5ymzT5fJt Jp muIQMs0 GI lvpX Tc3mtVywBjMP pQPYr7S9 0ijn o5Y dnB6E1, Uo2\n" +
"la6PRofKXkqu 7LWnZm7m mdHEUC’p wIYG ghe—cLAo, z XuuuOKn Bz Qj69 JAM7 p8\n" +
"xoW wk 3pv YQ l55l 7s B sLd. husr lg Gu v1JilWizle IId QxQgJj 1n7 SPSt.\n" +
"A7Rd G NP0zvrUOKucMc dI3gP4jZ kbdz yMuS8X PzMWLVY X3Zi uF5 blNEE; U\n" +
"okyUfCY PG59 Tc via d1zz. 7l8h7 AF FP4aLzN 3csHeXgTC4 WX q73I. J0 qKGm\n" +
"hlu FWUr Ba, 7pe8Je CbN xW3 XW IavnU bnvYnx, BFrL kSzD Mn 3vN41,\n" +
"85tFSiB iHqq cqTERD pUE kSK9 87d3GvbO LN14wfe d1O n57mG FE4Z iL."
);

private final static String KEYSET_JSON_FF31_256_ALPHANUMERIC = "{\"primaryKeyId\":1720617146,\"key\":[{\"keyData\":{\"typeUrl\":\"type.googleapis.com/ssb.crypto.tink.FpeFfxKey\",\"value\":\"EiBoBeUFkoew7YJObcgcz1uOmzdhJFkPP7driAxAuS0UiRpCEAIaPkFCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFlaYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXowMTIzNDU2Nzg5\",\"keyMaterialType\":\"SYMMETRIC\"},\"status\":\"ENABLED\",\"keyId\":1720617146,\"outputPrefixType\":\"RAW\"}]}";

// Define sample plaintext inputs for both methods
@Param(value = {SHORT, TINY, MEDIUM, LONG})
public String paramName;

public byte[] plaintextBytes;

public byte[] ciphertextBytes;

private Fpe fpe;
private FpeParams fpeParams;

// Prepare the byte array from plaintextString
@Setup(Level.Trial)
public void prepare() throws Exception {
KeysetHandle keysetHandle = TinkUtil.readKeyset(KEYSET_JSON_FF31_256_ALPHANUMERIC);
fpe = keysetHandle.getPrimitive(Fpe.class);
fpeParams = FpeParams.with().unknownCharacterStrategy(UnknownCharacterStrategy.SKIP);

plaintextBytes = ENCRYPT_PARAMS.get(paramName).getBytes(StandardCharsets.UTF_8);
ciphertextBytes = DECRYPT_PARAMS.get(paramName).getBytes(StandardCharsets.UTF_8);
}

// Benchmark for encrypt(byte[] plaintext) method
@Benchmark
public byte[] encryptBytes() throws Exception {
return fpe.encrypt(plaintextBytes, fpeParams);
}

@Benchmark
public byte[] decryptBytes() throws Exception {
return fpe.decrypt(ciphertextBytes, fpeParams);
}

}

0 comments on commit f0b0803

Please sign in to comment.