Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(PERFORMANCE): Update benchmark results #74

Merged
merged 1 commit into from
Apr 18, 2024
Merged

docs(PERFORMANCE): Update benchmark results #74

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 133 additions & 29 deletions docs/PERFORMANCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,74 +3,116 @@
For all benchmarks below, execute `ui5lint` without any parameters. Benchmarks created based on [Benchmarking UI5 Tooling](https://sap.github.io/ui5-tooling/stable/pages/Benchmarking/):

```sh
hyperfine --warmup 1 \
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ./bench.md
```

**TODO:** Memory benchmarking
## Small app: `openui5-sample-app`

_Reference commit: [`c9a0f7c51`](https://github.com/SAP/openui5-sample-app/tree/c9a0f7c51)_

**Metadata:**

* Resources to lint: `17` (31.59 KB)
* Latest Result: `0 problems (0 errors, 0 warnings)`

## Non-JS library, no findings: OpenUI5 `themelib_sap_horizon`
## Non-JS library with no findings: OpenUI5 `themelib_sap_horizon`

_Reference commit: [`0efb2cd`](https://github.com/SAP/openui5/tree/0efb2cd89a893b499f40c43c7534240524c5de42/src/themelib_sap_horizon) (tag `1.120.10`)_

**Metadata:**

* Resources to lint: `0`
* Result: `0 problems (0 errors, 0 warnings)`
* Latest Result: `0 problems (0 errors, 0 warnings)`

## Small app: `openui5-sample-app`
## Small library: OpenUI5 `sap.ui.testrecorder`

_Reference commit: [`fa6d40d`](https://github.com/SAP/openui5-sample-app/tree/fa6d40d2a3ef7c7bb7c416117a0efc675ec90c65)_
_Reference commit: [`0efb2cd`](https://github.com/SAP/openui5/tree/0efb2cd89a893b499f40c43c7534240524c5de42/src/sap.ui.testrecorder) (tag `1.120.10`)_

**Metadata:**

* Resources to lint: `16`
* Result: `6 problems (6 errors, 0 warnings)`
* Resources to lint: `68` (0.19 MB)
* Latest Result: `13 problems (13 errors, 0 warnings)`

## Medium library: OpenUI5 `sap.ui.layout`

_Reference commit: [`0efb2cd`](https://github.com/SAP/openui5/tree/0efb2cd89a893b499f40c43c7534240524c5de42/src/sap.ui.layout) (tag `1.120.10`)_

**Metadata:**

* Resources to lint: `497`
* Result: `1308 problems (1308 errors, 0 warnings)`
* Resources to lint: `572` (2.4 MB)
* Latest Result: `1140 problems (1140 errors, 0 warnings)`

## Large library: OpenUI5 `sap.m`

_Reference commit: [`0efb2cd`](https://github.com/SAP/openui5/tree/0efb2cd89a893b499f40c43c7534240524c5de42/src/sap.m) (tag `1.120.10`)_

**Metadata:**

* Resources to lint: `4914`
* Result: `11937 problems (11937 errors, 0 warnings)`
* This is around 10x more than sap.ui.layout
* Resources to lint: `5609` (25.67 MB)
* Latest Result: `11293 problems (11293 errors, 0 warnings)`


## Large library: OpenUI5 `sap.ui.core`

_Reference commit: [`0efb2cd`](https://github.com/SAP/openui5/tree/0efb2cd89a893b499f40c43c7534240524c5de42/src/sap.ui.core) (tag `1.120.10`)_

**Metadata:**

* Resources to lint: `4950` (45 MB)
* Latest Result: `4816 problems (4816 errors, 0 warnings)` + `2 fatal errors`

# Benchmark Runs

## April 17, 2024

* UI5 linter [`33d8cc5`](https://github.com/SAP/ui5-linter/commit/33d8cc5)
* Node.js `v20.11.1`
* MacBook Pro M1 Max

### themelib_sap_horizon

| Mean [ms] | Min [ms] | Max [ms] |
|---:|---:|---:|
| 595.7 ± 22.8 | 545.1 | 630.6 |

### openui5-sample-app

Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 2.242 ± 0.043 | 2.182 | 2.322 |

### sap.ui.testrecorder

| Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 3.605 ± 0.071 | 3.520 | 3.768 |

### sap.ui.layout

| Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 5.995 ± 0.045 | 5.934 | 6.086 |

### sap.m

| Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 38.769 ± 0.508 | 38.096 | 39.585 |

### sap.ui.core

| Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 53.864 ± 2.036 | 50.265 | 56.210 |

## March 11, 2024

* UI5 linter [`5466be5`](https://github.com/SAP/ui5-linter/commit/5466be5b983c4c6e6108c0d97d5221b8ad320a88)
* Node.js `v20.11.1`
* MacBook Pro M1 Max

```mermaid
---
config:
xyChart:
xAxis:
labelPadding: 20
yAxis:
titlePadding: 20
---
xychart-beta
title "March 11, 2024"
x-axis "Project (number of files)" ["themelib_sap_horizon (0)", "openui5-sample-app (16)", "sap.ui.layout (497)", "sap.m (4914)"]
y-axis "Runtime (in milliseconds)"
line [442.6, 2171, 6002, 34470]
```

### themelib_sap_horizon

| Mean [ms] | Min [ms] | Max [ms] |
Expand All @@ -79,6 +121,8 @@ xychart-beta

### openui5-sample-app

**Based on [`fa6d40d`](https://github.com/SAP/openui5-sample-app/tree/fa6d40d2a3ef7c7bb7c416117a0efc675ec90c65), which still contained linting errors**

Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
2.171 ± 0.072 | 2.076 | 2.315 |
Expand All @@ -94,3 +138,63 @@ Mean [s] | Min [s] | Max [s] |
| Mean [s] | Min [s] | Max [s] |
|---:|---:|---:|
| 34.470 ± 0.421 | 34.178 | 35.585 |


# Helper Scripts

```sh
# <Inside the openui5 repository root>
cd src/themelib_sap_horizon
ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ../../bench-themelib_sap_horizon.md

cd ../sap.ui.testrecorder
ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ../../bench-sap.ui.testrecorder.md

cd ../sap.ui.layout
ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ../../bench-sap.ui.layout.md

cd ../sap.m
ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ../../bench-sap.m.md

cd ../sap.ui.core
ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ../../bench-sap.ui.core.md
```

```sh
# <Inside the openui5-sample-app root>

ui5lint
hyperfine -i --warmup 1 \
'node $(which ui5lint)' \
--export-markdown ./bench.md
```

Helper script to determine the count and accumulated size of all files to be linted:

```js
// For a one-time evaluation, place this for example at the beginning of the lintWorkspace function
// *DO NOT USE THIS WHILE TAKING BENCHMARK MEASUREMENTS*
const allResources = await workspace.byGlob("**/{*.js,*.js.map,*.ts,*.html,manifest.json,manifest.appdescr_variant,*.view.xml,*.fragment.xml,ui5.yaml,*-ui5.yaml,*.ui5.yaml,ui5-*.yaml}")
const sizeOfAllFiles = await allResources.reduce(async (acc, res) => {
const currentLength = await acc
const str = await res.getBuffer();
return currentLength + str.length;
}, Promise.resolve(0))
console.log(`Total count of all files to be linted: ${allResources.length}`);
console.log(`Total size of all files: ${sizeOfAllFiles} bytes`);
````