Initial public code release

.github/workflows/linter.yml

@@ -0,0 +1,25 @@
+name: Lint code
+
+on: [ push, pull_request ]
+
+jobs:
+  python-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: chartboost/ruff-action@v1
+        with:
+          src: "./webapp"
+      - uses: psf/black@stable
+        with:
+          src: "./webapp"
+
+  javascript-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v1
+        with:
+          node-version: '*'
+      - run: npm install standard
+      - run: npx standard ./webapp/static/assets

.gitignore

@@ -0,0 +1,12 @@
+input_pcaps/*
+suricata/output/filestore
+suricata/output/tcpstore
+suricata/output/udpstore
+suricata/output/*.json
+suricata/output/*.log
+webapp/database/*
+.env
+
+# Python
+__pycache__
+.ruff_cache

README.md

@@ -0,0 +1,100 @@
+# Shovel
+
+Shovel is a web application that offers a graphical user interface to explore
+[Suricata EVE outputs](https://docs.suricata.io/en/suricata-7.0.1/output/eve/eve-json-output.html).
+Its primary focus is to help [Capture-the-Flag players](https://ctftime.org/ctf-wtf)
+analyse network traffic dumps during stressful and time-limited attack-defense games such as
+[FAUSTCTF](https://faustctf.net/) or [ECSC](https://ecsc.eu/).
+Shovel is developed in the context of
+[ECSC Team France](https://ctftime.org/team/159269/) training.
+
+![Shovel during ENOWARS7](./.github/demo.webp)
+
+You might also want to have a look at these other awesome traffic analyser tools:
+
+  - https://github.com/secgroup/flower (first commit in 2018)
+  - https://github.com/eciavatta/caronte (first commit in 2020)
+  - https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
+
+Compared to these traffic analyser tools, Shovel relies on Suricata while making
+some opinionated choices for the frontend. This has a few nice implications:
+
+  - dissection of all application protocols already supported by Suricata (TCP and UDP),
+  - use a single SQLite database,
+  - on disk TCP/UDP/HTTP payload deduplication,
+  - filters based on libmagic, e.g. quickly filter flows containing PDF documents or PNG images,
+  - no heavy build tools needed, Shovel is easy to tweak.
+
+Moreover, Shovel is batteries-included with Grafana visualizations and some Suricata alert rules.
+
+## Setup
+
+### 0. Before the Capture-the-Flag event begins
+
+Copy `example.env` to `.env` and tweak the configuration parameters.
+Also add the flag format in `suricata/custom.rules` if needed.
+
+If you are playing a CTF using an IPv6 network, you might want to [enable IPv6 support in Docker deamon](https://docs.docker.com/config/daemon/ipv6/) before the CTF starts.
+
+### 1. Network capture setup
+
+You should place network captures in `input_pcaps/` folder.
+Capture files should be splitted into chunks to be progressively imported.
+If the CTF event does not already provide PCAP files, then you can adapt one
+of the following commands for a GNU/Linux system:
+```bash
+ssh [email protected] tcpdump -i wg-faustctf -n -w - 'tcp port not 22' | tcpdump -n -r - -G 30 -w input_pcaps/trace-%Y-%m-%d_%H-%M-%S.pcap
+```
+For a Microsoft Windows system, you may adapt the following command (3389 is RDP):
+```powershell
+.\tshark.exe -b duration:60 -w \\share\captures\trace -f "tcp port not 3389"
+```
+
+### 2. Launch Suricata and webapp via Docker (option A)
+
+Start Suricata, the web application and Grafana using `docker compose up -d --build`.
+
+Please note that restarting Suricata will cause all network capture files to be loaded again from zero.
+
+### 2. Launch Suricata and webapp traditionally (option B)
+
+You may launch Suricata then the web application using the following:
+```bash
+# Start Suricata
+export $(grep -vE "^(#.*|\s*)$" .env)
+./suricata/entrypoint.sh
+```
+
+```bash
+# Start web app
+export $(grep -vE "^(#.*|\s*)$" .env)
+(cd webapp && uvicorn --host 0.0.0.0 main:app)
+```
+
+Please note that restarting Suricata will cause all network capture files to be loaded again from zero.
+
+## Frequently Asked Questions
+
+### Is Suricata `flow_id` really unique?
+
+`flow_id` is derived from timestamp (ms scale) and current flow parameters (such
+as source and destination ports and addresses). See source code:
+<https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680>.
+
+### How do I reload rules without rebuilding the database?
+
+You can edit suricata rules in `suricata/custom.rules`, then reload the rules
+using:
+```bash
+kill -USR2 $(pidof suricata)
+```
+
+## Licensing
+
+Copyright (C) 2023  ANSSI
+
+Shovel is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
+
+Shovel is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with Shovel. If not, see <https://www.gnu.org/licenses/>.

docker-compose.yml

@@ -0,0 +1,32 @@
+version: "3"
+
+services:
+  suricata:
+    build: ./suricata
+    image: shovel-suricata
+    volumes:
+      - "./input_pcaps:/input_pcaps:ro"
+      - "./suricata/custom.rules:/suricata/custom.rules:ro"
+      - "./suricata/output:/suricata/output:rw"
+    env_file:
+      - .env
+
+  webapp:
+    build: ./webapp
+    image: shovel-webapp
+    volumes:
+      - "./input_pcaps:/webapp/static/input_pcaps:ro"
+      - "./suricata/output:/suricata/output:ro"
+      - "./webapp/database:/webapp/database:rw"
+    ports:
+      - 8000:8000
+    env_file:
+      - .env
+
+  grafana:
+    build: ./grafana
+    image: shovel-grafana
+    volumes:
+      - "./webapp/database:/webapp/database:ro"
+    ports:
+      - 3000:3000

example.env

@@ -0,0 +1,62 @@
+# Examples from FAUSTCTF (2023-09-23)
+CTF_START_DATE=2023-09-23T15:00+02:00
+CTF_TICK_LENGTH=180
+CTF_HOME_NET=[fd66:666::0/32]
+CTF_SERVICES=chatapp,image_galoisry,jokes,rsamail,buerographie_app,tic_tac_toe,office_supplies,auction_service
+CTF_SERVICE_AUCTION_SERVICE=[fd66:666:798::2]:12345,[fd66:666:798::2]:12346
+CTF_SERVICE_BUEROGRAPHIE_APP=[fd66:666:798::2]:13731
+CTF_SERVICE_CHATAPP=[fd66:666:798::2]:3000
+CTF_SERVICE_IMAGE_GALOISRY=[fd66:666:798::2]:5005
+CTF_SERVICE_JOKES=[fd66:666:798::2]:5000
+CTF_SERVICE_OFFICE_SUPPLIES=[fd66:666:798::2]:1337
+CTF_SERVICE_RSAMAIL=[fd66:666:798::2]:5555
+CTF_SERVICE_TIC_TAC_TOE=[fd66:666:798::2]:3333
+
+# Examples from ENOWARS7 (2023-07-22)
+#CTF_START_DATE=2023-07-22T15:00+02:00
+#CTF_TICK_LENGTH=60
+#CTF_HOME_NET=[10.1.42.1/25]
+#CTF_SERVICES=asocialnetwork,bollwerk,granulizer,oldschool,phreaking,steinsgate,yvm
+#CTF_SERVICE_ASOCIALNETWORK=10.1.42.1:3000
+#CTF_SERVICE_BOLLWERK=10.1.42.1:6009
+#CTF_SERVICE_GRANULIZER=10.1.42.1:2345
+#CTF_SERVICE_OLDSCHOOL=10.1.42.1:9080
+#CTF_SERVICE_PHREAKING=10.1.42.1:3399,10.1.42.1:6060,10.1.42.1:9930,10.1.42.1:6061,10.1.42.1:9931,10.1.42.1:6062,10.1.42.1:9932,10.1.42.1:6063,10.1.42.1:9933,10.1.42.1:6064,10.1.42.1:9934,10.1.42.1:6065,10.1.42.1:9935,10.1.42.1:6066,10.1.42.1:9936,10.1.42.1:6067,10.1.42.1:9937,10.1.42.1:6068,10.1.42.1:9938,10.1.42.1:6069,10.1.42.1:9939
+#CTF_SERVICE_STEINSGATE=10.1.42.1:4433,10.1.42.1:4420
+#CTF_SERVICE_YVM=10.1.42.1:3165
+
+# Examples from ICC 2023 training (2023-07-09)
+#CTF_START_DATE=2023-07-09T11:00+02:00
+#CTF_TICK_LENGTH=120
+#CTF_HOME_NET=[10.20.9.0/24]
+#CTF_SERVICES=flag_prescription,navashield,win_dc1,win_srv1
+#CTF_SERVICE_FLAG_PRESCRIPTION=10.20.9.6:5001
+#CTF_SERVICE_NAVASHIELD=10.20.9.6:8001,10.20.9.6:5000
+#CTF_SERVICE_WIN_DC1=10.20.9.4:80,10.20.9.4:445,10.20.9.4:135
+#CTF_SERVICE_WIN_SRV1=10.20.9.5:80,10.20.9.5:31337,10.20.9.5:135
+
+# Examples from ECSC 2022 (2022-09-15)
+#CTF_START_DATE=2023-01-30T13:00+02:00
+#CTF_TICK_LENGTH=120
+#CTF_HOME_NET=[10.20.9.0/24]
+#CTF_SERVICES=dewaste,cantina,hps,aquaeductus,blinkygram,winds_of_the_past,techbay
+#CTF_SERVICE_DEWASTE=10.10.10.1:10010
+#CTF_SERVICE_CANTINA=10.10.10.1:10020,10.10.10.1:10021,10.10.10.1:10024
+#CTF_SERVICE_HPS=10.10.10.1:10030
+#CTF_SERVICE_AQUAEDUCTUS=10.10.10.1:10041
+#CTF_SERVICE_BLINKYGRAM=10.10.10.1:10050
+#CTF_SERVICE_WINDS_OF_THE_PAST=10.10.10.1:10060
+#CTF_SERVICE_TECHBAY=10.10.10.1:10070
+
+# Examples from FAUST CTF 2022 (2022-07-09)
+#CTF_START_DATE=2023-01-30T13:00+02:00
+#CTF_TICK_LENGTH=180
+#CTF_HOME_NET=[fd66:666::0/32]
+#CTF_SERVICES=compiler60,docsnotebook,digital_seconds_ago,fittyfit,fluxmail,notes_from_the_future,admincrashboard
+#CTF_SERVICE_COMPILER60=[fd66:666:534::2]:6061,[fd66:666:534::2]:6062
+#CTF_SERVICE_DOCSNOTEBOOK=[fd66:666:534::2]:9000
+#CTF_SERVICE_DIGITAL_SECONDS_AGO=[fd66:666:534::2]:13731
+#CTF_SERVICE_FITTYFIT=[fd66:666:534::2]:5001
+#CTF_SERVICE_FLUXMAIL=[fd66:666:534::2]:4242
+#CTF_SERVICE_NOTES_FROM_THE_FUTURE=[fd66:666:534::2]:1338
+#CTF_SERVICE_ADMINCRASHBOARD=[fd66:666:534::2]:5000,[fd66:666:534::2]:5002

grafana/Dockerfile

@@ -0,0 +1,10 @@
+FROM grafana/grafana-oss:10.1.2
+ENV GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false
+ENV GF_ANALYTICS_CHECK_FOR_UPDATES=false
+ENV GF_ANALYTICS_REPORTING_ENABLED=false
+ENV GF_AUTH_ANONYMOUS_ENABLED=true
+ENV GF_AUTH_ANONYMOUS_HIDE_VERSION=true
+ENV GF_INSTALL_PLUGINS=frser-sqlite-datasource
+ENV GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
+COPY ./provisioning /etc/grafana/provisioning
+COPY ./dashboards /var/lib/grafana/dashboards