Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Adding new projects fail due to an Django IntegrityError in v2025.12 #402

Open
tagnullde opened this issue Jan 30, 2025 · 9 comments
Open

[Bug] Adding new projects fail due to an Django IntegrityError in v2025.12 #402

tagnullde opened this issue Jan 30, 2025 · 9 comments

Comments

@tagnullde
Copy link

The newest version (2025.12) of Sysreptor has a potential bug, that caused us to revert the update to 2025.4.

Your current version is 2025.4
Checking if update is available...
Found newer version 2025.12

Image

Image

The error occurs when creating a new project or importing a report template. This is probably because the same API endpoint is used for both operations.

In both cases we get errors like these:

django.db.utils.IntegrityError: null value in column "usage_count" of relation "pentests_projecttype" violates not-null constraint
DETAIL:  Failing row contains (8b42b41b-<REDACTED>-ab6d1bc844ea, 2025-01-30 13:52:06.601232+00, 2025-01-30 13:52:06.605219+00, Freies_Schreiben, [{"id": "executive_summary", "label": "Executive Summary", "fiel..., [{"id": "title", "type": "string", "label": "Title", "origin": "..., de-DE, snapshot, null, \xc25950547<REDACTED>145532d47434d222c20226e6f6e63..., \xc25950547b22636<REDACTED>2d47434d222c20226e6f6e63..., \xc25950547<REDACTED>3a202241<REDACTED>c20226e6f6e63..., null, 4847da6c-<REDACTED>80c1a1dd0217, [{"field": "cvss", "order": "desc"}, {"field": "title", "order":..., finished, {}, {}, null).

Other operations like adding findings to an existing project works as expected.
@MWedl
Copy link
Contributor

MWedl commented Jan 30, 2025

Hi,
the database IntegrityError occur because of the downgrade. The problem is that database migrations cannot be automatically rolled back on downgrade leafing the DB in an inconsistent state. The column usage_count is present in the database (created by the new version) but the old version does not know about this column, which causes errors on insert.

Downgrading the database schema is quite tricky. Here are the steps:

  • Upgrade to the new version again (because the old version does not know about the new DB migrations)
  • Manually revert database migrations: 
    docker compose run api python3 manage.py migrate pentest 0059
docker compose run api python3 manage.py migrate users 0014
docker compose run api python3 mangae.py migrate tasks 0001
  • Downgrade to old version

What exactly is the bug why you downgraded in the first place?

@tagnullde
Copy link
Author

tagnullde commented Jan 30, 2025 •
edited
Loading

The bug I presented was the reason why we "downgraded". I should have made this more clear:
We didn't use the "downgrade" functionality of Sysreptor but rolled back a zfs snapshot. So, the error occured after the update to the newest version. After reverting the snapshot sysreptor worked fine.

Timeline:
Sysreptor 2025.4 works
Update to 2025.12 - errors
zfs rollback of snapshot to 2025.4 - systeptor works
Update again to 2025.12 - errors.

@MWedl
Copy link
Contributor

MWedl commented Jan 30, 2025

Ok. Thanks for the info.

I could not reproduce the database error locally. I found no (obvious) place in the source code where null is stored in usage_count. Could you provide the full stack trace where the database errors occur. Maybe then it's easier to debug and find the root cause.

@tagnullde
Copy link
Author

tagnullde commented Jan 31, 2025 •
edited
Loading

How and where do we can get the required stack-trace?
The only one I found is within the browser console, if this isn't helpful we need some advice what would.

TypeError: this.data is null
    useNotesCollab shareinfos.ts:89
    A pinia.mjs:1405
    clear shareinfos.ts:22
    A pinia.mjs:1405
    uqe auth.ts:44
    Vm runtime-core.esm-bundler.js:199
    Ma runtime-core.esm-bundler.js:206
    call runtime-core.esm-bundler.js:6139
    E reactivity.esm-bundler.js:1854
    Vm runtime-core.esm-bundler.js:199
    ZU runtime-core.esm-bundler.js:413
ETJRYF6a.js:14:2011
Will-change memory consumption is too high. Budget limit is the document surface area multiplied by 3 (1040384 px). Occurrences of will-change over the budget will be ignored. new
XHRPOST
http://sysreptor.internal.DOMAIN.TLD:8000/api/v1/pentestprojects/
[HTTP/1.1 500 Internal Server Error 2223ms]

dWe/window.fetch/<
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:300:16899
dWe/window.fetch
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:300:16837
_te<
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:32419
a
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:31222
s
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:32039
createProject
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:291:13850
A
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:30:2311
b
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/aiZ-JvFI.js:1:1072
K$/n[i]
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:20258
Vm
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1387
Ma
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1460
CJ
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:33169
u
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/BvGVkqee.js:1:409
Vm
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1387
Ma
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1460
n
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:8499
(Async: EventListener.handleEvent) Ys
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:7831
ree
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:8063
uee
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:8930
S
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:21664
T
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:21321
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20670
_e
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25401
run
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1741
Y
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25798
B
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24345
F
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24112
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20702
x
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:22300
S
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:21524
T
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:21321
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20670
_e
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25401
run
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1741
Y
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25798
B
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24345
F
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24112
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20702
_e
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25401
run
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1741
Y
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25798
B
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24345
F
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24112
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20702
_e
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25401
run
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1741
Y
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:25798
B
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24345
F
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24112
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20702
kJ
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:36974
process
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:35736
m
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:20749
_e
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:24908
run
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1741
runIfDirty
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:2051
Vm
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1395
ZU
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:3197
(Async: promise callback) WU
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:2498
PR
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:2455
Qm/o.scheduler
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:15:31820
Nj/u.scheduler<
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:17059
trigger
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:1993
xR
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:2292
notify
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:4951
trigger
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:4845
set value
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:10:14083
I
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:26:21479
Eae/v/<
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:26:20114
(Async: promise callback) v
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:26:19971
y
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:26:19229
o
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:26:15425
A
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:303:36732
Vm
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1387
Ma
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:14:1460
n
http://sysreptor.internal.DOMAIN.TLD:8000/static/_nuxt/ETJRYF6a.js:19:8499
Request error 
Object { error: FetchError, message: undefined }
 
<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>
ETJRYF6a.js:30:60947

Edit:

Here's the payload too. I figured it would be helpful to see what goes into the system.

body: '{"name":"Test","project_type":"4847da6c-bf4a-4ebb-a144-80c1a1dd0217","tags":[],"members":[{"id":"3c1e5a6c-adbc-47c1-b037-49dbf1531887","created":"2024-07-02T08:06:22.793125Z","updated":"2024-11-15T09:43:20.585454Z","last_login":"2025-01-31T08:04:57.932206Z","is_active":true,"username":"USER_A","name":"USER_A LASTNAME","title_before":null,"first_name":"USER_A","middle_name":null,"last_name":"LASTNAME","title_after":null,"email":"[email protected]","phone":null,"mobile":null,"scope":["admin","project_admin","template_editor","designer","user_manager"],"is_superuser":true,"is_project_admin":true,"is_designer":true,"is_template_editor":true,"is_user_manager":true,"is_guest":false,"is_system_user":false,"is_global_archiver":true,"is_mfa_enabled":false,"can_login_local":true,"can_login_sso":false,"roles":["pentester"]}]}'

@MWedl
Copy link
Contributor

MWedl commented Jan 31, 2025

You can get the stack trace of the SysReptor backend via cd deploy; docker compose logs app.

The payload looks fine. However, the JavaScript stacktrace is strange. It contains wrong line numbers and a reference to clear shareinfos.ts:22. The clear() function was removed some versions ago (way before v2025.4). Maybe try force reload and clearing browser caches to ensure the newest JavaScript files are loaded.

The caching problem, however, does not explain the IntegrityError. Could you please also check if the SysReptor update starts the new container versions (via docker ps looking at the image tags).

@tagnullde
Copy link
Author 

sysreptor-app  | 2025-01-31 08:05:16,850 [INFO] root: GET /api/v1/projecttypes/?ordering=status%2Cscope%2Cname%2C-created&scope=global&scope=private 200 (user=USER-X1)
sysreptor-app  | 2025-01-31 08:05:20,359 [ERROR] root: null value in column "usage_count" of relation "pentests_projecttype" violates not-null constraint
sysreptor-app  | DETAIL:  Failing row contains (9f79cb3b-ee74-4497-9e28-27030f67d4ca, 2025-01-31 08:05:20.34648+00, 2025-01-31 08:05:20.35494+00, DOMAIN.TLD_Freies_Schreiben, [{"id": "executive_summary", "label": "Executive Summary", "fiel..., [{"id": "title", "type": "string", "label": "Title", "origin": "..., de-DE, snapshot, null, \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., null, 4847da6c-bf4a-4ebb-a144-80c1a1dd0217, [{"field": "cvss", "order": "desc"}, {"field": "title", "order":..., finished, {}, {}, null).
sysreptor-app  | Traceback (most recent call last):
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
sysreptor-app  |     return self.cursor.execute(sql, params)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
sysreptor-app  |     raise ex.with_traceback(None)
sysreptor-app  | psycopg.errors.NotNullViolation: null value in column "usage_count" of relation "pentests_projecttype" violates not-null constraint
sysreptor-app  | DETAIL:  Failing row contains (9f79cb3b-ee74-4497-9e28-27030f67d4ca, 2025-01-31 08:05:20.34648+00, 2025-01-31 08:05:20.35494+00, DOMAIN.TLD_Freies_Schreiben, [{"id": "executive_summary", "label": "Executive Summary", "fiel..., [{"id": "title", "type": "string", "label": "Title", "origin": "..., de-DE, snapshot, null, \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., null, 4847da6c-bf4a-4ebb-a144-80c1a1dd0217, [{"field": "cvss", "order": "desc"}, {"field": "title", "order":..., finished, {}, {}, null).
sysreptor-app  | 
sysreptor-app  | The above exception was the direct cause of the following exception:
sysreptor-app  | 
sysreptor-app  | Traceback (most recent call last):
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/asgiref/sync.py", line 518, in thread_handler
sysreptor-app  |     raise exc_info[1]
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
sysreptor-app  |     response = await wrapped_callback(
sysreptor-app  |                ^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/adrf/viewsets.py", line 124, in async_view
sysreptor-app  |     return await self.dispatch(request, *args, **kwargs)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/adrf/views.py", line 73, in async_dispatch
sysreptor-app  |     response = self.handle_exception(exc)
sysreptor-app  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/rest_framework/views.py", line 469, in handle_exception
sysreptor-app  |     self.raise_uncaught_exception(exc)
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
sysreptor-app  |     raise exc
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/adrf/views.py", line 70, in async_dispatch
sysreptor-app  |     response = await sync_to_async(handler)(request, *args, **kwargs)
sysreptor-app  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/asgiref/sync.py", line 468, in __call__
sysreptor-app  |     ret = await asyncio.shield(exec_coro)
sysreptor-app  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/asgiref/current_thread_executor.py", line 40, in run
sysreptor-app  |     result = self.fn(*self.args, **self.kwargs)
sysreptor-app  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/asgiref/sync.py", line 522, in thread_handler
sysreptor-app  |     return func(*args, **kwargs)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/rest_framework/mixins.py", line 19, in create
sysreptor-app  |     self.perform_create(serializer)
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/rest_framework/mixins.py", line 24, in perform_create
sysreptor-app  |     serializer.save()
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py", line 208, in save
sysreptor-app  |     self.instance = self.create(validated_data)
sysreptor-app  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
sysreptor-app  |     return func(*args, **kwds)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
sysreptor-app  |     return func(*args, **kwds)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/app/api/reportcreator_api/pentests/serializers/project.py", line 375, in create
sysreptor-app  |     project_type = validated_data.pop('project_type').copy(linked_user=None, source=SourceEnum.SNAPSHOT)
sysreptor-app  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/app/api/reportcreator_api/pentests/models/project.py", line 253, in copy
sysreptor-app  |     return ProjectType.objects.copy(self, **kwargs)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
sysreptor-app  |     return func(*args, **kwds)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
sysreptor-app  |     return func(*args, **kwds)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/app/api/reportcreator_api/pentests/querysets.py", line 112, in copy
sysreptor-app  |     instance.save()
sysreptor-app  |   File "/app/api/reportcreator_api/pentests/models/project.py", line 250, in save
sysreptor-app  |     return super().save(*args, **kwargs)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/app/api/reportcreator_api/utils/models.py", line 24, in save
sysreptor-app  |     super().save(*args, **kwargs)
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 891, in save
sysreptor-app  |     self.save_base(
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 997, in save_base
sysreptor-app  |     updated = self._save_table(
sysreptor-app  |               ^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1160, in _save_table
sysreptor-app  |     results = self._do_insert(
sysreptor-app  |               ^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1201, in _do_insert
sysreptor-app  |     return manager._insert(
sysreptor-app  |            ^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
sysreptor-app  |     return getattr(self.get_queryset(), name)(*args, **kwargs)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/query.py", line 1847, in _insert
sysreptor-app  |     return query.get_compiler(using=using).execute_sql(returning_fields)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1836, in execute_sql
sysreptor-app  |     cursor.execute(sql, params)
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
sysreptor-app  |     return self._execute_with_wrappers(
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
sysreptor-app  |     return executor(sql, params, many, context)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
sysreptor-app  |     with self.db.wrap_database_errors:
sysreptor-app  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
sysreptor-app  |     raise dj_exc_value.with_traceback(traceback) from exc_value
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
sysreptor-app  |     return self.cursor.execute(sql, params)
sysreptor-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sysreptor-app  |   File "/usr/local/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
sysreptor-app  |     raise ex.with_traceback(None)
sysreptor-app  | django.db.utils.IntegrityError: null value in column "usage_count" of relation "pentests_projecttype" violates not-null constraint
sysreptor-app  | DETAIL:  Failing row contains (9f79cb3b-ee74-4497-9e28-27030f67d4ca, 2025-01-31 08:05:20.34648+00, 2025-01-31 08:05:20.35494+00, DOMAIN.TLD_Freies_Schreiben, [{"id": "executive_summary", "label": "Executive Summary", "fiel..., [{"id": "title", "type": "string", "label": "Title", "origin": "..., de-DE, snapshot, null, \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., \xc25950547b22636970686572223a20224145532d47434d222c20226e6f6e63..., null, 4847da6c-bf4a-4ebb-a144-80c1a1dd0217, [{"field": "cvss", "order": "desc"}, {"field": "title", "order":..., finished, {}, {}, null).
sysreptor-app  | 2025-01-31 08:05:22,514 [INFO] root: POST /api/v1/pentestprojects/ 500 (user=USER-X1)

root@sysreptor:/opt/sysreptor/deploy# docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                  PORTS                    NAMES
8ecd37f7344e   syslifters/sysreptor:latest   "/bin/bash /app/api/…"   19 hours ago   Up 19 hours (healthy)   0.0.0.0:8000->8000/tcp   sysreptor-app
6c578aaa2571   bitnami/redis:7.2             "/opt/bitnami/script…"   6 months ago   Up 20 hours             6379/tcp                 sysreptor-redis
a2b4ec57f938   sysreptor-languagetool        "bash start.sh"          6 months ago   Up 20 hours (healthy)   8010/tcp                 sysreptor-languagetool
0528b8feb4d4   postgres:14                   "docker-entrypoint.s…"   7 months ago   Up 20 hours (healthy)   5432/tcp                 sysreptor-db

@MWedl
Copy link
Contributor

MWedl commented Jan 31, 2025

Thank you. It seems that there is indeed an older version of the SysReptor docker image running. The line numbers from the django stacktrace do not fit the current version, but an older one released some months ago. I guess the update script downloaded the new SysReptor docker image and applied the database migrations, but then somehow did not start the new image. We will investigate why this is the case.

As a quick workaround you can force pull syslifters/sysreptor:latest from docker hub and restart the docker compose containers. Then the errors should be gone, since database schema and container versions are compatible again.

@tagnullde
Copy link
Author 

docker pull syslifters/sysreptor:latest
latest: Pulling from syslifters/sysreptor
Digest: sha256:70c396bfedb850b10329238afd6417db226f4d51fe2a68918290db7bcde192e1
Status: Downloaded newer image for syslifters/sysreptor:latest
docker.io/syslifters/sysreptor:latest

cd deploy/

docker compose up -d --force-recreate

[+] Running 3/3
 ✔ Container sysreptor-app    Started                                                                                                               385.5s 
 ✔ Container sysreptor-redis  Started                                                                                                               382.8s 
 ✔ Container sysreptor-db     Healthy                                                                                                               384.9s

That did it actually. Thanks for the help so far. We are now on 2025.12 and the error is gone.

@aronmolnar
Copy link
Contributor

aronmolnar commented Jan 31, 2025 •
edited
Loading

I think this error was caused by an error while bringing up the docker containers. It might also happen if the update script is cancelled using Ctrl+C while the containers are starting.

In this line, we try to bring up the containers. The containers start and migrations of the new SysReptor version are applied.
If the script is killed in this phase, the if-condition fails and the old SysReptor version is restored (in this line.

Recreating your docker containers might have worked because in the update script, we set the environment variable SYSREPTOR_VERSION. If this variable is unset, docker compose uses the latest tag.

I'm not quite sure, why your last output says Status: Downloaded newer image for syslifters/sysreptor:latest (because this must have been downloaded before, as the migrations were obviously applied).

Anyway, I suggest to remove the instruction to restore the previous version here. This would mean that if bringing up the docker containers fail, they can still be brought up manually (even if migrations were already applied).

pull bot pushed a commit to jakuta-tech/sysreptor that referenced this issue Feb 25, 2025
@MWedl
Merge branch 'Sysliftersgh-402-dont-catch-error-in-update-script' int…
550eaf9 
…o 'main'

Remove error_cleanup to not roll back version after migrations were applied

See merge request reportcreator/reportcreator!865
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aronmolnar @MWedl @tagnullde