Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY` #3637

karolszklarski · 2024-01-03T11:38:00Z

Steps To Reproduce

Install bitwarden image and standalone mariadb database (a service, not a docker image).

bitwarden/self-host b11825cadef4 (old dev) works fine

bitwarden/self-host 74ccd545df9c (current dev) and e1a1833fcc06 (current beta) - admin module enters a crash loop.

gitHash cf4d8a4 is working, ea162d5 is not

Expected Result

Bitwarden should start normally.

Actual Result

Admin enters a crash loop

root@helios4:/home/bitwarden/prod# docker-compose logs
Attaching to bitwarden-prod
bitwarden-prod | Adding group `bitwarden' (GID 2009) ...
bitwarden-prod | Done.
bitwarden-prod | Adding user `bitwarden' ...
bitwarden-prod | Adding new user `bitwarden' (2009) with group `bitwarden' ...
bitwarden-prod | Not creating home directory `/home/bitwarden'.
bitwarden-prod | 2024-01-03 10:35:37,349 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,350 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,352 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,379 INFO RPC interface 'supervisor' initialized
bitwarden-prod | 2024-01-03 10:35:37,379 CRIT Server 'unix_http_server' running without any HTTP authentication checking
bitwarden-prod | 2024-01-03 10:35:37,380 INFO supervisord started with pid 1
bitwarden-prod | 2024-01-03 10:35:38,386 INFO spawned: 'identity' with pid 68
bitwarden-prod | 2024-01-03 10:35:38,450 INFO spawned: 'admin' with pid 69
bitwarden-prod | 2024-01-03 10:35:38,461 INFO spawned: 'api' with pid 70
bitwarden-prod | 2024-01-03 10:35:38,475 INFO spawned: 'icons' with pid 71
bitwarden-prod | 2024-01-03 10:35:38,489 INFO spawned: 'nginx' with pid 72
bitwarden-prod | 2024-01-03 10:35:38,505 INFO spawned: 'notifications' with pid 73
bitwarden-prod | 2024-01-03 10:35:54,206 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,208 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,208 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,209 INFO success: icons entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,209 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,210 INFO success: notifications entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:36:37,760 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:36:38,767 INFO spawned: 'admin' with pid 172
bitwarden-prod | 2024-01-03 10:36:54,568 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:37:32,493 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:37:33,509 INFO spawned: 'admin' with pid 216
bitwarden-prod | 2024-01-03 10:37:48,535 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:38:24,773 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:38:25,778 INFO spawned: 'admin' with pid 253
bitwarden-prod | 2024-01-03 10:38:40,800 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)

Admin logs

root@helios4:/home/bitwarden/prod# cat log/bitwarden/admin.log
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (17ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant');
      ALTER TABLE `Grant` DROP PRIMARY KEY;
Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEX `PRIMARY`; check that it exists
   at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43
   at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130
   at MySqlConnector.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 468
   at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56
   at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296
   at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable`1 migrationCommands, IRelationalConnection connection)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
   at Bit.MySqlMigrations.MySqlDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/MySqlMigrations/MySqlDbMigrator.cs:line 30
   at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9

Screenshots or Videos

No response

Additional Context

docker image tag bitwarden/self-host:beta bitwarden/self-host:prod

docker-compose.yml

version: "3.8"

services:
  bitwarden:
    container_name: bitwarden-prod
    env_file:
      - settings.env
    image: bitwarden/self-host:prod
    restart: always
    networks:
      - bitwarden-prod
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 20489:8443
    volumes:
      - /home/bitwarden/prod/etc:/etc/bitwarden
      - /home/bitwarden/prod/log/bitwarden:/var/log/bitwarden
      - /home/bitwarden/prod/log/nginx:/var/log/nginx

networks:
  bitwarden-prod:
    driver: bridge

settings.env

BW_DOMAIN=xxx.com
BW_ENABLE_SSL=true
globalSettings__baseServiceUri__vault=https://xxx.com:20489

globalSettings__disableUserRegistration=true

BW_DB_PROVIDER=mariadb
BW_DB_SERVER=host.docker.internal
BW_DB_DATABASE=bitwarden
BW_DB_USERNAME=bitwarden
BW_DB_PASSWORD=xxx

BW_INSTALLATION_ID=xxx
BW_INSTALLATION_KEY=xxx

adminSettings__admins="[email protected]"

globalSettings__mail__smtp__host=smtp.gmail.com
globalSettings__mail__smtp__port=587
globalSettings__mail__smtp__ssl=true
globalSettings__mail__smtp__username=xxx
globalSettings__mail__smtp__password=xxx

PUID=2009
PGID=2009

Githash Version

919d759-dirty

Environment Details

Linux helios4 6.1.63-current-mvebu Identity Server #1 SMP Mon Nov 20 10:52:19 UTC 2023 armv7l GNU/Linux
Marvell Armada 388 (88F6828) ARM Cortex-A9
ARMv7 32-bit
Dual Core 1.6 Ghz
2 GB RAM
Docker version 20.10.5+dfsg1, build 55c4c88

Database Image

mysql Ver 15.1 Distrib 10.5.21-MariaDB, for debian-linux-gnueabihf (armv7l) using EditLine wrapper

user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
#datadir        = /var/lib/mysql
datadir         = /home/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#bind-address           = 127.0.0.1
bind-address            = 0.0.0.0

Issue-Link

#2480

Issue Tracking Info

I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

The text was updated successfully, but these errors were encountered:

karolszklarski · 2024-01-03T12:07:46Z

gitHash cf4d8a4 is working, ea162d5 is not

That commit modifies GrantRepository, and logs say something about 'grant' table

withinfocus · 2024-01-04T18:40:24Z

@karolszklarski would you be able to upgrade your MariaDB instance to the latest and try again? We aren't testing with Maria and also can't replicate this on MySQL 8. The Pomelo library that's used to perform these migrations reported such an issue but it was resolved ~3 years ago and we are suspecting that this has something to do with Maria installations.

karolszklarski · 2024-01-05T10:02:58Z

I upgraded to the latest available release in debian "testing" repository
Ver 15.1 Distrib 10.11.6-MariaDB
but still get the same error.

Isn't it the same issue as #3635 ?

withinfocus · 2024-01-05T14:30:13Z

Yes, same issue most likely even though the error messages for the migration aren't included.

I see you're still on Maria 10.x -- would you be willing to try using 11.x, even though it's not LTS? Or MySQL 8?

JoeGTN1 · 2024-01-07T16:12:57Z

I have upgraded to Maria 11.0, same issue persists.

withinfocus · 2024-01-11T21:08:41Z

@karolszklarski can you pull the latest from main and give this a try? We made a modification to the database migration.

dk-armada · 2024-01-17T10:17:49Z

I'm running a MariaDB backend also. Seeing this exact issue with the latest Docker image bitwarden/self-host:beta.

etaggart65 · 2024-01-20T19:59:22Z

Same exact issue using 10.11.2-MariaDB backend. Had to revert to 2024.1.0-beta.

withinfocus · 2024-01-22T13:47:34Z

2024.1.0 had this issue present so you'd be experiencing it on that version, but the latest images have resolved this.

etaggart65 · 2024-01-22T14:16:09Z

That's not what I'm seeing. If I run bitwarden/self-host:beta or specifically 2024.1.1-beta I get the error.

2024/01/22 09:10:44 | stdout | 2024-01-22 09:10:44,894 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)

I can run 2024.1.0-beta without any issues.

withinfocus · 2024-01-22T15:28:03Z

That error may be due to something else? There are admin logs that would reveal more.

etaggart65 · 2024-01-22T16:10:59Z

Here is the entry from the admin log when running 2024.1.1-beta

fail: Microsoft.EntityFrameworkCore.Database.Command[20102] Failed executing DbCommand (9ms) [Parameters=[], CommandType='Text', CommandTimeout='30'] CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant'); ALTER TABLE GrantDROP PRIMARY KEY; Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEXPRIMARY; check that it exists at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43 at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130 at MySqlConnector.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 468
at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296 at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107 at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject) at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary2 parameterValues)
at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable1 migrationCommands, IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration) at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade) at Bit.MySqlMigrations.MySqlDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/MySqlMigrations/MySqlDbMigrator.cs:line 30 at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29 at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9

etaggart65 · 2024-01-22T16:12:11Z

My docker file..

`version: "3.8"

services:
bitwarden:
container_name: bitwarden
hostname: bitwarden
env_file:
- settings2.env
environment:
- PUID=1034
- PGID=100
- TZ=America/New_York
image: bitwarden/self-host:beta
restart: unless-stopped
network_mode: synobridge
ports:
- "9890:8080"
volumes:
- /volume1/docker/bitwarden/data:/etc/bitwarden
- /volume1/docker/bitwarden/logs:/var/log/bitwarden`

withinfocus · 2024-01-22T16:20:11Z

#3651 (comment) may be what you need.

etaggart65 · 2024-01-22T16:37:44Z

Ok thanks @withinfocus! that worked.

karolszklarski · 2024-01-22T21:50:34Z

I confirm, it works :)

stop bitwarden
login to database
execute ALTER TABLE Grant DROP COLUMN Id;
execute ALTER TABLE Grant ADD PRIMARY KEYPK_Grant (Key);
exit database, start bitwarden

karolszklarski added bug bw-unified-deploy An Issue related to Bitwarden unified deployment labels Jan 3, 2024

karolszklarski mentioned this issue Jan 3, 2024

Visiting admin page shows 502 Bad Gateway nginx #3635

Closed

1 task

withinfocus self-assigned this Jan 4, 2024

withinfocus removed their assignment Jan 8, 2024

This was referenced Jan 8, 2024

Primary key drop when data present PomeloFoundation/Pomelo.EntityFrameworkCore.MySql#1828

Closed

[PM-5519] [PM-5526] [PM-5624] [PM-5600] Tweak EF settings for MySQL grant auto-increment #3662

Merged

withinfocus closed this as completed Jan 22, 2024

withinfocus self-assigned this Jan 22, 2024

MichaelSch mentioned this issue Feb 29, 2024

Bitwarden Unified - Admin interface brings 502 bad gateway #3854

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY` #3637

Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY` #3637

karolszklarski commented Jan 3, 2024 •

edited

Loading

karolszklarski commented Jan 3, 2024

withinfocus commented Jan 4, 2024

karolszklarski commented Jan 5, 2024

withinfocus commented Jan 5, 2024

JoeGTN1 commented Jan 7, 2024

withinfocus commented Jan 11, 2024

dk-armada commented Jan 17, 2024

etaggart65 commented Jan 20, 2024 •

edited

Loading

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

karolszklarski commented Jan 22, 2024

Admin terminated by SIGABRT - Can't DROP INDEX PRIMARY #3637

Admin terminated by SIGABRT - Can't DROP INDEX PRIMARY #3637

Comments

karolszklarski commented Jan 3, 2024 • edited Loading

Steps To Reproduce

Expected Result

Actual Result

Screenshots or Videos

Additional Context

Githash Version

Environment Details

Database Image

Issue-Link

Issue Tracking Info

karolszklarski commented Jan 3, 2024

withinfocus commented Jan 4, 2024

karolszklarski commented Jan 5, 2024

withinfocus commented Jan 5, 2024

JoeGTN1 commented Jan 7, 2024

withinfocus commented Jan 11, 2024

dk-armada commented Jan 17, 2024

etaggart65 commented Jan 20, 2024 • edited Loading

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

withinfocus commented Jan 22, 2024

etaggart65 commented Jan 22, 2024

karolszklarski commented Jan 22, 2024

Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY` #3637

Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY` #3637

karolszklarski commented Jan 3, 2024 •

edited

Loading

etaggart65 commented Jan 20, 2024 •

edited

Loading