Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dqlite/cowsql: flaky assertion errors on ppc64le #24

Open
ganto opened this issue Oct 15, 2023 · 5 comments
Open

dqlite/cowsql: flaky assertion errors on ppc64le #24

ganto opened this issue Oct 15, 2023 · 5 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@ganto
Copy link
Owner

ganto commented Oct 15, 2023

The test suite from dqlite and cowsql sometimes fail to correctly build on Fedora for the ppc64le architecture. A rebuild of the packages would usually "fix" the issue. E.g.:

cluster/restart                                             
  num_records=0                                             [ OK    ] [ 0.07839130 / 0.00605561 CPU ]
  num_records=1                                             [ OK    ] [ 0.08852291 / 0.00816774 CPU ]
  num_records=256                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=993                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=2200                                          [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)

membership/transferTwicePendingTransaction                  
  disk_mode=0                                               [ OK    ] [ 0.33617061 / 0.02036653 CPU ]
  disk_mode=1                                               [ ERROR ]
Error: test/integration/test_membership.c:335: assertion failed: rv_ == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)

Also see also canonical/dqlite#448 for another example.
@ganto ganto added bug Something isn't working help wanted Extra attention is needed labels Oct 15, 2023
@ganto
Copy link
Owner Author

ganto commented Oct 30, 2023

More failures:

cluster/modifyingQuery                                      
  num_records=0                                             [ OK    ] [ 1.20975552 / 0.01251383 CPU ]
  num_records=1                                             [ OK    ] [ 1.02317691 / 0.00807253 CPU ]
  num_records=256                                           [ OK    ] [ 1.70554498 / 0.18609776 CPU ]
  num_records=993                                           [ ERROR ]
Error: test/integration/test_cluster.c:233: assertion failed: rv_ == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=2200                                          [ OK    ] [ 4.17979201 / 0.88877483 CPU ]
cluster/hugeRow                                             [ OK    ] [ 0.80846385 / 0.43340652 CPU ]
cluster/dataOnNewNode                                       
  num_records=0                                             [ OK    ] [ 1.12412395 / 0.01304106 CPU ]
  num_records=1                                             [ OK    ] [ 1.04797664 / 0.00730391 CPU ]
  num_records=256                                           [ OK    ] [ 1.27305295 / 0.14829261 CPU ]
  num_records=993                                           [ OK    ] [ 2.35124113 / 1.24150868 CPU ]
  num_records=2200                                          [ OK    ] [ 1.88117602 / 0.59748403 CPU ]
cluster/restart                                             
  num_records=0                                             [ OK    ] [ 0.08935816 / 0.00556480 CPU ]
  num_records=1                                             [ OK    ] [ 0.08241669 / 0.00553597 CPU ]
  num_records=256                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=993                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=2200                                          [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
client/querySql                                             [ OK    ] [ 0.06301437 / 0.01260397 CPU ]
client/query                                                [ OK    ] [ 0.18037145 / 0.01696510 CPU ]
client/execSql                                              [ OK    ] [ 0.00078270 / 0.00076253 CPU ]
client/execWithOneParam                                     [ OK    ] [ 0.00123257 / 0.00117961 CPU ]
client/exec                                                 [ OK    ] [ 0.00080746 / 0.00078759 CPU ]
142 of 146 (97%) tests successful, 0 (0%) test skipped.

cluster/modifyingQuerySql                                   
  num_records=0                                             [ OK    ] [ 1.04761801 / 0.00642589 CPU ]
  num_records=1                                             [ OK    ] [ 1.12982367 / 0.00878118 CPU ]
  num_records=256                                           [ OK    ] [ 1.40291528 / 0.32246554 CPU ]
  num_records=993                                           [ ERROR ]
Error: test/integration/test_cluster.c:275: assertion failed: rv_ == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=2200                                          [ OK    ] [ 1.82140962 / 0.60401924 CPU ]
cluster/modifyingQuery                                      
  num_records=0                                             [ OK    ] [ 1.08134932 / 0.00940266 CPU ]
  num_records=1                                             [ OK    ] [ 1.04525732 / 0.00964778 CPU ]
  num_records=256                                           [ OK    ] [ 1.20541061 / 0.16225006 CPU ]
  num_records=993                                           [ OK    ] [ 2.51965598 / 1.28089786 CPU ]
  num_records=2200                                          [ OK    ] [ 4.21315465 / 0.91068976 CPU ]
cluster/hugeRow                                             [ OK    ] [ 1.09373537 / 0.65530618 CPU ]
cluster/dataOnNewNode                                       
  num_records=0                                             [ OK    ] [ 1.15127846 / 0.01281237 CPU ]
  num_records=1                                             [ OK    ] [ 1.16812780 / 0.01101628 CPU ]
  num_records=256                                           [ OK    ] [ 1.42087474 / 0.21431971 CPU ]
  num_records=993                                           [ OK    ] [ 2.70546761 / 1.23066388 CPU ]
  num_records=2200                                          [ OK    ] [ 2.82001127 / 0.82263246 CPU ]
cluster/restart                                             
  num_records=0                                             [ OK    ] [ 0.17286265 / 0.00663674 CPU ]
  num_records=1                                             [ OK    ] [ 0.10589352 / 0.00517449 CPU ]
  num_records=256                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=993                                           [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)
  num_records=2200                                          [ ERROR ]
Error: test/lib/server.c:113: assertion failed: rv == 0 (1 == 0)
Error: child killed by signal 6 (Aborted)

@ganto
Copy link
Owner Author

ganto commented Nov 19, 2023

Also Incus/LXD seem to be affected by regular build issues on ppc64le. E.g.:

=== RUN   TestHeartbeat
    heartbeat_test.go:88: create bootstrap node for test cluster
    testing.go:61: 
        	Error Trace:	/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/db/testing.go:122
        	            				/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/db/testing.go:61
        	            				/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/state/testing.go:22
        	            				/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/cluster/heartbeat_test.go:205
        	            				/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/cluster/heartbeat_test.go:89
        	            				/builddir/build/BUILD/lxd-5.18/_build/src/github.com/canonical/lxd/lxd/cluster/heartbeat_test.go:32
        	Error:      	Received unexpected error:
        	Test:       	TestHeartbeat
--- FAIL: TestHeartbeat (0.01s)

@ganto
Copy link
Owner Author

ganto commented Dec 22, 2023 

=== RUN   TestJoin
    testing.go:159: DEBUG: attempt 1: server @5db92: connected
    testing.go:61: 
        	Error Trace:	/builddir/build/BUILD/incus-0.4/_build/src/github.com/lxc/incus/internal/server/db/testing.go:122
        	            				/builddir/build/BUILD/incus-0.4/_build/src/github.com/lxc/incus/internal/server/db/testing.go:61
        	            				/builddir/build/BUILD/incus-0.4/_build/src/github.com/lxc/incus/internal/server/state/testing.go:22
        	            				/builddir/build/BUILD/incus-0.4/_build/src/github.com/lxc/incus/internal/server/cluster/membership_test.go:349
        	Error:      	Received unexpected error:
        	Test:       	TestJoin
--- FAIL: TestJoin (1.76s)

=== RUN   TestJoin
    testing.go:159: DEBUG: attempt 1: server @8b22c: connected
    membership_test.go:340: 
        	Error Trace:	/builddir/build/BUILD/incus-0.4/_build/src/github.com/lxc/incus/internal/server/cluster/membership_test.go:340
        	Error:      	Received unexpected error:
        	            	Failed to re-initialize gRPC SQL gateway: Failed to start dqlite server: 
        	Test:       	TestJoin
--- FAIL: TestJoin (0.68s)

@ganto
Copy link
Owner Author

ganto commented Apr 13, 2024 •
edited
Loading

While incus builds where fine recently cowsql is acting up again. This time cowsql-1.15.6 on x86_64 COPR builds:

[...]
========================================
   libcowsql 1.15.6: ./test-suite.log
========================================

# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: integration-test
======================
[...]
role_management/promote                                     
  role_management=1, target_voters=3, target_standbys=3     [ SKIP  ]
node/errMsg                                                 [ OK    ] [ 0.00936423 / 0.00889586 CPU ]
node/errMsgNodeNull                                         [ OK    ] [ 0.00001499 / 0.00001459 CPU ]
node/recoverExtNewFieldsNotZero                             [ OK    ] [ 0.00864626 / 0.00785400 CPU ]
node/recoverExtNewFields                                    [ OK    ] [ 0.00880934 / 0.00787987 CPU ]
node/recoverExtTooSmall                                     [ OK    ] [ 0.00865852 / 0.00787793 CPU ]
node/recoverExtUnaligned                                    [ OK    ] [ 0.00872139 / 0.00789399 CPU ]
node/recoverExt                                             [ OK    ] [ 0.00888109 / 0.00790890 CPU ]
node/recover                                                [ OK    ] [ 0.00878848 / 0.00793096 CPU ]
node/blockSizeRunning                                       [ OK    ] [ 0.00889276 / 0.00794085 CPU ]
node/blockSize                                              [ OK    ] [ 0.00895145 / 0.00892311 CPU ]
node/networkLatencyMsTooLarge                               [ OK    ] [ 0.00887066 / 0.00794323 CPU ]
node/networkLatencyMsTooSmall                               [ OK    ] [ 0.00879187 / 0.00794285 CPU ]
node/networkLatencyMsRunning                                [ OK    ] [ 0.00884490 / 0.00793981 CPU ]
node/networkLatencyMs                                       [ OK    ] [ 0.00885578 / 0.00793960 CPU ]
node/networkLatencyTooLarge                                 [ OK    ] [ 0.00880439 / 0.00795119 CPU ]
node/networkLatencyRunning                                  [ OK    ] [ 0.00877635 / 0.00794313 CPU ]
node/networkLatency                                         [ OK    ] [ 0.00882532 / 0.00790822 CPU ]
node/snapshotParamsThresholdLargerThanTrailing              [ OK    ] [ 0.00880151 / 0.00793981 CPU ]
node/snapshotParamsTrailingTooSmall                         [ OK    ] [ 0.00876777 / 0.00794411 CPU ]
node/snapshotParamsRunning                                  [ OK    ] [ 0.00884409 / 0.00793833 CPU ]
node/snapshotParams                                         [ OK    ] [ 0.00881310 / 0.00794737 CPU ]
node/startInet                                              [ OK    ] [ 0.00886101 / 0.00882379 CPU ]
node/start                                                  [ OK    ] [ 0.00879850 / 0.00792415 CPU ]
membership/transferTwicePendingTransaction                  [ OK    ] [ 0.00732651 / 0.01094210 CPU ]
membership/transferAndSqlExecWithBarrier                    [ FAIL  ]
membership/transferPendingTransaction                       [ OK    ] [ 0.00644413 / 0.01000433 CPU ]
membership/transfer                                         [ OK    ] [ 0.00684966 / 0.00905354 CPU ]
membership/join                                             [ OK    ] [ 0.00481469 / 0.00738347 CPU ]
[...]
144 of 145 (99%) tests successful, 1 (1%) test skipped.
FAIL integration-test (exit status: 1)

============================================================================
Testsuite summary for libcowsql 1.15.6
============================================================================

========================================
   libcowsql 1.15.6: ./test-suite.log
========================================

# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: integration-test
======================

Running test suite with seed 0x21b05244...
[...]
role_management/promote                                     
  role_management=1, target_voters=3, target_standbys=3     [ SKIP  ]
node/errMsg                                                 [ OK    ] [ 0.00917211 / 0.00889996 CPU ]
node/errMsgNodeNull                                         [ OK    ] [ 0.00001781 / 0.00001736 CPU ]
node/recoverExtNewFieldsNotZero                             [ OK    ] [ 0.01258177 / 0.01230517 CPU ]
node/recoverExtNewFields                                    [ OK    ] [ 0.00942395 / 0.00912374 CPU ]
node/recoverExtTooSmall                                     [ OK    ] [ 0.00871280 / 0.00790771 CPU ]
node/recoverExtUnaligned                                    [ OK    ] [ 0.01010653 / 0.00961686 CPU ]
node/recoverExt                                             [ OK    ] [ 0.00897441 / 0.00885376 CPU ]
node/recover                                                [ OK    ] [ 0.00891557 / 0.00887934 CPU ]
node/blockSizeRunning                                       [ OK    ] [ 0.00967689 / 0.00883380 CPU ]
node/blockSize                                              [ OK    ] [ 0.00924213 / 0.00882581 CPU ]
node/networkLatencyMsTooLarge                               [ OK    ] [ 0.00906353 / 0.00880603 CPU ]
node/networkLatencyMsTooSmall                               [ OK    ] [ 0.00916287 / 0.00881532 CPU ]
node/networkLatencyMsRunning                                [ OK    ] [ 0.00904845 / 0.00886644 CPU ]
node/networkLatencyMs                                       [ OK    ] [ 0.00893306 / 0.00793783 CPU ]
node/networkLatencyTooLarge                                 [ OK    ] [ 0.00923957 / 0.00890918 CPU ]
node/networkLatencyRunning                                  [ OK    ] [ 0.00918738 / 0.00893132 CPU ]
node/networkLatency                                         [ OK    ] [ 0.01017436 / 0.00990412 CPU ]
node/snapshotParamsThresholdLargerThanTrailing              [ OK    ] [ 0.00900184 / 0.00891053 CPU ]
node/snapshotParamsTrailingTooSmall                         [ OK    ] [ 0.00892890 / 0.00891314 CPU ]
node/snapshotParamsRunning                                  [ OK    ] [ 0.00901255 / 0.00891794 CPU ]
node/snapshotParams                                         [ OK    ] [ 0.00911258 / 0.00890955 CPU ]
node/startInet                                              [ OK    ] [ 0.00906797 / 0.00879312 CPU ]
node/start                                                  [ OK    ] [ 0.00901723 / 0.00890003 CPU ]
membership/transferTwicePendingTransaction                  [ OK    ] [ 0.00790516 / 0.01199870 CPU ]
membership/transferAndSqlExecWithBarrier                    [ ERROR ]
Error: test/integration/test_membership.c:271: assertion failed: rv == COWSQL_CLIENT_PROTO_ERROR (1 == 3)
Error: child killed by signal 6 (Aborted)
membership/transferPendingTransaction                       [ OK    ] [ 0.00660181 / 0.01092808 CPU ]
membership/transfer                                         [ OK    ] [ 0.00684693 / 0.00966645 CPU ]
membership/join                                             [ OK    ] [ 0.00774051 / 0.00959320 CPU ]
[...]
144 of 145 (99%) tests successful, 1 (1%) test skipped.
FAIL integration-test (exit status: 1)

============================================================================
Testsuite summary for libcowsql 1.15.6
============================================================================

========================================
   libcowsql 1.15.6: ./test-suite.log
========================================

# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: integration-test
======================

Running test suite with seed 0xd48baf2f...
[...]
role_management/promote                                     
  role_management=1, target_voters=3, target_standbys=3     [ SKIP  ]
node/errMsg                                                 [ OK    ] [ 0.00909122 / 0.00894209 CPU ]
node/errMsgNodeNull                                         [ OK    ] [ 0.00001195 / 0.00001160 CPU ]
node/recoverExtNewFieldsNotZero                             [ OK    ] [ 0.00862515 / 0.00790030 CPU ]
node/recoverExtNewFields                                    [ OK    ] [ 0.00872641 / 0.00788597 CPU ]
node/recoverExtTooSmall                                     [ OK    ] [ 0.00862186 / 0.00790223 CPU ]
node/recoverExtUnaligned                                    [ OK    ] [ 0.00859049 / 0.00792757 CPU ]
node/recoverExt                                             [ OK    ] [ 0.01002011 / 0.01016630 CPU ]
node/recover                                                [ OK    ] [ 0.00869786 / 0.00791231 CPU ]
node/blockSizeRunning                                       [ OK    ] [ 0.00877409 / 0.00795563 CPU ]
node/blockSize                                              [ OK    ] [ 0.00900041 / 0.00895192 CPU ]
node/networkLatencyMsTooLarge                               [ OK    ] [ 0.00879339 / 0.00794014 CPU ]
node/networkLatencyMsTooSmall                               [ OK    ] [ 0.00882906 / 0.00794991 CPU ]
node/networkLatencyMsRunning                                [ OK    ] [ 0.00881832 / 0.00794883 CPU ]
node/networkLatencyMs                                       [ OK    ] [ 0.00878390 / 0.00794220 CPU ]
node/networkLatencyTooLarge                                 [ OK    ] [ 0.00880768 / 0.00794139 CPU ]
node/networkLatencyRunning                                  [ OK    ] [ 0.00884084 / 0.00795243 CPU ]
node/networkLatency                                         [ OK    ] [ 0.00881039 / 0.00795014 CPU ]
node/snapshotParamsThresholdLargerThanTrailing              [ OK    ] [ 0.00885263 / 0.00789294 CPU ]
node/snapshotParamsTrailingTooSmall                         [ OK    ] [ 0.00883178 / 0.00795432 CPU ]
node/snapshotParamsRunning                                  [ OK    ] [ 0.00877172 / 0.00795308 CPU ]
node/snapshotParams                                         [ OK    ] [ 0.00878856 / 0.00795422 CPU ]
node/startInet                                              [ OK    ] [ 0.00879401 / 0.00785092 CPU ]
node/start                                                  [ OK    ] [ 0.00877830 / 0.00794167 CPU ]
membership/transferTwicePendingTransaction                  [ OK    ] [ 0.00597236 / 0.00961142 CPU ]
membership/transferAndSqlExecWithBarrier                    [ FAIL  ]
membership/transferPendingTransaction                       [ OK    ] [ 0.00698940 / 0.00979400 CPU ]
membership/transfer                                         [ OK    ] [ 0.00604055 / 0.00932722 CPU ]
membership/join                                             [ OK    ] [ 0.00489872 / 0.00787593 CPU ]
[...]
144 of 145 (99%) tests successful, 1 (1%) test skipped.
FAIL integration-test (exit status: 1)

============================================================================
Testsuite summary for libcowsql 1.15.6
============================================================================

After some re-tries these packages still could be built successfully.

@ganto
Copy link
Owner Author

ganto commented Nov 3, 2024

And yet another report of dqlite-1.17.1 failing on various architectures and Fedora releases:

========================================
   libdqlite 1.17.1: ./test-suite.log
========================================

# TOTAL: 7
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: integration-test
======================

Running test suite with seed 0x800f8e60...
[...]
node/start                                                  
  disk_mode=0, snapshot_compression=0                       [ OK    ] [ 0.00036797 / 0.00015527 CPU ]
  disk_mode=0, snapshot_compression=1                       [ OK    ] [ 0.00035821 / 0.00015792 CPU ]
  disk_mode=1, snapshot_compression=0                       [ OK    ] [ 0.00039219 / 0.00017983 CPU ]
  disk_mode=1, snapshot_compression=1                       [ OK    ] [ 0.00037543 / 0.00016988 CPU ]
membership/transferTwicePendingTransaction                  
  disk_mode=0                                               [ OK    ] [ 0.07930467 / 0.06283064 CPU ]
  disk_mode=1                                               [ OK    ] [ 0.08318027 / 0.07891331 CPU ]
membership/transferAndSqlExecWithBarrier                    [ FAIL  ]
membership/transferPendingTransaction                       
  disk_mode=0                                               [ OK    ] [ 0.08689213 / 0.07533570 CPU ]
  disk_mode=1                                               [ OK    ] [ 0.08092719 / 0.05507081 CPU ]
[...]
212 of 213 (100%) tests successful, 4 (2%) test skipped.
FAIL integration-test (exit status: 1)

============================================================================
Testsuite summary for libdqlite 1.17.1
============================================================================
# TOTAL: 7
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to https://github.com/canonical/dqlite
============================================================================

========================================
   libdqlite 1.17.1: ./test-suite.log
========================================

# TOTAL: 7
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: integration-test
======================

Running test suite with seed 0x7d2879f7...
[...]
node/start                                                  
  disk_mode=0, snapshot_compression=0                       [ OK    ] [ 0.00032226 / 0.00025776 CPU ]
  disk_mode=0, snapshot_compression=1                       [ OK    ] [ 0.00031610 / 0.00025525 CPU ]
  disk_mode=1, snapshot_compression=0                       [ OK    ] [ 0.00032808 / 0.00026992 CPU ]
  disk_mode=1, snapshot_compression=1                       [ OK    ] [ 0.00033459 / 0.00027129 CPU ]
membership/transferTwicePendingTransaction                  
  disk_mode=0                                               [ OK    ] [ 0.07159363 / 0.04743576 CPU ]
  disk_mode=1                                               [ OK    ] [ 0.06077344 / 0.04104518 CPU ]
membership/transferAndSqlExecWithBarrier                    [ FAIL  ]
membership/transferPendingTransaction                       
  disk_mode=0                                               [ OK    ] [ 0.06606221 / 0.04322567 CPU ]
  disk_mode=1                                               [ OK    ] [ 0.06171847 / 0.03712827 CPU ]
[...]
212 of 213 (100%) tests successful, 4 (2%) test skipped.
FAIL integration-test (exit status: 1)

============================================================================
Testsuite summary for libdqlite 1.17.1
============================================================================
# TOTAL: 7
# PASS:  6
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to https://github.com/canonical/dqlite
============================================================================

And the same errors also for:

  • Fedora 41 ppc64le
  • Fedora 41 x86_64
  • Fedora Rawhide ppc64le

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ganto