[Tracking] Outstanding ActiveRecord 5.2 tests

[alimi]

Below is a list of all the ActiveRecord tests that get run against PostgreSQL on Rails 5.2. Check off tests once they're passing against master.

There will be tests that don't work against CockroachDB. To skip these tests

1. Add a ruby file to test/excludes that matches the name of the test class if one doesn't already exist. For example to exclude a test from ActiveRecord::AdapterTest, create test/excludes/ActiveRecord/AdapterTest.rb.
2. Add an exclude statement to the file with the name of the test to exclude and a description. For example to exclude test_indexes from ActiveRecord::AdapterTest:
   

   activerecord-cockroachdb-adapter/test/excludes/ActiveRecord/AdapterTest.rb

   Line 1 in c3cb637

   exclude :test_indexes, "Rails transactional tests are being used while making schema changes. See https://www.cockroachlabs.com/docs/stable/online-schema-changes.html#limited-support-for-schema-changes-within-transactions."

3. Finally, if the test can run against CockroachDB with a few changes, add it to test/cases. Everything from the ActiveRecord test suite will be available, so a lot of the excluded test can be copied over. Namespace the test under the CockroachDB module to avoid name collisions. See test/cases/adapter_test.rb for an example.

For information on running tests, see the Contributing guide.

ActiveRecord tests

• test/cases/adapter_test.rb
• test/cases/adapters/postgresql/active_schema_test.rb
• test/cases/adapters/postgresql/array_test.rb
• test/cases/adapters/postgresql/bit_string_test.rb
• test/cases/adapters/postgresql/bytea_test.rb
• test/cases/adapters/postgresql/case_insensitive_test.rb
• test/cases/adapters/postgresql/change_schema_test.rb
• test/cases/adapters/postgresql/cidr_test.rb
• test/cases/adapters/postgresql/citext_test.rb
• test/cases/adapters/postgresql/collation_test.rb
• test/cases/adapters/postgresql/composite_test.rb
• test/cases/adapters/postgresql/connection_test.rb
• test/cases/adapters/postgresql/datatype_test.rb
• test/cases/adapters/postgresql/date_test.rb
• test/cases/adapters/postgresql/domain_test.rb
• test/cases/adapters/postgresql/enum_test.rb
• test/cases/adapters/postgresql/explain_test.rb
• test/cases/adapters/postgresql/extension_migration_test.rb
• test/cases/adapters/postgresql/foreign_table_test.rb
• test/cases/adapters/postgresql/full_text_test.rb
• test/cases/adapters/postgresql/geometric_test.rb
• test/cases/adapters/postgresql/hstore_test.rb
• test/cases/adapters/postgresql/infinity_test.rb
• test/cases/adapters/postgresql/integer_test.rb
• test/cases/adapters/postgresql/json_test.rb
• test/cases/adapters/postgresql/ltree_test.rb
• test/cases/adapters/postgresql/money_test.rb
• test/cases/adapters/postgresql/network_test.rb
• test/cases/adapters/postgresql/numbers_test.rb
• test/cases/adapters/postgresql/partitions_test.rb
• test/cases/adapters/postgresql/postgresql_adapter_test.rb
• test/cases/adapters/postgresql/prepared_statements_disabled_test.rb
• test/cases/adapters/postgresql/quoting_test.rb
• test/cases/adapters/postgresql/range_test.rb
• test/cases/adapters/postgresql/referential_integrity_test.rb
• test/cases/adapters/postgresql/rename_table_test.rb
• test/cases/adapters/postgresql/schema_authorization_test.rb
• test/cases/adapters/postgresql/schema_test.rb
• test/cases/adapters/postgresql/serial_test.rb
• test/cases/adapters/postgresql/statement_pool_test.rb
• test/cases/adapters/postgresql/timestamp_test.rb
• test/cases/adapters/postgresql/transaction_test.rb
• test/cases/adapters/postgresql/type_lookup_test.rb
• test/cases/adapters/postgresql/utils_test.rb
• test/cases/adapters/postgresql/uuid_test.rb
• test/cases/adapters/postgresql/xml_test.rb
• test/cases/aggregations_test.rb
• test/cases/ar_schema_test.rb
• test/cases/associations/cascaded_eager_loading_test.rb
• test/cases/associations/belongs_to_associations_test.rb
• test/cases/associations/bidirectional_destroy_dependencies_test.rb
• test/cases/associations/callbacks_test.rb
• test/cases/associations/eager_load_includes_ful l_sti_class_test.rb
• test/cases/associations/eager_load_nested_include_test.rb
• test/cases/associations/eager_singularization_test.rb
• test/cases/associations/eager_test.rb
• test/cases/associations/extension_test.rb
• test/cases/associations/has_and_belongs_to_many_associations_test.rb
• test/cases/associations/has_many_associations_test.rb
• test/cases/associations/has_many_through_associations_test.rb
• test/cases/associations/has_one_associations_test.rb
• test/cases/associations/has_one_through_associations_test.rb
• test/cases/associations/inner_join_association_test.rb
• test/cases/associations/inverse_associations_test.rb
• test/cases/associations/join_model_test.rb
• test/cases/associations/left_outer_join_association_test.rb
• test/cases/associations/nested_through_associations_test.rb
• test/cases/associations/required_test.rb
• test/cases/associations_test.rb
• test/cases/attribute_decorators_test.rb
• test/cases/attribute_methods/read_test.rb
• test/cases/attribute_methods_test.rb
• test/cases/attributes_test.rb
• test/cases/autosave_association_test.rb
• test/cases/base_test.rb (There might be problems running this test on macOS. ActiveRecord base test fixes #62)
• test/cases/batches_test.rb
• test/cases/binary_test.rb
• test/cases/bind_parameter_test.rb
• test/cases/boolean_test.rb
• test/cases/cache_key_ test.rb
• test/cases/calculations_test.rb
• test/cases/callbacks_test.rb
• test/cases/clone_test.rb
• test/cases/coders/json_test.rb
• test/cases/coders/yaml_column_test.rb
• test/cases/collection_cache_key_test.rb
• test/cases/column_alias_test.rb
• test/cases/column_definition_test.rb
• test/cases/comment_test.rb
• test/cases/connecti on_adapters/schema_cache_test.rb
• test/cases/connection_adapters/adapter_leasing_test.rb
• test/cases/connection_adapters/connection_handler_test.rb
• test/cases/connection_adapters/connection_specification_test.rb
• test/cases/connection_adapters/merge_and_resolve_default_url_config_test.rb
• test/cases/connection_adapters/mysql_type_lookup_test.rb
• test/cases/connection_adapters/type_lookup_test.rb
• test/cases/connection_management_test.rb
• test/cases/connection_pool_test.rb
• test/cases/connection_specification/resolver_test.rb
• test/cases/core_test.rb
• test/cases/counter_cache_test.rb
• test/cases/custom_locking_test.rb
• test/cases/database_statements_test.rb
• test/cases/date_test.rb
• test/cases/date_time_precision_test.rb
• test/cases/date_time_test.rb
• test/cases/defaults_test.rb
• test/cases/dirty_test.rb
• test/cases/disconnected_test.rb
• test/cases/dup_test.rb
• test/cases/enum_test.rb
• test/cases/errors_test.rb
• test/cases/explain_subscriber_test.rb
• test/cases/explain_test.rb
• test/cases/finder_respond_to_test.rb
• test/cases/finder_test.rb
• test/cases/fixture_set/file_test.rb
• test/cases/fixtures_test.rb (Ocassionaly fails locally. Curious to see how it does against CI.)
• test/cases/forbidden_attributes_protection_test.rb
• test/cases/habtm_destroy_order_test.rb
• test/cases/hot_compatibility_test.rb
• test/cases/i18n_test.rb
• test/cases/inheritance_test.rb
• test/cases/instrumentation_test.rb
• test/cases/integration_test.rb
• test/cases/invalid_connection_test.rb
• test/cases/invertible_migration_test.rb
• test/cases/json_attribute_test.rb
• test/cases/json_serialization_test.rb
• test/cases/locking_test.rb
• test/cases/log_subscriber_test.rb
• test/cases/migration/change_schema_test.rb
• test/cases/migration/change_table_test.rb
• test/cases/migration/column_attributes_test.rb
• test/cases/migration/column_positioning_test.rb
• test/cases/migration/columns_test.rb
• test/cases/migration/command_recorder_test.rb
• test/cases/migration/compatibility_test.rb
• test/cases/migration/create_join_table_test.rb
• test/cases/migration/foreign_key_test.rb
• test/cases/migration/index_test.rb
• test/cases/migration/logger_test.rb
• test/cases/migration/pending_migrations_test.rb
• test/cases/migration/references_foreign_key _test.rb
• test/cases/migration/references_index_test.rb
• test/cases/migration/references_statements_test.rb
• test/cases/migration/rename_tabl e_test.rb
• test/cases/migration_test.rb
• test/cases/migrator_test.rb
• test/cases/mixin_test.rb
• test/cases/modules_test.rb
• test/cases/multiparameter_attributes_test.rb
• test/cases/multiple_db_test.rb
• test/cases/nested_attributes_test.rb
• test/cases/nested_attributes_with_callbacks_test .rb
• test/cases/null_relation_test.rb
• test/cases/numeric_data_test.rb
• test/cases/persistence_test.rb
• test/cases/pooled_connections_test.rb
• test/cases/primary_keys_test.rb
• test/cases/query_cache_test.rb
• test/cases/quoting_test.rb
• test/cases/readonly_test.rb
• test/cases/reaper_test.rb
• test/cases/reflection_test.rb
• test/cases/relation/delegation_test.rb
• test/cases/relation/merging_test.rb
• test/cases/relation/mutation_test.rb
• test/cases/relation/or_test.rb
• test/cases/relation/predicate_builder_test.rb
• test/cases/relation/record_fetch_warning_test.rb
• test/cases/relation/where_chain_test.rb
• test/cases/relation/where_clause_test.rb
• test/cases/relation/where_test.rb
• test/cases/relation_test.rb
• test/cases/relations_test.rb
• test/cases/reload_models_test.rb
• test/cases/reserved_word_test.r b
• test/cases/result_test.rb
• test/cases/sanitize_test.rb
• test/cases/schema_dumper_test.rb
• test/cases/schema_loading_test.rb
• test/cases/scoping/default_scoping_test.rb
• test/cases/scoping/named_scoping_test.rb
• test/cases/scoping/relation_scoping_test.rb
• test/cases/secure_token_test.rb
• test/cases/serialization_test.rb
• test/cases/serialized_attribute_test.rb
• test/cases/statement_cache_test.rb
• test/cases/store_test.rb
• test/cases/suppressor_test.rb
• test/cases/tasks/database_tasks_test.rb
• test/cases/tasks/mysql_rake_test.rb
• test/cases/tasks/postgresql_rake_test.rb
• test/cases/tasks/sqlite_rake_test.rb
• test/cases/test_fixtures_test.rb
• test/cases/time_precision_test.rb
• test/cases/timestamp_test.rb
• test/cases/touch_later_test.rb
• test/cases/transaction_callbacks_test.rb
• test/cases/transaction_isolation_test.rb
• test/cases/transactions_test.rb
• test/cases/type/adapter_specific_registry_test.rb
• test/cases/type/date_time_test.rb
• test/cases/type/integer_test.rb
• test/cases/type/string_test.rb
• test/cases/type/type_map_test.rb
• test/cases/type/unsigned_integer_test.rb
• test/cases/type_test.rb
• test/cases/types_test.rb
• test/cases/unconnected_test.rb
• test/cases/unsafe_raw_sql_test.rb
• test/cases/validations/absence_validation_test.rb
• test/cases/validations/association_validation_test.rb
• test/cases/validations/i18n_generate_message_validation_test.rb
• test/cases/validations/i18n_validation_test.rb
• test/cases/validations/length_validation_test.rb
• test/cases/validations/presence_validation_test.rb
• test/cases/validations/uniqueness_validation_test.rb
• test/cases/validations_test.rb
• test/cases/view_test.rb
• test/cases/yaml_serialization_test.rb

[rafiss]

Thanks for compiling this! I did a quick scan of the tests here, and a few jumped out to me as things that are pretty likely not to work, based on what CockroachDB supports right now:

• test/cases/adapters/postgresql/active_schema_test.rb (sql: support user-defined schemas between database and table cockroach#26443)
• test/cases/adapters/postgresql/change_schema_test.rb (sql: support user-defined schemas between database and table cockroach#26443)
• test/cases/adapters/postgresql/composite_test.rb (sql: support user-defined composite/record types cockroach#27792)
• test/cases/adapters/postgresql/enum_test.rb (sql: support ENUM types cockroach#24873)
• test/cases/adapters/postgresql/geometric_test.rb (sql: add support for Postgres geometric types cockroach#21286)
• test/cases/adapters/postgresql/schema_authorization_test.rb (sql: support user-defined schemas between database and table cockroach#26443)
• test/cases/adapters/postgresql/schema_test.rb (sql: support user-defined schemas between database and table cockroach#26443)
• test/cases/adapters/postgresql/serial_test.rb (technically might pass with the sql.defaults.serial_normalization cluster setting but I think it's safe to skip)
• test/cases/adapters/postgresql/money_test.rb (sql: support money type cockroach#41578)

[alimi]

After a few PRs get merged, we'll be down to 60 test files to get through. Thanks to @jordanlewis's tip to skip load_schema after it's been run once, we can run most tests much faster.

It can take anywhere from 1 to 4 hours to run a test file, figure out why it's failing, and apply a fix or skip the tests. With 60 outstanding test files, we're looking at between 60 and 240 hours of work. The last 10 files have come in at an hour or less, and if that continues we’ll be closer to 60 hours.

To help speed this up a bit, people could quickly scan the tests like @rafiss did (#48 (comment)) and pick out anything that we know isn't going to work with CockroachDB. People are also welcome to jump in and investigate the failing tests.

[rafiss]

I took a rough stab at trying to prioritize the remaining test files. This is very unscientific, but I think it still could be a helpful heuristic to figure out which things to tackle next with our remaining time. I did my best, but it's still quite possible that some of the ones that I've marked as higher priority won't be able to work with CockroachDB, so as always, if you hit a stumbling block, let us know, so we can determine if the test should just be ignored.

High priority

• test/cases/adapters/postgresql/numbers_test.rb
• test/cases/adapters/postgresql/postgresql_adapter_test.rb
• test/cases/migration/change_schema_test.rb
• test/cases/migration/columns_test.rb
• test/cases/migration/create_join_table_test.rb
• test/cases/migration/foreign_key_test.rb
• test/cases/migration/references_foreign_key _test.rb
• test/cases/migration_test.rb
• test/cases/multiple_db_test.rb
• test/cases/persistence_test.rb
• test/cases/primary_keys_test.rb
• test/cases/relation/or_test.rb

Medium priority

• test/cases/adapters/postgresql/array_test.rb
• test/cases/adapters/postgresql/bit_string_test.rb
• test/cases/adapters/postgresql/bytea_test.rb
• test/cases/adapters/postgresql/case_insensitive_test.rb
• test/cases/adapters/postgresql/datatype_test.rb
• test/cases/adapters/postgresql/infinity_test.rb
• test/cases/adapters/postgresql/json_test.rb
• test/cases/adapters/postgresql/rename_table_test.rb
• test/cases/adapters/postgresql/type_lookup_test.rb
• test/cases/adapters/postgresql/uuid_test.rb
• test/cases/migration/compatibility_test.rb
• test/cases/migration/index_test.rb
• test/cases/migration/references_index_test.rb
• test/cases/migration/references_statements_test.rb
• test/cases/schema_dumper_test.rb

Low priority

• test/cases/adapters/postgresql/citext_test.rb
• test/cases/adapters/postgresql/collation_test.rb
• test/cases/adapters/postgresql/connection_test.rb
• test/cases/adapters/postgresql/explain_test.rb
• test/cases/adapters/postgresql/full_text_test.rb
• test/cases/adapters/postgresql/network_test.rb
• test/cases/adapters/postgresql/referential_integrity_test.rb
• test/cases/adapters/postgresql/transaction_test.rb (also note that cockroachdb doesn't support pg_cancel_backend() right now)
• test/cases/inheritance_test.rb
• test/cases/migration/column_attributes_test.rb
• test/cases/reload_models_test.rb
• test/cases/scoping/default_scoping_test.rb

May not work with cockroachdb

• test/cases/adapters/postgresql/domain_test.rb (sql: support user-defined DOMAIN types cockroach#27796)
• test/cases/adapters/postgresql/extension_migration_test.rb
• test/cases/adapters/postgresql/foreign_table_test.rb
• test/cases/adapters/postgresql/hstore_test.rb (sql: Support the postgres hstore type cockroach#41284)
• test/cases/adapters/postgresql/uuid_test.rb (see discussion here Uuid for unique ids in Rails 5.x fails due to direct inheritance with PostgreSQL adapter #12)
• test/cases/locking_test.rb

[marlabrizel]

I don't have edit permissions on the issue, but test/cases/primary_keys_test.rb passes for me against AR. I think this one may be safe to mark as completed unless I'm missing something!

[rafiss]

Sounds good! I was just basing this off of the boxes that are currently unchecked in Ali's issue description above.

[alimi]

I created individual issues for the outstanding high/medium/low priority test files. I included the failures/errors to make it easier to see what's wrong in the tests. It might make sense to re-prioritize based on the failures/errors.

You can find the issues here: https://github.com/cockroachdb/activerecord-cockroachdb-adapter/issues?q=is%3Aissue+is%3Aopen+%5BActiveRecord+5.2+Test+Failure%5D.

There are a few test files that are no longer failing, and I didn't create issues for them.

It looks like most of the "May not work with cockroachdb" tests don't work with CockroachDB. I didn't create issues for them either.