Eliminate orphan objects #3732

nidhi-soni1104 · 2024-03-21T08:35:01Z

What this PR does / why we need it:

We need a rake task to go over all tables with the tenant_id field and make check whether that tenant still exists, reporting and optionally removing these on the way.

complete description is in JIRA

Which issue(s) this PR fixes

https://issues.redhat.com/browse/THREESCALE-10855

Note: I have commented the deleted line for now but added puts so whoever will run it while reviewing can see the puts

lib/tasks/database_cleanup.rake

nidhi-soni1104 · 2024-04-08T06:02:23Z

@akostadinov Updated

mayorova · 2024-04-10T13:55:11Z

lib/tasks/database_cleanup.rake

@@ -6,72 +6,27 @@ namespace :database do
    puts 'Checking and removing orphaned objects...'

    # Tables to exclude from orphaned objects check
-    excluded_tables = ['accounts', 'audits', 'categories', 'category_types', 'cms_templates', 'legal_term_acceptances', 'legal_term_bindings',
-                      'legal_term_versions', 'proxy_logs', 'schema_migrations', 'service_cubert_infos', 'slugs', 'taggings', 'tags']
+    excluded_tables = ['cms_templates']


Why exclude this?

I think this exclusion can be removed

I removed the exclusion, but I don't know if a coincidence or not - the task got stuck on Found orphaned objects in cms_templates: when I tried it in staging...

If it doesn't persist, let's think it's a coincidence. Why should cms_templates be special?

mayorova · 2024-04-10T14:22:36Z

lib/tasks/database_cleanup.rake

+
+      if orphaned_objects.exists?
+        puts "Found orphaned objects in #{model.table_name}:"
+        orphaned_objects.each { |obj| puts "- ID: #{obj.id}, Tenant ID: #{obj.tenant_id}" }


I'd use find_each here instead of each to use batching.

Addressed in ee9e0a8

jlledom · 2024-06-10T12:30:58Z

I rebased this, It was too old.

jlledom

I wrote a couple of comments but this LGTM. I haven't tried to run the script locally though.

It would be good to write a couple of tests. Do we have tests for rake tasks?

lib/tasks/database_cleanup.rake

jlledom · 2024-06-10T12:35:14Z

lib/tasks/database_cleanup.rake

@@ -6,72 +6,27 @@ namespace :database do
    puts 'Checking and removing orphaned objects...'

    # Tables to exclude from orphaned objects check
-    excluded_tables = ['accounts', 'audits', 'categories', 'category_types', 'cms_templates', 'legal_term_acceptances', 'legal_term_bindings',
-                      'legal_term_versions', 'proxy_logs', 'schema_migrations', 'service_cubert_infos', 'slugs', 'taggings', 'tags']
+    excluded_tables = ['cms_templates']


I think this exclusion can be removed

lib/tasks/database_cleanup.rake

github-actions · 2024-07-11T03:34:59Z

This PR is stale because it has not received activity for more than 30 days. Remove stale label or comment or this will be closed in 15 days.

github-actions · 2024-08-12T03:35:10Z

This PR is stale because it has not received activity for more than 30 days. Remove stale label or comment or this will be closed in 15 days.

mayorova · 2024-08-16T11:22:15Z

lib/tasks/multitenant/tenants.rake

+    task :cleanup_orphans, [:mode] => :environment do |_task, args|
+      puts 'Checking and removing orphaned objects...'
+
+      destroy = args[:mode] == "destroy"


So, the idea is that by default calling the task would only print the logs.
To actually destroy stuff, one would need to run

bundle exec rake "multitenant:tenants:cleanup_orphans[destroy]"

By the way, the logs are probably too verbose... Or maybe that's just the first time, as we have lots of orphans. Once the big batch is deleted, probably it will be fine.

mayorova · 2024-08-16T11:24:59Z

lib/tasks/multitenant/tenants.rake

+
+      puts "WARNING: the found orphan objects will be destroyed" if destroy
+
+      provider_account_ids = Account.where(provider: true).pluck(:id) + [Account.master.id]


I've added Master's ID here, to avoid deleting master's objects. Specifically, the master itself was being reported as an orphan 😅 (because it does not have provider: true)

Found orphaned objects in accounts: - ID: 1, Tenant ID: 1

And we probably don't want to destroy the master 🙃

You're too goodhearted..

But this could be hundreds of thousands of objects. Can't we filter this out in the query itself?

mayorova · 2024-08-16T11:25:53Z

lib/tasks/multitenant/tenants.rake

+
+          if destroy
+            puts "Destroying orphan #{model.table_name}..."
+            orphaned_objects.in_batches(of: 100).destroy_all


Added batched deletion, but still not sure if it still may have negative effects on the DB...

We might also need some more logging to see whether the operation was succeeded.

we need to revisit how we iterate over many objects

Added batched deletion, but still not sure if it still may have negative effects on the DB...

For sure batching is an improvement, what else could we do? We should be sure to run this after the billing jobs have finished and heads up ops.

Ideally we would use the new Sidekiq 7.3 iteration jobs. But if we just do in a rake task in the foreground, batches would be better.

If we want to run thins in produciton, we most likely need to use this iteration sidekiq job type. Because running haevy deletion in production may overburden the database. While recent hardware is much faster, I'm not sure how good this would be.

lib/tasks/multitenant/tenants.rake

akostadinov · 2024-09-17T16:40:34Z

lib/tasks/multitenant/tenants.rake

+
+      provider_account_ids = Account.where(provider: true).pluck(:id) + [Account.master.id]
+
+      ApplicationRecord.descendants.each do |model|


We can filter the classes further by filtering out children STI classes. Because if we run the query with the base STI class, running with the children will be redundant. SOmething along the lines of:

def base_models all_models = ApplicationRecord.descendants.select(&:arel_table).reject(&:abstract_class?) all_models.select! { _1.attribute_names.include? "tenant_id"} # we only want base STI classes, not the children all_models.select! do |model| base_class = model.base_class # either current model is the base_class or we can't find a base class amongst the discovered models /which would be very weird/ base_class == model || all_models.none? { |potential_parent| potential_parent == base_class } end end

Good idea, thanks!

akostadinov · 2024-10-03T15:51:03Z

lib/tasks/multitenant/tenants.rake

+      puts "Checking orphaned objects..."
+      puts "WARNING: the found orphan objects will be destroyed" if destroy


Suggested change

puts "Checking orphaned objects..."

puts "WARNING: the found orphan objects will be destroyed" if destroy

Rails.logger.warn "Checking orphaned objects..."

Rails.logger.warn "WARNING: the found orphan objects will be destroyed" if destroy

@akostadinov by default, in rake tasks Rails logger prints to the log file, and not to stdout.
We can force it by adding RAILS_LOG_TO_STDOUT=1 in front of bundle exec rake command, but then it prints all the debug logs, which I am not interested in... to fix it we'd need to also add log level in development environment, managed by an env var.

Currently, in most of rake tasks we use puts, I think it's fine.

hmm, in container it prints to stdout by default.. anyway there are multiple considerations.

If we want to run things automatially, better would be logging because it will keep track of what we did.

For this task I'm not sure.

If we want to run things once or multiple times only manually, then we need to use puts and warn discriminatively.

puts outputs to stdout. stdout is usually used to the information that needs to be passed down the execution pipeline. Informative messages usually go to stderr so that they don't ess up data that should go down a pipeline.

In this case warn seems more appropriate for these messages.

So you suggest to replace both with warn? what about the rest of puts ?

This specific tasks is meant to be run once. In fact, I am not quite sure we'll ever run it, because it's probably too much 😵

seems like all outputs in this PR are for stderr. wrt how many times we run, idk. We can always end-up with orphan objects if deletions fail for whatever reason.

akostadinov · 2024-10-03T16:31:59Z

lib/tasks/multitenant/tenants.rake

+      provider_account_ids = Account.where(provider: true).pluck(:id) + [Account.master.id]
+
+      base_models.each do |model|
+        orphaned_objects = model.where.not(tenant_id: provider_account_ids)


Suggested change

orphaned_objects = model.where.not(tenant_id: provider_account_ids)

orphaned_objects = model.where.not(tenant_id: Account.unscoped.providers_with_master.select(:id))

akostadinov · 2024-10-03T17:11:17Z

lib/tasks/multitenant/tenants.rake

+
+        if orphaned_objects.exists?
+          puts "Found #{orphaned_objects.size} orphaned objects for model #{model.name}:"
+          orphaned_objects.find_in_batches(batch_size: 100).with_index do |batch, index|


seems like find_each will simplify this

The idea (that I haven't finished pushing, it seems), is to add some "sleep" time between each batch, to space the deletion jobs.

The whole thing should happen in a different way. Most likely background deletion. Optimize this within a single rake task will probably be fragile in any case. If you want leave this alone until we figure out background deletion and then we may have a better method to handle this.

lib/tasks/multitenant/tenants.rake

Co-authored-by: Joan Lledó <[email protected]>

nidhi-soni1104 self-assigned this Mar 21, 2024

akostadinov reviewed Mar 21, 2024

View reviewed changes

lib/tasks/database_cleanup.rake Outdated Show resolved Hide resolved

akostadinov reviewed Mar 21, 2024

View reviewed changes

lib/tasks/database_cleanup.rake Outdated Show resolved Hide resolved

mayorova reviewed Apr 10, 2024

View reviewed changes

github-actions bot added the Stale label May 11, 2024

3scale deleted a comment from github-actions bot May 11, 2024

github-actions bot removed the Stale label May 12, 2024

jlledom force-pushed the THREESCALE-10855-eliminate-orphan-objects branch from 89a470f to 8e47c60 Compare June 10, 2024 12:30

jlledom reviewed Jun 10, 2024

View reviewed changes

github-actions bot added Stale and removed Stale labels Jul 11, 2024

github-actions bot added the Stale label Aug 12, 2024

mayorova force-pushed the THREESCALE-10855-eliminate-orphan-objects branch 2 times, most recently from 23b6efa to ee9e0a8 Compare August 16, 2024 11:17

mayorova reviewed Aug 16, 2024

View reviewed changes

github-actions bot removed the Stale label Aug 17, 2024

jlledom previously approved these changes Aug 20, 2024

View reviewed changes

lib/tasks/multitenant/tenants.rake Outdated Show resolved Hide resolved

lib/tasks/multitenant/tenants.rake Outdated Show resolved Hide resolved

mayorova dismissed jlledom’s stale review via 6af9bf9 September 16, 2024 09:25

mayorova force-pushed the THREESCALE-10855-eliminate-orphan-objects branch 2 times, most recently from 4d3c822 to e2c5276 Compare September 16, 2024 09:45

akostadinov reviewed Sep 17, 2024

View reviewed changes

mayorova force-pushed the THREESCALE-10855-eliminate-orphan-objects branch 2 times, most recently from eea080b to a9d31ad Compare September 30, 2024 14:08

akostadinov reviewed Oct 3, 2024

View reviewed changes

jlledom reviewed Oct 14, 2024

View reviewed changes

lib/tasks/multitenant/tenants.rake Outdated Show resolved Hide resolved

nidhi-soni1104 and others added 2 commits October 15, 2024 11:34

Add a task for removing objects with non-existing tenant

9d228f9

Co-authored-by: Joan Lledó <[email protected]>

Improvements for orphan objects removal

5a16d0a

mayorova force-pushed the THREESCALE-10855-eliminate-orphan-objects branch 2 times, most recently from f3f3411 to 442ed67 Compare October 15, 2024 10:12

Destroy asynchronously

682bb6e

mayorova force-pushed the THREESCALE-10855-eliminate-orphan-objects branch from 442ed67 to 682bb6e Compare October 15, 2024 10:13

github-actions bot added the Stale label Nov 15, 2024

3scale deleted a comment from github-actions bot Nov 15, 2024

akostadinov removed the Stale label Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate orphan objects #3732

Eliminate orphan objects #3732

nidhi-soni1104 commented Mar 21, 2024 •

edited

Loading

nidhi-soni1104 commented Apr 8, 2024

mayorova Apr 10, 2024

jlledom Jun 10, 2024

mayorova Aug 16, 2024

jlledom Aug 20, 2024

mayorova Apr 10, 2024

mayorova Aug 16, 2024

jlledom commented Jun 10, 2024 •

edited

Loading

jlledom left a comment •

edited

Loading

jlledom Jun 10, 2024

github-actions bot commented Jul 11, 2024

github-actions bot commented Aug 12, 2024

mayorova Aug 16, 2024 •

edited

Loading

mayorova Aug 16, 2024 •

edited

Loading

akostadinov Aug 17, 2024 •

edited

Loading

akostadinov Oct 3, 2024

mayorova Aug 16, 2024 •

edited

Loading

akostadinov Aug 17, 2024

jlledom Aug 20, 2024

akostadinov Sep 16, 2024

akostadinov Sep 17, 2024

akostadinov Sep 17, 2024 •

edited

Loading

mayorova Sep 25, 2024

akostadinov Oct 3, 2024

mayorova Oct 15, 2024

akostadinov Oct 15, 2024

mayorova Oct 15, 2024

mayorova Oct 15, 2024

akostadinov Oct 15, 2024

akostadinov Oct 3, 2024

akostadinov Oct 3, 2024

mayorova Oct 15, 2024

akostadinov Oct 15, 2024


		puts "WARNING: the found orphan objects will be destroyed" if destroy

		provider_account_ids = Account.where(provider: true).pluck(:id) + [Account.master.id]


		provider_account_ids = Account.where(provider: true).pluck(:id) + [Account.master.id]

		ApplicationRecord.descendants.each do \|model\|

		puts "Checking orphaned objects..."
		puts "WARNING: the found orphan objects will be destroyed" if destroy

	orphaned_objects = model.where.not(tenant_id: provider_account_ids)
	orphaned_objects = model.where.not(tenant_id: Account.unscoped.providers_with_master.select(:id))

Eliminate orphan objects #3732

Are you sure you want to change the base?

Eliminate orphan objects #3732

Conversation

nidhi-soni1104 commented Mar 21, 2024 • edited Loading

nidhi-soni1104 commented Apr 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlledom commented Jun 10, 2024 • edited Loading

jlledom left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 11, 2024

github-actions bot commented Aug 12, 2024

mayorova Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

mayorova Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

akostadinov Aug 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayorova Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akostadinov Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nidhi-soni1104 commented Mar 21, 2024 •

edited

Loading

jlledom commented Jun 10, 2024 •

edited

Loading

jlledom left a comment •

edited

Loading

mayorova Aug 16, 2024 •

edited

Loading

mayorova Aug 16, 2024 •

edited

Loading

akostadinov Aug 17, 2024 •

edited

Loading

mayorova Aug 16, 2024 •

edited

Loading

akostadinov Sep 17, 2024 •

edited

Loading