[Epic] Improve the data migration process b/w prod and stage

[harshad16]

Is your feature request related to a problem? Please describe.

Moving data from stage to prod in an easier way.
Currently, we take the whole dump of the stage to prod.
The idea is to take only the required table to the prod and stop all the ingestion in the prod and make the prod DB read-only.

Additional context

• If users request a new package, where would we ingest the data?
  In the thoth-station/support, we have issues through which the user can request the package and we would ingest in the stage.
• How to make sure the schema is consistent with the suggested solution?
  Schema is updated on release date based on the requirement, management-API has a method to update the schema and is different from the data migration. It is done before the migration.
• Handling on the transitive dependent tables?
  To experiment

Acceptance criteria

• [Epic] Managing data across deployments #2513
• Managing data across deployments #2282

[harshad16]

/triage accepted
/priority important-soon
/sig devops
/sig knowledge-graph
/sig docs

[harshad16]

The stage is where data aggregation. every week update in the stage.
Which would be used to update prod.

Experiments:

• get the latest db dump to local system.
  aws s3 --endpoint https://s3.upshift.redhat.com/ cp s3://thoth/data/ocp4-stage/graph-backup/<latest_date> /tmp/aws-dump

can get the latest_date: aws s3 --endpoint https://s3.upshift.redhat.com/ ls s3://thoth/data/ocp4-stage/graph-backup/
NOTE: requires secrets and VPN.

• restore db with a single table
  reference: https://www.postgresql.org/docs/8.1/backup.html

[Gregory-Pereira]

/assign

[Gregory-Pereira]

• identify which tables are to be restored ("update")
  • solvers
  • reach out to thoth-team on this for clarification
  • identify the restoration order, in case any rows depend on other table's rows
  • take a dump of production which we will use to emulate a production environment
  • identify additional issues and document them

[codificat]

/remove-sig user-experience
/remove-sig stack-guidance

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[harshad16]

/remove-lifecycle stale
/lifecycle frozen