Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not required schema migration happening after using schema passed from batcher #192

Open
alok87 opened this issue Apr 9, 2021 · 2 comments
Labels
bug Something isn't working p2 intermittent issue, not very urgent, but need to be done soon

Comments

@alok87
Copy link
Contributor

alok87 commented Apr 9, 2021

I0409 19:14:56.032218       1 redshift.go:432] tOps: [ALTER TABLE "db"."table" ADD COLUMN  "plan_id" character varying(144)   ], cOps: [], vOps: []
I0409 19:14:56.032274       1 redshift.go:457] Strategy2: starting inplace-migration, table
I0409 19:14:56.294294       1 redshift.go:474] Running: ALTER TABLE "table"."col" ADD COLUMN  "plan_id" character varying(144)

Using the latest schema version is better than always doing migrations for each schema change. Assuming the latest schema is backward compatible makes things fast. This is for the first time sink case.

This issue was born after #189

@alok87 alok87 added the bug Something isn't working label Apr 9, 2021
@alok87
Copy link
Contributor Author

alok87 commented Apr 14, 2021

Will come back to this.

@alok87
Copy link
Contributor Author

alok87 commented Apr 14, 2021

This can be solved by using latest schemaID instead of schemaID from the job.

schemaId = job.SchemaId

https://github.com/practo/tipoca-stream/blob/master/redshiftsink/pkg/redshiftloader/load_processor.go#L659-L663

Using latest schemaID always to make the redshift table should solve this problem. Later.

@alok87 alok87 added the p2 intermittent issue, not very urgent, but need to be done soon label Apr 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p2 intermittent issue, not very urgent, but need to be done soon
Projects
None yet
Development

No branches or pull requests

1 participant