Upload Flytekit runtime metrics to S3 #1683
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
This is part of Flytekit Metrics Exploration.
This PR uploads all the flytekit metrics to the remote storage(highlighted in green).
![image](https://private-user-images.githubusercontent.com/51814063/245658700-07333bba-64c6-448b-9154-8e532574a88c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxMTMxMTEsIm5iZiI6MTczOTExMjgxMSwicGF0aCI6Ii81MTgxNDA2My8yNDU2NTg3MDAtMDczMzNiYmEtNjRjNi00NDhiLTkxNTQtOGU1MzI1NzRhODhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA5VDE0NTMzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyODQ3OTBlYzQ1ZDMzNzMyYmY5NmNmY2E5NGZmZjIwN2Y3ZWE5NmFmY2E4M2RjNGM3NjQ5MmE3OTQ0MzY5M2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.8DutWflmDkF_Qwe5kxwmyieERXAKS0h1hpV-vdn7fd8)
Flytekit Metrics Exploration includes:
flytekit: Upload Flytekit runtime metrics to S3 #1683
flyteplugins: Add GetSpanPath and SpanExists flyteplugins#361
flyteidl: Add span uri flyteidl#415
flytepropeller: Add span uri to node event flytepropeller#577
flyteadmin: Implement GetFlyteKitMetrics endpoint flyteadmin#575
Checks
ran task in the sand box, the span file is under the same folder with deck.html.
![image](https://private-user-images.githubusercontent.com/51814063/244500421-ccd57781-cf5e-4e8d-8879-48405d6a2fce.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxMTMxMTEsIm5iZiI6MTczOTExMjgxMSwicGF0aCI6Ii81MTgxNDA2My8yNDQ1MDA0MjEtY2NkNTc3ODEtY2Y1ZS00ZThkLTg4NzktNDg0MDVkNmEyZmNlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA5VDE0NTMzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlYTkyMDQwZmNjYjUwMTQzZGVhYTQwOWIyYjdmN2RjNDIzNTJjNmM2Y2E3NWMwNDNlZjliN2NiNzYxZGU1NzYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.6JcNwFUb4nAqUWYelkDrk313gjP4Nnwo_cWsT3QN0pA)
![image](https://private-user-images.githubusercontent.com/51814063/244500399-104f2c37-d5a5-40b7-90a2-f77ba0e5915d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxMTMxMTEsIm5iZiI6MTczOTExMjgxMSwicGF0aCI6Ii81MTgxNDA2My8yNDQ1MDAzOTktMTA0ZjJjMzctZDVhNS00MGI3LTkwYTItZjc3YmEwZTU5MTVkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA5VDE0NTMzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzMTdlZGI0YjhhNWM0MDRhMDNjZDRlNDM1NDZlMDAyOTRhODZhYmJiMDIyMzU0MDdiM2VmYjc1NjdhNTI0ZTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.94Hp6FcobjsE4Op0p99VYrSwWLmzeFxzsZOo07Lim4U)