r.2023.09.11.0

Summary:

This pr includes updates to the API spec and improvements to ingress flow for Azure and GCP.

What's Changed

• [COST-4104] Update API spec for new settings APIs by @samdoran in #4651
• [COST-4070] - clear gcp files for ingress flow by @lcouzens in #4671
• [COST-4188] - Allow split azure ingress files by @lcouzens in #4677

Full Changelog: r.2023.09.07.0...r.2023.09.11.0

r.2023.09.07.0

Summary:

• various data processing fixes
• use daily constructed daily archives for OCP
• return persistent volume claims created by namespace

What's Changed

• Dont concat frames if we have none by @lcouzens in #4663
• Fix name overwrites once and for all by @lcouzens in #4664
• update cluster_alias in OCPUsageReportPeriod if the source name changed by @maskarb in #4666
• [COST-3609] Return PVCs created by namespace by @myersCody in #4618
• [COST-3572] - actually use the constructed daily archives for OCP by @maskarb in #4667

Full Changelog: r.2023.08.29.0...r.2023.09.07.0

r.2023.08.29.0

Summary:

This release addresses some minor and major bugs in our recent daily archive pipeline improvements. This also includes some updated documentation references and improved tag settings apis.

What's Changed

• [COST-4038] Make sure API documentation references to sources is updated to integrations by @chambridge in #4648
• [COST-4118] - make all the workers attempt deletes before creating new files by @lcouzens in #4660
• [COST-4124] Improve /settings/tags/ API by @samdoran in #4657
• [COST-4133] Fix old ocp/cloud daily parquet file names by @lcouzens in #4658
• Azure date fix + delete try/except by @lcouzens in #4661
• fix azure date field checks by @lcouzens in #4662

Full Changelog: r.2023.08.25.0...r.2023.08.29.0

r.2023.08.25.0

Summary:

• various data processing fixes
• API documentation update
• add source-id for ROS

What's Changed

• [COST-3833] Remove link from API Documentation by @chambridge in #4647
• [COST-4122] fixing date range for aws/azure batch processing by @lcouzens in #4645
• [COST-4125] adding usecols to AWS downloader flow by @lcouzens in #4649
• [COST-4117] Skipping NaN values by @lcouzens in #4641
• [COST-4133] Fix file naming for batched files by @lcouzens in #4653
• [COST-4087] Add source_id correctly in ros data kafka event by @chambridge in #4650
• [COST-4087] Update ros_report_shipper.py by @lcouzens in #4656
• [COST-4118] clear parquet files for range prior to creating new ones by @lcouzens in #4655

Full Changelog: r.2023.08.22.0...r.2023.08.25.0

r.2023.08.22.0

Summary:

• various bug fixes
• updates for settings endpoint move
• fix OCP-on-GCP correlation when Node and PVC names are null.

What's Changed

• [COST-4111] Use source_type instead of provider_type in settings tag api by @myersCody in #4637
• [COST-4098] Add last processed date for cost model sources. by @myersCody in #4635
• [COST-4119] utilize the org-id and account-id saved in the source by @maskarb in #4638
• [COST-4014] Resolve account alias issues with various APIs by @chambridge in #4642
• Update Nise yaml default StorageClass by cloud type by @maskarb in #4640
• Fix timing issue with currency smoke test. by @myersCody in #4643
• Filter empty Node and PVC names for OCP-on-GCP correlation by @lcouzens in #4646

Full Changelog: r.2023.08.17.0...r.2023.08.22.0

r.2023.08.17.0

Summary:

• Update to HCS data filters and finalized object metadata
• Updates to subscription kafka messages
• Downloading reports now splits AWS/Azure files into daily archives
• OCP on GCP resource matching on empty string bugfix
• New tags API for settings are available
• Empty status source creation bugfix

What's Changed

• [COST-3892] Make subs event type lowercase by @cgoodfred in #4610
• [COST-4012] Don't check serviceinfo2 for Azure HCS filter by @cgoodfred in #4565
• [COST-3991] daily-archive-idea by @lcouzens in #4533
• Fix unleash check for subs by @cgoodfred in #4613
• chore(image): update base image by @github-actions in #4619
• [COST-3991] - Option 2 - Dont usecols by @lcouzens in #4621
• Ford OCP/GCP potentially matching on "" fix by @lcouzens in #4615
• [COST-4067] Include finalized-date on object metadata for HCS finalized files by @cgoodfred in #4612
• chore(image): update base image by @github-actions in #4624
• [COST-4099] - fix initial daily date by @lcouzens in #4625
• [COST-3585] Create new APIs for enabling and disabling tags by @samdoran in #4590
• [COST-4094] Add limit and offset to default paginated metadata response by @samdoran in #4617
• [COST-4103] - Batchy for Azure/AWS + usecols for AWS archives by @lcouzens in #4627
• [COST-4002] remove empty-status as requirement to create provider by @maskarb in #4575
• [COST-3585] Handle validation errors in custom filter syntax by @samdoran in #4628
• adjust required AWS columns by @lcouzens in #4629
• [COST-4097] Settings aws category keys use or for same filter. by @myersCody in #4623
• lets plan usecols better first by @lcouzens in #4636
• [COST-4108] Use pandas to parse date during azure downloader by @myersCody in #4632
• Update nise to fix azure local downloader. by @myersCody in #4634

Full Changelog: r.2023.08.03.0...r.2023.08.17.0

r.2023.08.03.0

Summary:

• Add SUBS Data Extraction and kafka messaging
• dependency updates and improvements

What's Changed

• [COST-3892] split Identity/TimeInterval into start and end to have aware timestamps for SUBS by @cgoodfred in #4609
• drop date override from OCP flow by @lcouzens in #4606
• Pin kombu so dependabot can update dependencies without breaking everything by @maskarb in #4608
• [COST-3753] Add an enabled tag limit by @myersCody and @samdoran in #4603
• [COST-3890/3892] Data Extraction and Kafka Messaging for SUBS by @cgoodfred and @djnakabaale in #4556
• bump oci, certifi, cryptography by @maskarb in #4601

Full Changelog: r.2023.07.27.0...r.2023.08.03.0

r.2023.07.27.0

Summary:

• Add workers for SUBS
• Tag model transition from 5 independent table to one unified table.

What's Changed

• Just for Eva - Adding crud provider tests to hot-fix-smoke-tests by @lcouzens in #4592
• [COST-3975] Create new model to hold all tags by @samdoran in #4594
• Deprecate pr_check.sh script by @Victoremepunto in #4596
• [COST-3891/COST-3893] Add workers for SUBS data extraction and transmission by @djnakabaale in #4597
• [COST-3996] Unleash flag to disable tag enablement on settings. by @myersCody in #4598
• [COST-3974] Switch data process to use new unified enabled tag table by @myersCody in #4576

Full Changelog: r.2023.07.24.0...r.2023.07.27.0

r.2023.07.24.0

Summary:

• Sources: add external-id to sources integration for AWS sources
• Handle daily operator payloads
• Adds write-access check to the user-access endpoints

What's Changed

• [COST-3990] Add write check to user_access endpoint by @myersCody in #4570
• [COST-3572] handle daily operator payloads by @maskarb in #4546
• [COST-3891/COST-3893] Add workers for SUBS data extraction and transmission by @djnakabaale in #4582
• [COST-4001] - handle OCI exception by @lcouzens in #4588
• [COST-4026] Update Grafana dashboard for XL queue by @samdoran in #4579
• [COST-4044] log schema_name rather than customer id by @maskarb in #4591
• [COST-3808] add external-id to sources integration by @maskarb in #4589
• Revert "[COST-3891/COST-3893] Add workers for SUBS data extraction a… by @maskarb in #4593

Full Changelog: r.2023.07.20.0...r.2023.07.24.0

r.2023.07.20.0

Summary:

• optimizing resource usage
• update settings endpoint in anticipation of new RBAC roles
• prevent summary when no bills exist for a cloud provider

What's Changed

• scale base resources below what we have set in stage by @lcouzens in #4569
• [COST-4002] Add check for provider in source status flow by @lcouzens in #4574
• fix unleash flag for XL customers by @lcouzens in #4577
• Remove deprecation for rbac setting by @myersCody in #4578
• Update settings if condition. by @myersCody in #4585
• [COST-3956] others included in ordering by cost & limit by @myersCody in #4532
• [COST-4026] Correct refresh XL queue name by @samdoran in #4581
• use inclause sqljinja filter in templates by @maskarb in #4573
• [COST-3896] bump nise version to include subs columns in test data by @djnakabaale in #4583
• [COST-3984] Do not update summary tables if there are no AWS bills by @samdoran in #4552
• Add settings to rbac python file. by @myersCody in #4586
• [COST-3984] - check Azure and GCP bills too by @maskarb in #4587

Full Changelog: r.2023.07.14.0...r.2023.07.20.0