0.21.2 (2020-08-18)

• Increase max batch_size_rows to 1000k from 500k
• Increase max stream_buffer_size to 2500

0.21.1 (2020-08-05)

Tap MySQL

• Fix two issues when a new discovery is done after detecting new changes in binlogs.

0.21.0 (2020-08-04)

• Improve alert messages to include botocore and generic python exception and error patterns in the alerts

Tap S3 CSV, Target Snowflake, Target S3 CSV, Target Redshift

• Add aws_profile option to support Profile based authentication to S3
• Add option to authenticate to S3 using AWS_PROFILE, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables

Target Snowflake

• Fixed an issue when target-snowflake was failing when QUOTED_IDENTIFIERS_IGNORE_CASE snowflake parameter set to True
• Fixed an issue when new SCHEMA singer message triggered a flush event even if the newly received SCHEMA message is the same as the previous one
• Add s3_endpoint_url option to support non-native S3 accounts

Target S3 CSV

• Add naming_convention option to create custom and dynamically named files on S3

Tap Snowflake

• Eliminate some warning messages of optional pandas and/or pyarrow not installed.

Tap Zendesk

• Fixed and issue when rate_limit, max_workers and batch_size were not configurable via the tap-zendesk YAML file

0.20.2 (2020-07-27)

• Fixed an issue when stop_tap command didn't kill running tap and child processes

0.20.1 (2020-07-24)

tap-mysql

• revert back to 1.3.2 as 1.3.3 causes replication to fail.

0.20.0 (2020-07-24)

• Fixed an issue when stop_tap command doesn't kill child processes only the parent PPW executable

Tap MongoDB

• Bump pipelinewise-tap-mongodb to 1.1.0
  • Add await_time_ms parameter to control how long the log_based method would wait for new change streams before stopping, default is 1000ms=1s which is the default anyway in the server.
  • Add update_buffer_size parameter to control how many update operation we should keep in the memory before having to make a call to find operation to get the documents from the server. The default value is 1, i.e every detected update will be sent to stdout right away.

Tap MySQL

• Bump pipelinewise-tap-mysql to 1.3.3
  • During LOG_BASED runtime, detect new columns, incl renamed ones, by comparing the columns in the binlog event to the stream schema, and if there are any additional columns, run discovery and send a new SCHEMA message to target. This helps avoid data loss.

Tap Zendesk

• Bump pipelinewise-tap-zendesk to 1.2.1
  • Use start_time query parameter to load satisfaction_ratings stream incrementally

Target Snowflake

• Bump pipelinewise-target-snowflake to 1.7.0
  • Add s3_acl option to support ACL for S3 upload

Target Redshift

• Bump pipelinewise-target-redshift to 1.5.0
  • Add s3_acl option to support ACL for S3 upload

0.19.0 (2020-07-21)

New features

• Add tap-github
• Extract and send known error patterns from logs to alerts

Tap Zendesk

• Bump pipelinewise-tap-zendesk to 1.2.0
  • Configurable rate_limit, max_workers and batch_size parameters

Bug fixes

Tap MongoDB

• Bump pipelinewise-tap-mongodb to 1.0.1
  • Fix case where resume tokens are not json serializable by extracting and saving _data only

0.18.1 (2020-07-15)

Bug fixes

• Fixed an issue when vault encrypted values were not in loaded from config.yml

0.18.0 (2020-07-14)

New features

• Add generic alert sender with Slack and VictorOps integration

Bug fixes

Tap Postgres

• Bump pipelinewise-tap-postgres to 1.6.3
  • Fixed a data loss issue when running LOG_BASED the tap not sending new SCHEMA

0.17.1 (2020-07-09)

Bug fixes

• Fixed an issue when using FastSync on big MongoDB collections caused memory errors
• Fixed an issue when sync_tables command was not working and failed with exception
• Fixed an issue when custom stream_buffer_size option produced unreadable log files