3.22 #6
Open
3.22 #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ges" This reverts commit 7583728. Change-Id: I24f2576e8971abf799e8880bee61052fc0f24b1a Reviewed-on: https://review.streamsets.net/c/datacollector/+/40040 Reviewed-by: Jarcec Cecho <[email protected]>
…Properties validation" This reverts commit deda62f. Change-Id: I3eefe6cb184e5df11904eed062a7a17d4411f0a6 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40041 Reviewed-by: Jarcec Cecho <[email protected]>
…atch Transformer" This reverts commit 3545576. Change-Id: I83d40a1c4db1463c1d0d2bfdc3ce91438a709926 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40042 Reviewed-by: Jarcec Cecho <[email protected]>
Change-Id: Id04eef4aa323b7625cbf4e8960bc8739be1fd432 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40054 Reviewed-by: Alex Sanchez Cabana <[email protected]> Reviewed-by: Loretta Chen <[email protected]> Tested-by: StreamSets CI <[email protected]> (cherry picked from commit 761266b) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40014
Change-Id: I7f9ee0f22038d8861347b39ccf3c2c3428f3e4c5 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40048 Reviewed-by: Jarcec Cecho <[email protected]> (cherry picked from commit 171c7be) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40015
…o JDK 8 Change-Id: I6698a044acf281238c10925236b40241bf5acd9a Reviewed-on: https://review.streamsets.net/c/datacollector/+/40104 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Jarcec Cecho <[email protected]> (cherry picked from commit 71c9085) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40089 Reviewed-by: Xavier Baqués <[email protected]>
* Verified manually it is there and things work correctly. Change-Id: I840b13da186f84d09706137b7dbcf14d7e7441f8 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40310 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Xavier Baqués <[email protected]> (cherry picked from commit 92ae353) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40333 Reviewed-by: Jarcec Cecho <[email protected]>
…sion Change-Id: Ia90c6d9dd83674146005446a2dc42969918f9f9c Reviewed-on: https://review.streamsets.net/c/datacollector/+/40066 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Alex Sanchez Cabana <[email protected]> (cherry picked from commit 199ad0e) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40335 Reviewed-by: Jarcec Cecho <[email protected]>
…when database and SDC are in different timezones We used CURRENT_TIMESTAMP AT TIME ZONE DBTIMEZONE as the NEXT_TIME value configured for the CURRENT online redo log in the LogMinerSession queries. However, CURRENT_TIMESTAMP carries timezone info (it's an Oracle TIMESTAMP WITH TIMEZONE value), while the rest of NEXT_TIME values don't (they are Oracle DATE values). LogMinerSession implementation expects DATE values expressed in local database time, so that they can be directly converted from Timestamps to LocalDateTimes. When doing so over a TIMESTAMP WITH TIMEZONE type, the result is wrong -- the time ends up being expressed as a local time in the configured JVM's timezone. This patch avoids using CURRENT_TIMESTAMP (i.e. a TIMESTAMP WITH TIMEZONE value) in favor of LOCALTIMESTAMP (i.e. a DATE value). Note that in order for this to work, the DB session is configured with the DB timezone. Change-Id: Id7b928a4deb44529b362321d6f112a91b1bc43b6 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40392 Reviewed-by: Alex Sanchez Cabana <[email protected]> Tested-by: StreamSets CI <[email protected]> (cherry picked from commit 98a2659) Reviewed-on: https://review.streamsets.net/c/datacollector/+/40336
… sentences. Change-Id: I6656aea07df4970903ecca41ea7e552978f961a0 Reviewed-on: https://review.streamsets.net/c/datacollector/+/39993 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Toni Martinez <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40967 Reviewed-by: Hugo Prol <[email protected]>
…e transactions. Change-Id: Id08863cdb2bb038ec086588cdc4e8033908c96d2 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40610 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Adrià Aguilà <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40968 Reviewed-by: Hugo Prol <[email protected]>
… space until service is restarted. Upgraded version of MapDB. Remove transactionallity for file-based maps as we are not relying on it. Remove all file based maps when Rollback or Expiration. Improve performance using smaller chunks for incremental file based maps space allocations. Implement element() method necessary for the expire() method for file based maps. Force deletion of parent temporary folders when file based maps released and deleted. Set to null all resources related to file base maps when released to help GC free asap opened mmap file descriptors. Use specific techniques of the new version of MapDB to enforce freeing FDs and other resources related to file based maps. Change-Id: Ib7e67f5a536afa21a250b90f5881e1364a19a001 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40620 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Toni Martinez <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40969 Reviewed-by: Hugo Prol <[email protected]>
… data loss on high throughput conditions On high throughput load conditions and depending on specific redo log settings (e.g. number and size of the online redo logs), Oracle can perform several rotations of online redo logs in a short period of time. If an active LogMiner session is processing one of the affected online logs, the session can silenty fail – it won't retrieve more data and could finish with no distinction from a normal "no more pending data" scenario. In order to solve this situation, Oracle CDC needs to check the integrity of the LogMiner session after finishing the mining process for the current window (this involves comparing redo log metadata from before and after the LogMiner session for all the online redo logs, if any, used in that mining window). And if the integrity has potentially been compromised, Oracle CDC must replay the process for the current mining window. Change-Id: If6af8c351ed7a8f1f229f2e2054004ff789b7990 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40524 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Toni Martinez <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40970 Reviewed-by: Hugo Prol <[email protected]> Reviewed-by: Xavier Baqués <[email protected]>
…ision In LogMinerSession#updateLogList, we are comparing timestamps to detect online redo log transient states for the redo logs returned by the SELECT_REDO_LOGS_QUERY. However, while the redo log timestamps has no fractional seconds precision and the query is run with seconds precision, the LogMinerSession#updateLogList is running comparisons with milliseconds precision. We need to truncate the timestamps limits, otherwise, under some conditions (i.e. when log.getNextTime >= end returns false due to the millisecond precision), the method will wrongly detect a transient state which leads Oracle CDC to unnecessarily enter in retry mode. Following the same approach in other methods that compare redo log timestamps with time limits (albeit in these other places there was no harm). Change-Id: I448a51e85088a6ffe0d6be845842cb4e4edcdb74 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40525 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Toni Martinez <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40971 Reviewed-by: Hugo Prol <[email protected]> Reviewed-by: Xavier Baqués <[email protected]>
1. When selecting redo logs covering a given time window (start, end), Oracle CDC must pick also redo logs with NEXT_TIME = start. They can indeed contain operations logged at timestamp `start`. 2. Similar reason when selecting redo logs covering a given SCN -- Oracle CDC must pick also redo logs with NEXT_CHANGE# = SCN, as they can store operations logged at that SCN. Change-Id: Ied628f8ee4ddef85686932ae02502709be028c00 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40554 Reviewed-by: Toni Martinez <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40972 Reviewed-by: Hugo Prol <[email protected]> Reviewed-by: Xavier Baqués <[email protected]>
…acle 19+ on CDC does not work as expected. Change-Id: Ibd401e7bcc6e5d6e342c9768304aad2ea34d0953 Reviewed-on: https://review.streamsets.net/c/datacollector/+/40723 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Hugo Prol <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40973
… database incarnation On databases with multiple incarnations, we should ensure that the redo logs registered in LogMiner belong to the current incarnation. Otherwise, when starting from a SCN or timestamp which is present in several incarnations, Oracle CDC could load redo log files from the wrong incarnation, leading to LogMiner start failures or wrong CDC data returned. Change-Id: I140c99d8ad7af8b20a59f632289306fa496c3dac Reviewed-on: https://review.streamsets.net/c/datacollector/+/40633 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Dimas Cabré i Chacón <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40974 Reviewed-by: Hugo Prol <[email protected]> Reviewed-by: Xavier Baqués <[email protected]>
…Headers produced by Oracle CDC Client Origin. Change-Id: I26f51b443460b65ce8124ab13b9dfbac28d3323f Reviewed-on: https://review.streamsets.net/c/datacollector/+/40956 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Alex Sanchez Cabana <[email protected]> Reviewed-on: https://review.streamsets.net/c/datacollector/+/40975 Reviewed-by: Hugo Prol <[email protected]>
Change-Id: Ia027b6742bf1d96591e91e910c045e5c413b7d21 Reviewed-on: https://review.streamsets.net/c/datacollector/+/41003 Tested-by: StreamSets CI <[email protected]> Reviewed-by: Alex Sanchez Cabana <[email protected]> (cherry picked from commit b5472cd) Reviewed-on: https://review.streamsets.net/c/datacollector/+/41023 Reviewed-by: Hugo Prol <[email protected]>
|
Rohit Pawar seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
I want to write a plugin to add-on. Any suggestion on template/doc to write?
Thank in advanced,