-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ISSUE#47]Optimize ConnectRecord api #50
Open
sunxiaojian
wants to merge
33
commits into
openmessaging:master
Choose a base branch
from
sunxiaojian:dev
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
c15757b
SinkTask and SourceTask implement the validate method https://github.…
sunxiaojian dcc706d
Adjust the init and start methods of the component interface
sunxiaojian fa134ee
Set pause and resume to deprecated methods. It feels like they can be…
sunxiaojian 55b9ac0
Add struct object and optimize schema and schema builder API #41
sunxiaojian c72659e
add offset storage writer #41
sunxiaojian e6deeb8
add getter and setter method #41
sunxiaojian aa9f315
add SchemaAndValue #41
sunxiaojian 23ed62a
add logical type #41
sunxiaojian 26fe155
Schemabuilder add required method
sunxiaojian 3e6f1e6
schema add hashCode and equals method
sunxiaojian 14765cb
fixed doc method
sunxiaojian a6b3c79
Field add equals and hashcode method
sunxiaojian 427e009
optimize api #85
sunxiaojian 1dc98a3
Merge branch 'dev'
sunxiaojian a38fa65
Merge branch 'optimize-api'
sunxiaojian c2314fb
Merge branch 'openmessaging:master' into master
sunxiaojian c4b1836
Optimize transform api #45
sunxiaojian c76d2fb
Optimize transform api and add RecordConverter
sunxiaojian ed258b0
Merge branch 'openmessaging:master' into master
sunxiaojian 5de17b2
Optimize ConnectRecord api #47
sunxiaojian 898863c
upgrade api to 0.1.4-SNAPSHOT;
sunxiaojian 6c08958
optimize connector
sunxiaojian db8f31f
Merge branch 'openmessaging:master' into master
sunxiaojian 1b698b7
Merge branch 'openmessaging:master' into dev
sunxiaojian ea7e3c0
[ISSUE#47]Optimize ConnectRecord api #50
sunxiaojian fabb29e
Optimize ConnectRecord remove queueId field #50
sunxiaojian e94593e
Optimize ConnectRecord remove queueId field #47
sunxiaojian 80fb698
Merge branch 'dev' of https://github.com/sunxiaojian/openconnect into…
sunxiaojian 2fa5bb1
Merge branch 'master' into dev
sunxiaojian d539510
Merge branch 'openmessaging:master' into master
sunxiaojian c254a7a
Connectrecord add a key field to identify the unique data #53
sunxiaojian 58a9143
Merge branch 'openmessaging:master' into master
sunxiaojian 7036d48
merge new features #47
sunxiaojian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
...rc/main/java/io/openmessaging/connector/api/component/task/sink/SinkConnectorContext.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package io.openmessaging.connector.api.component.task.sink; | ||
|
||
|
||
import io.openmessaging.connector.api.component.connector.ConnectorContext; | ||
|
||
/** | ||
* source connector context | ||
*/ | ||
public interface SinkConnectorContext extends ConnectorContext { | ||
|
||
} |
115 changes: 115 additions & 0 deletions
115
connector/src/main/java/io/openmessaging/connector/api/component/task/sink/SinkRecord.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,115 @@ | ||
/* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package io.openmessaging.connector.api.component.task.sink; | ||
|
||
import io.openmessaging.KeyValue; | ||
import io.openmessaging.connector.api.data.ConnectRecord; | ||
import io.openmessaging.connector.api.data.Schema; | ||
|
||
import java.util.Objects; | ||
|
||
/** | ||
* sink connect record | ||
*/ | ||
public class SinkRecord extends ConnectRecord<SinkRecord> { | ||
|
||
private final Integer queueId; | ||
private final String brokerName; | ||
private final long queueOffset; | ||
|
||
public SinkRecord(String brokerName, long queueOffset,String topic, Integer queueId, Schema schema, Object data) { | ||
this(brokerName, queueOffset, topic, queueId, null, null, schema, data ,null ); | ||
} | ||
|
||
public SinkRecord(String brokerName, long queueOffset, String topic, Integer queueId, Schema keySchema, Object key, Schema schema, Object data) { | ||
this(brokerName, queueOffset, topic, queueId, null, keySchema, key, schema, data ,null ); | ||
} | ||
|
||
public SinkRecord(String brokerName, long queueOffset, String topic, Integer queueId, Schema keySchema, Object key, Schema schema, Object data, KeyValue extensions) { | ||
this(brokerName, queueOffset, topic, queueId, null, keySchema, key, schema, data ,extensions ); | ||
} | ||
|
||
public SinkRecord(String brokerName, long queueOffset, String topic, Integer queueId, Long timestamp, Schema keySchema, Object key, Schema schema, Object data, KeyValue extensions) { | ||
super(topic, timestamp, keySchema, key, schema, data, extensions); | ||
this.brokerName = brokerName; | ||
this.queueOffset = queueOffset; | ||
this.queueId = queueId; | ||
} | ||
|
||
|
||
public Integer queueId(){ | ||
return queueId; | ||
} | ||
|
||
public String brokerName(){ | ||
return brokerName; | ||
} | ||
|
||
public long queueOffset(){ | ||
return queueOffset; | ||
} | ||
|
||
/** | ||
* new record | ||
* | ||
* @param topic | ||
* @param schema | ||
* @param data | ||
* @param timestamp | ||
* @return | ||
*/ | ||
@Override | ||
public SinkRecord newRecord(String topic, Long timestamp, Schema keySchema, Object key,Schema schema, Object data) { | ||
return newRecord(topic, timestamp, keySchema, key, schema, data, null); | ||
} | ||
|
||
/** | ||
* new record | ||
* | ||
* @param topic | ||
* @param schema | ||
* @param data | ||
* @param timestamp | ||
* @param extensions | ||
* @return | ||
*/ | ||
@Override | ||
public SinkRecord newRecord(String topic, Long timestamp, Schema keySchema, Object key, Schema schema, Object data, KeyValue extensions) { | ||
return new SinkRecord(brokerName(), queueOffset(), topic, queueId(), timestamp, keySchema, key, schema, data, extensions); | ||
} | ||
|
||
@Override | ||
public boolean equals(Object o) { | ||
if (this == o) return true; | ||
if (!(o instanceof SinkRecord)) return false; | ||
if (!super.equals(o)) return false; | ||
SinkRecord that = (SinkRecord) o; | ||
return queueOffset == that.queueOffset && Objects.equals(queueId, that.queueId) && Objects.equals(brokerName, that.brokerName); | ||
} | ||
|
||
@Override | ||
public int hashCode() { | ||
return Objects.hash(super.hashCode(), queueId, brokerName, queueOffset); | ||
} | ||
|
||
@Override | ||
public String toString() { | ||
return "SinkRecord{" + | ||
"queueId=" + queueId + | ||
", brokerName='" + brokerName + '\'' + | ||
", queueOffset=" + queueOffset + | ||
"} " + super.toString(); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
...ain/java/io/openmessaging/connector/api/component/task/source/SourceConnectorContext.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package io.openmessaging.connector.api.component.task.source; | ||
|
||
|
||
import io.openmessaging.connector.api.component.connector.ConnectorContext; | ||
import io.openmessaging.connector.api.storage.OffsetStorageReader; | ||
|
||
/** | ||
* source connector context | ||
*/ | ||
public interface SourceConnectorContext extends ConnectorContext { | ||
|
||
/** | ||
* @return the OffsetStorageReader for this connector. | ||
*/ | ||
OffsetStorageReader offsetStorageReader(); | ||
} |
103 changes: 103 additions & 0 deletions
103
...ctor/src/main/java/io/openmessaging/connector/api/component/task/source/SourceRecord.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
/* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package io.openmessaging.connector.api.component.task.source; | ||
|
||
import io.openmessaging.KeyValue; | ||
import io.openmessaging.connector.api.data.ConnectRecord; | ||
import io.openmessaging.connector.api.data.RecordOffset; | ||
import io.openmessaging.connector.api.data.RecordPartition; | ||
import io.openmessaging.connector.api.data.RecordPosition; | ||
import io.openmessaging.connector.api.data.Schema; | ||
|
||
import java.util.Objects; | ||
|
||
/** | ||
* source connect record | ||
*/ | ||
public class SourceRecord extends ConnectRecord<SourceRecord> { | ||
|
||
private final RecordPosition position; | ||
|
||
public SourceRecord(RecordPartition recordPartition, RecordOffset recordOffset,String topic, Schema schema, Object data) { | ||
this(recordPartition, recordOffset, topic, null, null, null, schema , data, null); | ||
} | ||
|
||
public SourceRecord(RecordPartition recordPartition, RecordOffset recordOffset,String topic, Schema keySchema, Object key, Schema schema, Object data) { | ||
this(recordPartition, recordOffset, topic, keySchema, key, schema , data, null); | ||
} | ||
|
||
public SourceRecord(RecordPartition recordPartition, RecordOffset recordOffset, String topic, Schema keySchema, Object key, Schema valueSchema, Object value, KeyValue extensions) { | ||
this(recordPartition, recordOffset, topic, null, keySchema, key, valueSchema, value, extensions); | ||
} | ||
|
||
public SourceRecord(RecordPartition recordPartition, RecordOffset recordOffset, String topic, Long timestamp, Schema keySchema, Object key, Schema schema, Object data, KeyValue extensions) { | ||
super(topic, timestamp, keySchema, key, schema, data, extensions); | ||
this.position = new RecordPosition(recordPartition, recordOffset); | ||
} | ||
|
||
public RecordPosition position() { | ||
return position; | ||
} | ||
|
||
/** | ||
* new record | ||
* | ||
* @param topic | ||
* @param schema | ||
* @param data | ||
* @param timestamp | ||
* @return | ||
*/ | ||
@Override | ||
public SourceRecord newRecord(String topic, Long timestamp, Schema keySchema, Object key,Schema schema, Object data) { | ||
return newRecord(topic, timestamp, keySchema, key, schema , data, null ); | ||
} | ||
|
||
/** | ||
* new record | ||
* | ||
* @param topic | ||
* @param schema | ||
* @param data | ||
* @param timestamp | ||
* @param extensions | ||
* @return | ||
*/ | ||
@Override | ||
public SourceRecord newRecord(String topic, Long timestamp, Schema keySchema, Object key,Schema schema, Object data, KeyValue extensions) { | ||
return new SourceRecord(position().getPartition(), position().getOffset(), topic, timestamp, keySchema, key, schema, data, extensions); | ||
} | ||
|
||
@Override | ||
public boolean equals(Object o) { | ||
if (this == o) return true; | ||
if (!(o instanceof SourceRecord)) return false; | ||
if (!super.equals(o)) return false; | ||
SourceRecord that = (SourceRecord) o; | ||
return Objects.equals(position, that.position); | ||
} | ||
|
||
@Override | ||
public int hashCode() { | ||
return Objects.hash(super.hashCode(), position); | ||
} | ||
|
||
@Override | ||
public String toString() { | ||
return "SourceRecord{" + | ||
"position=" + position + | ||
"} " + super.toString(); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenMessaging Connect是一个通用的框架,不适合将RMQ个性化的元素(queueId,brokerName,queueOffset)放到框架里。