Specgen

[lovromazgon]

Description

Depends on ConduitIO/conduit-commons#131 and https://github.com/ConduitIO/evolviconf (we should clean up that repo and release it so we depend on a released tag).

This contains the new specgen tool which extracts the connector specification into a YAML file. It should always be in the root of the repository and called connector.yaml (or connector.yml).

Usage

After this PR is merged and released, the developer should be able to install the tool using:

go install github.com/conduitio/conduit-connector-sdk/specgen

It should be added in a connector repository in the tools.go file as an import instead of paramgen:

  import (
- 	_ "github.com/conduitio/conduit-commons/paramgen"
+ 	_ "github.com/conduitio/conduit-connector-sdk/specgen"
  )

In the connector.go file it should be added in a go:generate directive, it shouldn't need any flags, just simply:

//go:generate specgen

When upgrading an existing connector to use specgen, you should update the code according to the code changes described in the chapter below, and then generate the initial connector.yaml so that the existing specification is copied over. After that, you can delete the spec.go file and change the connector.go file according to this:

//go:embed connector.yaml
var specs string

var Connector = sdk.Connector{
	NewSpecification: sdk.YAMLSpecification(specs),
	NewSource:        NewSource,
	NewDestination:   NewDestination,
}

This will ensure that the specifications are read from connector.yaml, making it the source of truth for the connector specifications.

Code changes in connectors

This PR changes a few things in the SDK, which will make it necessary to update the connector code. The good news is that it actually simplifies things.

No more Parameters

Neither the Source nor Destination interfaces require the Parameters method anymore. In fact, they were completely removed and are not used anymore. Connectors are expected to fetch the parameter specifications from the YAML file instead.

No more Configure

The Configure method was removed from Source and Destination. This simplifies the interfaces and does not require explicit parsing of the config anymore. Instead, the SDK will take care of parsing the config returned from the interfaces in Config.

Introduced Config

Config replaces both Parameters and Configure in Source and Destination. The connector should return the struct representing the connector configuration as a pointer. The pointer is required so that the SDK can parse the configuration map into that value, making the parsed configuration visible to the connector.

The returned config struct needs to implement SourceConfig / DestinationConfig. Both look the same, they allow the config (but don't require) to implement a method Validate. This will be called by the SDK after parsing the configuration, in case there are any further manual validations that need to happen. This essentially replaces Configure and makes it clear that it should only execute validations.

Note that the defined config struct should normally embed sdk.DefaultSourceMiddleware / sdk.DefaultDestinationMiddleware, so that the middleware is attached to the connector:

type MyConfig struct {
	sdk.DefaultSourceMiddleware
	// other fields ...
}

Another improvement is that the developer can't really "forget" to add the middleware, since the config needs to embed sdk.UnimplementedSourceConfig / sdk.UnimplementedDestinationConfig, not doing so will result in a compile-time error. So they need to either embed it directly (resulting in no middleware) or embed the default middleware struct mentioned above, which already includes the unimplemented config.

Middleware

Source/destination middleware implementations were drastically simplified because of these changes. The middleware is now essentially just a config struct, which needs to be embedded in the connector config. The parameters are automatically parsed when parsing the config, since they are pointers to fields in the config struct defined in the connector. This means that the middleware now doesn't have to manually implement Parameters and Configure, it is all done using specgen and the SDK.

The default middleware is defined in sdk.DefaultSourceMiddleware / sdk.DefaultDestinationMiddleware, which needs to be embedded by the config struct defined in the connector. That config struct now dictates which middleware will wrap the source/destination. If the developer wants to overwrite any defaults in the middleware, they can do so by setting the corresponding values when instantiating the connector:

type Source struct {
	sdk.UnimplementedSource

	config MyConfig
}

func NewSource() sdk.Source {
	return sdk.SourceWithMiddleware( // Extract the middleware from the config and wrap the source
		&Source{
			config: MyConfig{
				DefaultSourceMiddleware: sdk.DefaultSourceMiddleware{ // Change defaults in the default middleware
					SourceWithSchemaExtraction: sdk.SourceWithSchemaExtraction{
						PayloadEnabled: lang.Ptr(false), // Set default value to false
						KeyEnabled:     lang.Ptr(false), // Set default value to false
					},
				},
			},
		},
	)
}

// Config returns a pointer to the config struct. Values will be automatically populated by the SDK.
func (s *Source) Config() sdk.SourceConfig {
	return &s.config
}

Normally, when the developer doesn't need to overwrite any defaults, this can be simplified:

func NewSource() sdk.Source {
	return sdk.SourceWithMiddleware(&Source{})
}

Open work

What's still missing in this PR:

• Rename connector field in the YAML file to specification. Will be done in specgen: add more tests, rename connector to specification #206.
• specgen: Order fields (required first, then by name alphabetically) #209
• Update the validation function to use struct tags instead of requiring the parameter specifications (NOTE: this part needs to be done in conduit-commons and will affect processors as well, although positively).
  • Kept it simple for now and used the same validation strategy we used before (using parameters from the specification), only that it's executed in the SDK. We can change this in the future, since this is now an internal thing in the SDK.
• Make sure that Validate is called on SourceConfig / DestinationConfig and make sure that the same method is called for all middleware configs as well.
  • Kept this simple as well, if the developer overwrites the Validate method in their own connector they need to call the middleware Validation method as part of the implementation. It's called out in the documentation.
• Acceptance tests need to be uncommented and adjusted, same goes for the benchmark and probably a lot of the test files.
• Tests, a lot of tests...

Related to ConduitIO/conduit#1523.

Related issues:

• paramgen sets the type 'string' for array fields conduit-commons#137
• paramgen generates wrong parameter name for an all-uppercase field conduit-commons#138

Quick checks:

• I have followed the Code Guidelines.
• There is no other pull request for the same update/change.
• I have written unit tests.
• I have made sure that the PR is of reasonable size and can be easily reviewed.

[raulb]

Everything I would comment on before considering this in a state other than draft is already tracked on ConduitIO/conduit#1523, so I'm good with making this part of the feature branch so we can iterate on the main feature independently.