Add input and output type to connectors about connectors HOT 16 CLOSED

@gashcrumb @chirino @kahboom just a note that the camel connectors now have information about what data they accept as input and produce as output.

There is some docs about it here
https://github.com/apache/camel/blob/master/connectors/camel-connector/src/main/docs/connector-component.adoc#input-and-output-data-type

The connectors for ipaas has been updated so they all have

• inputDataType
• outputDataType

So for example the twitter -> salesforce example in spring 10, you can see that twitter

 "inputDataType": "none",
 "outputDataType": "java:twitter4j.Status",

https://github.com/redhat-ipaas/connectors/blob/master/connectors/twitter-mention-connector/src/main/resources/camel-connector.json#L15-L16

And that salesforce connector has:

  "inputDataType": "java:org.foo.salesforce.upsert.contact.Contact",
  "outputDataType": "none",

https://github.com/redhat-ipaas/connectors/blob/master/connectors/salesforce-upsert-contact-connector/src/main/resources/camel-connector.json#L15-L16

That way the ipaas can know that the output from the twitter is a Java type of twitter4j.Status class. And that the input of the salesforce is also Java type but of type org.foo.salesforce.upsert.contact.Contact which means they are not 1:1 and a data mapper must be added as a step.

from connectors.

If we look at that other example with the http-push-pull then we start from a timer which has

  "inputDataType": "none",
  "outputDataType": "*",

https://github.com/redhat-ipaas/connectors/blob/master/connectors/timer-connector/src/main/resources/camel-connector.json#L15-L16

The timer outputs an empty (null) message and I am actually in doubt if we should have a special data type that says "empty" or something? I set it to * which mean "any". I could also set it to "none" but maybe that is confusing when both are none? Or we can add that "empty" ? or something.

Then the next step is a HTTP GET connector, that has

  "inputDataType": "none",
  "outputDataType": "*",

https://github.com/redhat-ipaas/connectors/blob/master/connectors/http-get-connector/src/main/resources/camel-connector.json#L15-L16

The HTTP GET connector does not use any input as its a HTTP GET call and there it was set to none. The output of the HTTP GET can be anything, eg its the response from the HTTP call so its * to indicate any. If the connector would only be used for calling HTTP services that returns JSon then we could set it to that etc. Its just meta-data after all.

Then the example supposedly does a transformation and then do a HTTP POST. I am not sure what that transformation was supposed to return as output.

But the HTTP POST is currently set to be open ending with

  "inputDataType": "*",
  "outputDataType": "*",

https://github.com/redhat-ipaas/connectors/blob/master/connectors/http-post-connector/src/main/resources/camel-connector.json#L15-L16

from connectors.

FYI: @davsclaus If we are going to pick a java class for one of the input types, I'd recommend that it not be an interface as the datamapper won't know how to construct one.

from connectors.

@chirino It would only be the output type the data mapper would need to create an instance. The input is already coming from Camel at runtime.

So the data mapper MUST ideally be able to introspect an incoming interface, as that is what twitter4j.Status is.

It seems the twitter classes are not public scoped which means the data mapper would not easily be able to create an instance of those twitter classes anyway.

from connectors.

@chirino Also I wonder if we can make the data mapper introspect at built time, so we can generate some kind of xml or json data dictionary of these models. Then the ipaas web tooling would likely be easier to use as we can feed this information into the web ui without having to run java code - and have trouble with not having connector JARs on the classpath.

from connectors.

@davsclaus input to connector comes from the output of the datamapper right? So we are saying the same thing. They are introspecting at mapping design time. We have talked about optionally doing the introspection when the connector is being designed and storing that introspection in the connector to avoid the need to do it at mapping design time.

from connectors.

We'll need to get these fields into the model on the rest API, whose handling that at the moment?

from connectors.

Yes Camel takes the output of the data mapper and routes that. So its the data mapper that creates a new instance of the output java type and then map the fields to that from the existing input message.

But that happens when you run the integration, eg Camel inside funktion.

But for the web console you need something different. Ideally we have those models pre-generated. If not then the ipaas java backend need somehow to generate that model, which is much harder due those java class are not in its classpath. Which means you need to put every single camel connector and all its java dependencies into a giant mud of JVM.

It would be much better to generte those models when you build the camel connector. We have a maven plugin that runs. It could then call the data mapper that builds that data model which then gets embedded in the JAR.

from connectors.

Also is there any thoughts on documentation for those fields?

And also @kcbabo talked about limiting number of fields. For example the twitter-mention connector you may only want to show N number of fields in the data mapper, and not all 1500+ fields in the full model going 3 level deep etc.

So if the developer creating the connector could pre-configure this so we have

• the full model with 1500 fields (just in case)
• configured which fields to only show in the data mapper

Also it would allow the developer to amend that model, to add for example documentation to those fields. So the data mapper UI can show that to the citizen developer. Not everyone knows what a ScreenName is etc.

from connectors.

@gashcrumb Someone also needs to take a peek at what the datamapper UI component actually needs. It might need more than "java:org.foo.salesforce.upsert.contact.Contact"

from connectors.

@davsclaus the datamapper won't be generating any models, it expects that the ipaas will supply it with all the classes that it needs. So yeah the connector has to have embedd all the data models in it.

from connectors.

@davsclaus I think keith called that supporting a data dictionary which I do think will be valuable in the future and we might be able to generate much of it from stuff like javadocs. But I think the real fix for citizen developers is to use instance data in the mappings.

from connectors.

@davsclaus where you said in a previous comment "Also is there any thoughts on documentation for those fields?" are you referring to user doc for the fields that an ipaas user can select for configuring a connector to be a connection?
If yes, is it possible for us to provide hover help in the ipaas GUI when the ipaas user is selecting the fields for the connection?

from connectors.

Yes I mean some kind of description to what all those fields are? Or just some general documentation what that model is.

eg in this example its a twitter mention connector. So what you would see is a twitter status model. We could then have some kind of description to show the user what that is.

As the twitter status model has a lot of fields there is a lot to document if we should do for all of them. And those fields are twitter specific, so not so sure how much extra value we can bring

from connectors.

Hmmm. So, what is the model that we need to document?

It sounds to me like we are talking about two different things here. I was asking about user doc for the fields that an ipaas user chooses in order to configure a connector and make it a connection. I would think those fields correspond to Camel component options. Am I making a correct assumption about that?

It sounds like you are responding about the fields that will show up in the data mapper. These are the fields that are specific to each application's connector. Correct?

Please set me straight, and I sure hope I am not wasting your time. Thanks!

from connectors.

Yes its two different things.

For questions about the ipaas web console and what to document from user pov, then I suggest to talk to the PM @kcbabo and also the front end developers like @gashcrumb and @kahboom knows the screens the best.

from connectors.