Schema JSON type Schema Registry Client version 5.5.1 about kloadgen HOT 35 CLOSED

Thank you for collaborate with the project giving us feedback!!' You will help us a lot if you can attach and example AVRO and an example Test Plan in order to be able to reproduce the error. If that is the case. Cheers

from kloadgen.

HI @Desire456,
Can you please provide some test plan and json schema format as a way to test the solution?
Cheers!

from kloadgen.

Hi @kszosze,
There are test plan and json schema.
test-plan-schema.zip

from kloadgen.

Hi @Desire456,
There is a branch with a first approach to support Json Schema..
Can you please test if that suits you?
Cheers!!

from kloadgen.

Hi @kszosze,
Test plan what I attach first time work fine. But in the same test plan I with this json schema I have this stacktrace while I'm trying load subject.
schema.zip

2020-11-06 14:32:52,372 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[AWT-EventQueue-0,6,main]
java.lang.NullPointerException: null
	at net.coru.kloadgen.extractor.parser.impl.JSONSchemaParser.buildCombinedField(JSONSchemaParser.java:404) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.parser.impl.JSONSchemaParser.chooseAnyOf(JSONSchemaParser.java:382) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.parser.impl.JSONSchemaParser.buildProperty(JSONSchemaParser.java:307) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.parser.impl.JSONSchemaParser.lambda$parse$0(JSONSchemaParser.java:74) ~[kloadgen-plugin-3.0.0.jar:?]
	at org.apache.commons.collections4.CollectionUtils.collect(CollectionUtils.java:1151) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.parser.impl.JSONSchemaParser.parse(JSONSchemaParser.java:73) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.extractors.JsonExtractor.processSchema(JsonExtractor.java:37) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.extractor.impl.SchemaExtractorImpl.flatPropertiesList(SchemaExtractorImpl.java:94) ~[kloadgen-plugin-3.0.0.jar:?]
	at net.coru.kloadgen.input.avro.AvroSubjectPropertyEditor.actionPerformed(AvroSubjectPropertyEditor.java:88) ~[kloadgen-plugin-3.0.0.jar:?]
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967) ~[?:?]
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308) ~[?:?]
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405) ~[?:?]
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262) ~[?:?]
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:279) ~[?:?]
	at java.awt.Component.processMouseEvent(Component.java:6635) ~[?:?]
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3342) ~[?:?]
	at java.awt.Component.processEvent(Component.java:6400) ~[?:?]
	at java.awt.Container.processEvent(Container.java:2263) ~[?:?]
	at java.awt.Component.dispatchEventImpl(Component.java:5011) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2321) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4843) ~[?:?]
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918) ~[?:?]
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547) ~[?:?]
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2307) ~[?:?]
	at java.awt.Window.dispatchEventImpl(Window.java:2772) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4843) ~[?:?]
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:772) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:721) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:715) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:745) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:743) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) ~[?:?]
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:742) ~[?:?]
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) [?:?]
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) [?:?]
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) [?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) [?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) [?:?]
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:90) [?:?]

from kloadgen.

Hi Vladimir,
actual implementation doesn't support optional fields, as your schema looks to have. But not sure if that schema is totally right. Which version are you actually using?
Cheers

from kloadgen.

I'm using Draft-07 version. This schema is being validated by org.everit.json-schema.
Cheers!

from kloadgen.

I think it validates becouse anyOf, oneOf, allOf only requires an array of valid Json, and that is what you have, but for nullable properties the way is use an array of types... as you can see here and here I saw your format here but as a failure for Draft 4. Can you please attach where in the specification is this a way to specify a possible null value for a property?
Cheers!!

from kloadgen.

Ok. But if I use anyOf, oneOf, allOf without null-type I don't see types and properties for my schema. It is normal? For example, schema: schema.zip
Cheers!

from kloadgen.

I think you want to have a property with multiple possible types, what as far as I know it is not possible with json-schema... here you can see the specification that in theory it should but there are couple of issues about here and here. I think what you want to achieve in your schema is something like
{ "$schema": "http://json-schema.org/draft-07/schema#", "title": "Transformed Event", "type": "object", "additionalProperties": false, "properties": { "id": { "type": ["null", "integer", "string"] }, "version": { "type": ["null", "integer", "string"] }, "dtype": { "type": ["null", "integer", "string"] }, "timestamp": { "type": ["null", "integer", "string"] }, "event_type": { "type": ["null", "integer", "string"] } } }
at the moment we are not supporting multiple types for a property, since the official Draft does not too.
We are using our own parser due arquitectural reasons, so we can apply some changes if those are justified enough.
Cheers!!

from kloadgen.

But official Draft supports multiple types: https://json-schema.org/understanding-json-schema/reference/combining.html#allof. I'm really need that. If u don't do this say me about this please!
Cheers!

from kloadgen.

Well documentation say so and also they say no. We can give a try, I'll prepare something to handle that.

from kloadgen.

Thanks you!

from kloadgen.

Hi Vlad,
I just added some code to support that.. do you mind test it in you use case if it fit?
Cheers

from kloadgen.

Hi Jose,
That works better, but I want to see muti types in field type column Jmeter. For example: oneOf : {string, integer} : field type - string, integer. Can you do it?
Cheers!

from kloadgen.

Well, that cannot be possible with this tool, since we use those value on the table to generate the artificial objets which will be sent to the topic. But that field is editable, so you can replace string or integer as you need.
For now is the most we can do. Is in our roadmap create a better component but still on consideration.

from kloadgen.

Ok, thank you very much!!

from kloadgen.

So current approach works for you? without the double representation for multiple types? can you produce valid messages and deserialise them correctly?

from kloadgen.

Not sure about correct message. Because in AKHQ if my schema is json - I see that :
.
But if I use avro - I see that:
.
First symbols for KafkaSerializer in AVRO case exist.
And I haven't keys: in both cases I see null key. Don't sure is it right?
Cheers!

from kloadgen.

Had you configure the serialiser as KafkaJsonSchemaSerializer? or are you trying to use the AVRO one?

from kloadgen.

Yes, I configure the serializer as KafkaJsonSchemaSerializer.

from kloadgen.

Sorry for null keys. It's my bad: I don't use property keyed.message...
Cheers!

from kloadgen.

No problem, so now what is the status?

from kloadgen.

First unreadable symbols for schema in case of JSON doesn't exist.
What I get:

What I want:

Without this symbols KafkaJsonSerializer can't deserialize my data.

from kloadgen.

Well, we generate a JSON object and then client apply the serialiser you have configured. We will take a look about the serialisation process.
Cheers!!

from kloadgen.

Yes, thank you.
Cheers!

from kloadgen.

Hi Vlad,
sorry for the delay, busy days.
I have a new candidate. I was able to consume messages generated with the KafkaJsonSerializer.
Can you please verify that solves your problem?
Cheers

from kloadgen.

Hi Jose!
Yes, it's solved my problem!
Please help me If I want to set auto.register.schemas = false in kafka producer properties: what I must do? Just add new property in java request doesn't work.
Cheers!

from kloadgen.

HI Vlad,
It should work if you added that property. I'll double check the java request properties. Which Sampler are you using? Confluent?

from kloadgen.

GenericKafkaSampler. I add this property and it deleted. I don't know why

from kloadgen.

Aye, is a weird behaviour of the latest versions of JMeter.. whatever property is not in the default set is removed. Still analysing if is a feature or a bug.
We design Generic and Kafka Samplers to be able to work without an schema registry, just allowing upload a file and send messages, so is why that property is not in the default set.

from kloadgen.

Hi Vlad,
added the property as default for now.
Now you can enable and or disable as you wish.
Cheers!!

from kloadgen.

Hi Jose,
Thank you very much!
Cheers!

from kloadgen.

HI @Desire456,

Hd you been able to success on your tests?

Regards

from kloadgen.

Hi @kszosze,

Yes, thank you.

Cheers!

from kloadgen.