canadensys / canadensys-harvester Goto Github PK

Allow user to choose on config file if the harvester should check or not for non DwC-A terms

It could be handy to run the harvester without the distributed nodes.

The task 'CheckProcessingCompletenessTask' should also monitor the 'occurrence' table. The table name should be passed in parameter so we could create 2 tasks.

It would be handy to have the list (with line numbers) of duplicated id when we import an archive with such data.

Create canadensys-harvester-lib and canadensys-harvester-gui sub-projects.

Allow developers to register custom class to handle different type of extensions.

JMS messages could be optimized by combining ProcessOccurrenceMessage and InsertRawOccurrenceMessage since they both require the same input data. This would allow the system to use half of the message it is currently using.

We could also transport the data inside the message in a more efficient way by avoiding to repeat the field name for each records.

Nodes should be able to report fatal errors to the job 'initiator'.

• Change return type of handleMessage() to boolean in JMSConsumerMessageHandler interface.
• Create WriterException as generic exception for ItemWriterIF
• Create JMS classes for node error reporting: JMSErrorControlMessage, JMSErrorReceiver, JMSErrorReporter
• Create MonitorNodeJob, MonitorNodeStep (async)

Currently the 'node' is embedded in the canadensys-harvester-ui module. It doesn't make a lot of sense. The node should have it's own module. By allowing that, the node will only its own configuration class and file.

org.apache.commons.math.util.MathUtils > OccurrenceProcessor.java class

Currently, we can set our main IPT RSS feed address in the config file. Then, we can visualize it in the UI. Since we can have resources coming from various IPT, this view may not be expected.

We should have a dialog to create new resources (resource_management table).

If you use a default param in your DwcA* Descriptor file

e.g field default="PreservedSpecimen" term="http://rs.tdwg.org/dwc/terms/basisOfRecord"

then data[] are smaller than expected

Add a list view to manage resource : add, delete, update

Use the management tables to compare the last import date with the last publication date from the IPT RSS feed. How should this feature work? Simple option to the jar file that would be called by a cron job? Integrated to the current GUI? Simple option to the .jar file but the app run indefinitely? All of the above?

With the addition of extensions support some refactoring is now needed in order to keep things in order.

• ProcessingStepIF will be renamed to StepIF
• 2 new abstract classes will be created : AbstractStreamStep and AbstractReceiverStep
• Create StepResult class that StepIF would return on doStep();