Apache Flink is quickly gaining momentum as an alternative to Spark Streaming, Storm, etc.

Apache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

What are your thoughts on developing a plugin for Flink Streaming in StreamFlow? The rationale is that Flink provides a Storm compatible API:

Flink provides a Storm compatible API (org.apache.flink.storm.api) that offers replacements for the following classes:

TopologyBuilder replaced by FlinkTopologyBuilder
StormSubmitter replaced by FlinkSubmitter
NimbusClient and Client replaced by FlinkClient
LocalCluster replaced by FlinkLocalCluster

In order to submit a Storm topology to Flink, it is sufficient to replace the used Storm classes with their Flink replacements in the Storm client code that assembles the topology. The actual runtime code, ie, Spouts and Bolts, can be uses unmodified. If a topology is executed in a remote cluster, parameters nimbus.host and nimbus.thrift.port are used as jobmanger.rpc.address and jobmanger.rpc.port, respectively. If a parameter is not specified, the value is taken from flink-conf.yaml.

How can I run a tuple generator spout? There is no source code in streamflow.spout.core.TupleGenerator, but core framework defines it in framework.yml.

Currently, error notifications for framework uploads and topology submissions do not provide contextual information as to the source of the error. This enhancement will improve the error notification system to provide detailed information about the problem so it can be resolved more quickly.

During framework upload the framework.yml file will be analyzed for correctness and ensure any required properties are specified. In addition, any class names referenced in the configuration will dynamically loaded to ensure all classes exists as defined and to catch a class loading error that will currently manifest during topology submission.

During topology submission, the streamflow engine will verify that all specified dependencies (e.g. frameworks, resources, files) are available and can be built into the final topology jar. In addition, all topology properties will be validated to ensure all required properties have been specified and configured properly.

In each of the above cases when an error is found, the notification will provide detailed information to correct the problem.

In rektide/streamflow-wiki@e3951ee6 I took the liberty of going through the POM for Streamflow and extracted project names and links that should serve as a viable start to the presently blank section of the wiki, https://github.com/lmco/streamflow/wiki/How-It-Works#key-technologies . Please consider merging the references I'm providing into your wiki.

Your wiki is very well skeletoned out, and I look forward to seeing it fleshed out from there; I'm expecting a ton of great, interesting content.

I've also filed isaacs/github#333 in hopes that maybe someday there'll be a better process for contributing to wikis than this weird filing an issue & linking you the commit to merge yourself.

let streamflow to support storm 1.0 and use storm-ui‘s api to get the cluster's info and the topology's metric

The application version is currently being displayed as ${project.version} in the UI. This is most likely due to a resources filter not being properly configured in the Maven pom.

The existing Shiro authentication mechanism should be improved with the following features to enhance security:

• Lock user account after providing an incorrect password a specific number of times in a given time window.
• Allow custom realm implementations to provide a custom dao implementation to create/edit/retrieve user accounts. This is necessary to support using custom implementations which store user data such as LDAP and Active Directory.
• Add support for user account pictures in the user service

The JDBC datastore does persist framework JARs as Blobs, but they are not being fetched from the database during topology submission. This causes issues if you clear out the $STREAMFLOW_HOME/temp directory and attempt to redeploy a topology. This issue also seems to be affecting framework icons.

When user accounts are deleted, user created entities such as topologies and resource entries are not deleted. To prevent the creation of zombie entities, these entities should be deleted when user accounts are deleted.

When we create a topology and build the topology. then we want to move this topology to another computer which install the streamflow. We want to export the structure of this topology and import this topology in another streamflow. I suggest it has these function

The current implementation of the topology kill feature in the UI simply kills the topology using a background request. Due to time it may take a topology to be fully killed, a kill dialog should be displayed allowing the user to specify the kill wait time in seconds. Once the topology kill command has completed, the dialog should update with the status.

We should upgrade to the latest Storm 0.9.3 version

Additionally, this would lend itself to upgrading to Kafka 0.8.1.1

Add support for paging and text search for all pages in the user interface which provide a list of results. This change will better support long lists and allow for search when the lists get too long to manage. The paged results should be sortable by name or last modified date.

The StreamFlow UI needs to be updated to include additional properties added in Storm 0.9.5 to the topology properties auto-complete list.

The topology connectors implemented in jsPlumb currently do not use LESS styling. This results in changes to the theme not being reflected in the connectors of the topology editor. This is the only section of code not changed when updating the Bootswatch theme and should be changed to allow for easier theme modifications and updates.

Javadoc comments need to be developed for all API worthy classes. REST documentation should be included in this Javadoc documentation to produce REST API documentation.

The file upload feature does not currently have a method to delete previously uploaded files. Rather than support inline file upload, the user should have access to a file manager user interface which allows for the creation of folders and files. The file manager should be available as a separate tab in the user interface and files should be accessible from the properties dialog.

This is a very nice project. Thank you guys !

I am thinking of integrating Spark Streaming and Flink in to StreamFlow. So that the end user can choose the streaming framework of their choice from StreamFlow.

It would be of great help if you can give some pointers on which all are the classes/packages we need to expand or consider to accomplish above.

Thank You

Currently, when a topology is submitted by a user from streamflow there are no protections that it will continue to run if there is a cluster outage.

This feature should implement a persistent service within the streamflow server that ensures that a topology that has been submitted will be resubmitted if the topology goes down for any reason. This will improve topology uptime in the event that a storm cluster goes down and comes back up for any reason.

Once a user triggers the kill operation in streamflow, the topology will be flagged as killed and should no longer be auto deployed.

The streamflow configuration should allow for enabling/disabling of this feature and controlling the polling interval of the service. A suggested config format is as follows:

monitor:
    enabled: true
    pollingInterval: 60

Allow users to provide javascript implementations and HTML implementations of custom property types. This will help users inject new property types when the provided set of properties do not cover their needs. The custom types should be registered in the framework.yml file which will allow the UI to reference the provided implementations.

This enhancement will require refactoring of the existing property system to be more modular and utilize this system to reimplement the existing property types. Changes will be required in the framework service to support parsing and validation of this new capability so it can be registered in the UI.

Currently the only way to register new frameworks is through the creation of a framework.yml file. It would be best if the user could also define frameworks through the use of custom annotations in the Spout or Bolt implementations. During framework upload, the framework service could scan for new Spouts and Bolts using these annotations and register them in the UI. This would simplify the process to register new Spouts and Bolts and reduce the chances of misconfiguring the framework.yml file.

This would require developers to import a custom streamflow library containing all available annotations. Changes would also be required in the framework service to properly parse the framework annotations.

Update the default component icons so they are higher resolution.

Currently working on this and hopefully this will be the catalyst for this to be supported.

Cheers,
Montana

Can we upgrade to StreamFlow JDK 1.7?

This is feasible,

We'd need to create a Kafka table using DDL. To try this run:

docker-compose exec kafka bash -c 'kafka-console-consumer.sh --topic user_behavior --bootstrap-server kafka:9094 --from-beginning --max-messages 10

From there you'll see a boot screen, then finally the dashboard:

See screenshot. Maybe a version issue.

P.S. All my respect to a really good and professional made project. Wiki is one of the best I ever saw. You have to be paid twice.

Upgrade the storm-core references to utilize version 0.9.5. Associated Kafka dependencies should be updated as well to match that of Storm 0.9.5

It's possible to use the following code to create a list:

[i+1 for i in(0,1,2)]
[1, 2, 3]

Can a similar thing be done with tuples in Streamflow?

(i+1 for i in(0,1,2)),
(<generator object <genexpr> at 0x03A53CF0>,)

Thanks Lockheed,
Montana Mendy

Support download of compiled topologies which combines framework jars, resources, and topology configuration data. This topology download should produce a single jar and should be executable outside of StreamFlow using the Storm binary executables

Conversely, upload of topologies should allow for upload of non-StreamFlow topologies which do not allow modification in the topology editor. These native Storm topologies can only be submitted or killed in the StreamFlow dashboard. StreamFlow topology jars which are uploaded will have the option of installing inbuilt dependencies such as framework jars and resources. Users should be prompted to replace each of the dependencies to ensure more up to date entities are not replaced.

The existing topology metrics capabilities have been refactored out of the user interface due to changes in the storm engine. This feature should be modified and retested to ensure all topology metrics are reported in the user interface. Due to the increased Storm cluster integration, native Storm metrics should also be retrieved using the new JSON API.

The web interface should be updated along with the server code to display useful metrics to the user.

Hi

Thanks a lot for this beautiful app for learning.

I dont think this was a bug. please find below problem which i am facing,

i can able to build in windows machine but unable to launch bat file(opening cmd prompt and closing immediately ).

as per the instructions

For Windows Systems

copy streamflow-{VERSION}.zip /opt
cd /opt
// Unzip using Windows explorer or using your installed compression library

but in windows we doesn't have any /opt like linux.

Thanks in advance.

Additional unit and integration tests are required for the streamflow-engine module to increase code coverage of the server code. This task should also ensure that all existing test cases are up to date and properly test each critical piece of code.

AngularJS test cases should be implemented to verify controller logic in the web code.

When the browser window is too small, it is possible for the toolbar in the StreamFlow UI to span multiple lines. When this occurs, the toolbar offset is not recalculated causing some sections of the page to be hidden by the toolbar. The toolbar should take into account its actual height and adjust the offset of pages accordingly.

Currently, StreamFlow requires the definition of a framework.yml file to register new components in the framework.

It would allow for much quicker development to also include the ability to define the component settings such as the name/label/description at the class level using custom annotations. In addition, properties could also be annotated used field annotations on the properties directly.

If possible, it would be good to support the annotation approach AND the framework.yml approach in case user's have a specific preference or are working with legacy code.

The topology editor currently requires clicking on each component in the canvas in order to delete or edit it. The click event rather than the hover event was used to display the component icons simply to support mobile devices which do not trigger hover events on elements.

It would be preferable to allow non-mobile browsers to bind to both hover and click events to make selection of the options more intuitive. Mobile browsers will only support the click event for selection of components. These changes will require binding the event listeners based on the browser type in the AngularJS code.

Fix the following Internet Explorer specific issues in the web interface.

• IE8 Dragging elements highlights text as it drags (fixed this is streams)
• IE8 Parallelism box is taking style from "input" instead of btn class
• IE8 Component accordion showing properly. Group button does not change accordion
• IE8 Ignore changes is very slow when leaving topology editor
• IE8/IE9 Framework upload is not completing or showing progress
• IE8/IE9 Topology properties are not saving properly.
• IE8/IE9 Notification does not display at all
• IE8/IE9 Caching service responses which causes invalid list states

Dates are current displayed in typical date format (e.g. July 4, 2014 08:00 EST). It would be preferable to display dates in a more natural format using moment.js (e.g. 5 minutes ago, a moment ago).

This will require use of an existing open source AngularJS directive or development of a new directive to support use of moment.js date formats.