Code Monkey home page Code Monkey logo

infrastructure-and-programming-models-for-stream-data-analysis's Introduction

Infrastructure and Programming Models for Stream Data Analysis

Infraestruturas e Modelos de Programação para Análise de Dados em Stream (or Infrastructure and Programming Models for Stream Data Analysis, with acronym IMPADS) is an thesis theme which the main objective is to evaluate, test and benchmark various Streaming Processing Data Systems.

Some Sistemas de Processamento de Dados em Stream (or Streaming Data Processing Systems, with acronym SPDS) that have significant popularities are:

For the limitation of time base on thesis delivery, the main SPDS that will be approached are Apache Storm, Apache Flink and Apache Kafka (with the use of Kafka Stream API).

The main objective of this projects are:

  • Evaluate their infrastructure needed
  • Evaluate their programming model
  • Propose an deployment process for infrastructure
  • Evaluate development process of an topology
  • Benchmark built topology for each choosen SPDS

infrastructure-and-programming-models-for-stream-data-analysis's People

Contributors

renovate-bot avatar rubenribgarcia avatar

Watchers

avatar

Forkers

yutingyao

infrastructure-and-programming-models-for-stream-data-analysis's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • Update slf4j monorepo to v1.7.36 (org.slf4j:slf4j-simple, org.slf4j:slf4j-api)
  • Update adoptopenjdk/openjdk11 Docker tag to x86_64-windows-jre-11.0.4_11
  • Update bitnami/kafka Docker tag
  • Update bitnami/redis Docker tag to v6.2.14
  • Update bitnami/zookeeper Docker tag to v3.9.2
  • Update dependency com.fasterxml.jackson.core:jackson-databind to v2.17.1
  • Update dependency com.fasterxml.jackson.core:jackson-databind to v2.17.1
  • Update dependency com.github.ben-manes.caffeine:caffeine to v2.9.3
  • Update dependency com.rabbitmq:amqp-client to v5.21.0
  • Update dependency io.lettuce:lettuce-core to v6.3.2.RELEASE
  • Update dependency org.apache.commons:commons-csv to v1.11.0
  • Update dependency org.apache.flink:flink-streaming-java_2.12 to v1.14.6
  • Update dependency org.apache.maven.plugins:maven-assembly-plugin to v3.7.1
  • Update dependency org.apache.maven.plugins:maven-clean-plugin to v3.3.2
  • Update dependency org.apache.maven.plugins:maven-compiler-plugin to v3.13.0
  • Update dependency org.apache.maven.plugins:maven-jar-plugin to v3.4.1
  • Update dependency org.apache.maven.plugins:maven-shade-plugin to v3.6.0
  • Update dependency org.geotools:gt-main to v24.7
  • Update dependency redis.clients:jedis to v3.10.0
  • Update flink Docker tag to v1.19.0
  • Update grafana/grafana Docker tag to v7.5.17
  • Update micrometer.version to v1.13.1 (io.micrometer:micrometer-registry-statsd, io.micrometer:micrometer-core)
  • Update obsidiandynamics/kafdrop Docker tag to v3.31.0
  • Update python Docker tag
  • Update rabbitmq Docker tag to v3.13.3
  • Update telegraf Docker tag to v1.31
  • 🔐 Create all rate-limited PRs at once 🔐

Warning

Renovate failed to look up the following dependencies: Failed to look up maven package org.isel.thesis.impads:giragen-data-adapters, Failed to look up maven package org.apache.maven.plugins:maven-resource-plugin, Failed to look up maven package org.isel.thesis.impads:spds-benchmark, Failed to look up maven package org.isel.thesis.impads.connectors:spds-connectors, Failed to look up maven package org.isel.thesis.impads:spds-common, Failed to look up maven package org.isel.thesis.impads:spds-flink, Failed to look up maven package org.isel.thesis.impads:spds-flink-rabbitmq-connector, Failed to look up maven package org.isel.thesis.impads.connectors:spds-redis-connector, Failed to look up maven package org.isel.thesis.impads:spds-kafka, Failed to look up maven package org.isel.thesis.impads:spds-storm.

Files affected: data-generator/giragen-generator/pom.xml, pom.xml, spds-benchmark/spds-common/pom.xml, spds-benchmark/spds-connectors/pom.xml, spds-benchmark/spds-connectors/spds-rabbitmq-connector/pom.xml, spds-benchmark/spds-connectors/spds-redis-connector/pom.xml, spds-benchmark/spds-flink/pom.xml, spds-benchmark/spds-flink/spds-flink-gira-topology/pom.xml, spds-benchmark/spds-flink/spds-flink-rabbitmq-connector/pom.xml, spds-benchmark/spds-kafka/pom.xml, spds-benchmark/spds-kafka/spds-kafka-rabbitmq-connector/pom.xml, spds-benchmark/spds-kafka/spds-kafka-redis-connector/pom.xml, spds-benchmark/spds-kafka/spds-kafka-stream-gira-topology/pom.xml, spds-benchmark/spds-storm/pom.xml, spds-benchmark/spds-storm/spds-storm-gira-topology/pom.xml, spds-benchmark/spds-storm/spds-storm-rabbitmq-connector/pom.xml

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

ansible
spds-infrastructure/ansible/roles/flink-infrastructure/job-manager/tasks/job-manager.yml
  • flink 1.11.1-scala_2.12-java11
spds-infrastructure/ansible/roles/flink-infrastructure/task-manager/tasks/task-manager.yml
  • flink 1.11.1-scala_2.12-java11
spds-infrastructure/ansible/roles/giragen/tasks/giragen.yml
  • impads/giragen-generator 0.0.1-SNAPSHOT
spds-infrastructure/ansible/roles/kafka-infrastructure/kafka-connect/tasks/kafka-connect.yml
  • impads/kafka-connect 2.7.0
  • impads/kafka-connect 2.7.0
  • impads/kafka-connect 2.7.0
  • impads/kafka-connect 2.7.0
spds-infrastructure/ansible/roles/kafka-infrastructure/kafka/tasks/kafka.yml
  • bitnami/zookeeper 3.6.1
  • bitnami/kafka 2.7.0
  • obsidiandynamics/kafdrop 3.27.0
spds-infrastructure/ansible/roles/metrics-monitor/agent/tasks/metrics-agent.yml
  • telegraf 1.17-alpine
spds-infrastructure/ansible/roles/metrics-monitor/dashboard/tasks/metrics-dashboard.yml
  • influxdb 1.8-alpine
  • grafana/grafana 7.1.5
spds-infrastructure/ansible/roles/misc-infrastructure/tasks/misc-infrastructure.yml
  • bitnami/redis 6.0.9
  • rabbitmq 3.8.9-management
spds-infrastructure/ansible/roles/storm-infrastructure/nimbus/tasks/nimbus.yml
  • bitnami/zookeeper 3.6.1
  • impads/storm 2.2.0
  • impads/storm 2.2.0
spds-infrastructure/ansible/roles/storm-infrastructure/supervisor/tasks/supervisor.yml
  • impads/storm 2.2.0
spds-infrastructure/ansible/roles/topology-submitter/kafka-stream-job-submitter/tasks/kafka-stream-job-submitter.yml
  • impads/kafka-streams-gira-travels-pattern 0.0.1
dockerfile
data-generator/giragen-generator/Dockerfile
  • adoptopenjdk/openjdk11 debianslim-jre
scripts/ipma-data-to-redis/Dockerfile
  • python 3.8.6-alpine
scripts/redis-list-consumer/Dockerfile
  • python 3.9.1-alpine
spds-benchmark/spds-kafka/spds-kafka-stream-gira-topology/Dockerfile
  • adoptopenjdk/openjdk11 debianslim-jre
spds-infrastructure/components/apache-kafka/kafka-connect/Dockerfile
  • bitnami/kafka 2.7.0
spds-infrastructure/components/apache-storm/Dockerfile
  • adoptopenjdk/openjdk11 debianslim-jre
maven
data-adapters/giragen-data-adapters/pom.xml
data-generator/giragen-generator/pom.xml
  • org.isel.thesis.impads:giragen-data-adapters 0.0.1-SNAPSHOT
pom.xml
  • com.google.dagger:dagger-compiler 2.26
  • com.rabbitmq:amqp-client 5.9.0
  • com.google.dagger:dagger 2.26
  • com.typesafe:config 1.4.1
  • com.fasterxml.jackson.core:jackson-databind 2.10.2
  • org.slf4j:slf4j-api 1.7.26
  • org.slf4j:slf4j-simple 1.7.26
  • org.apache.commons:commons-csv 1.8
  • io.vavr:vavr 0.10.4
  • io.micrometer:micrometer-core 1.6.2
  • com.google.guava:guava 29.0-jre
  • org.geotools:gt-main 24.1
  • io.micrometer:micrometer-registry-statsd 1.6.2
  • redis.clients:jedis 3.3.0
  • io.lettuce:lettuce-core 6.0.1.RELEASE
  • com.github.ben-manes.caffeine:caffeine 2.8.8
  • org.apache.maven.plugins:maven-clean-plugin 3.1.0
  • org.apache.maven.plugins:maven-jar-plugin 3.2.0
  • org.apache.maven.plugins:maven-resource-plugin 3.1.0
  • org.apache.maven.plugins:maven-compiler-plugin 3.8.1
  • org.apache.maven.plugins:maven-assembly-plugin 3.3.0
  • org.apache.maven.plugins:maven-shade-plugin 3.2.4
  • org.apache.flink:flink-streaming-java_2.12 1.11.1
  • org.apache.kafka:kafka-clients 2.7.0
  • org.apache.kafka:kafka-streams 2.7.0
  • org.apache.kafka:connect-api 2.7.0
  • org.apache.storm:storm 2.2.0
  • org.apache.storm:storm-client 2.2.0
  • org.apache.storm:storm-server 2.2.0
  • org.apache.storm:storm-metrics 2.2.0
spds-benchmark/pom.xml
spds-benchmark/spds-common/pom.xml
  • org.isel.thesis.impads:spds-benchmark 0.0.1-SNAPSHOT
  • javax.inject:javax.inject 1
spds-benchmark/spds-connectors/pom.xml
  • org.isel.thesis.impads:spds-benchmark 0.0.1-SNAPSHOT
spds-benchmark/spds-connectors/spds-rabbitmq-connector/pom.xml
  • org.isel.thesis.impads.connectors:spds-connectors 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-common 0.0.1-SNAPSHOT
spds-benchmark/spds-connectors/spds-redis-connector/pom.xml
  • org.isel.thesis.impads.connectors:spds-connectors 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-common 0.0.1-SNAPSHOT
spds-benchmark/spds-flink/pom.xml
  • org.isel.thesis.impads:spds-benchmark 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-common 0.0.1-SNAPSHOT
spds-benchmark/spds-flink/spds-flink-gira-topology/pom.xml
  • org.isel.thesis.impads:spds-flink 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-flink-rabbitmq-connector 0.0.1-SNAPSHOT
  • org.isel.thesis.impads.connectors:spds-redis-connector 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:giragen-data-adapters 0.0.1-SNAPSHOT
spds-benchmark/spds-flink/spds-flink-rabbitmq-connector/pom.xml
  • org.isel.thesis.impads:spds-flink 0.0.1-SNAPSHOT
spds-benchmark/spds-kafka/pom.xml
  • org.isel.thesis.impads:spds-benchmark 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-common 0.0.1-SNAPSHOT
spds-benchmark/spds-kafka/spds-kafka-rabbitmq-connector/pom.xml
  • org.isel.thesis.impads:spds-kafka 0.0.1-SNAPSHOT
spds-benchmark/spds-kafka/spds-kafka-redis-connector/pom.xml
  • org.isel.thesis.impads:spds-kafka 0.0.1-SNAPSHOT
  • org.isel.thesis.impads.connectors:spds-redis-connector 0.0.1-SNAPSHOT
spds-benchmark/spds-kafka/spds-kafka-stream-gira-topology/pom.xml
  • org.isel.thesis.impads:spds-kafka 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:giragen-data-adapters 0.0.1-SNAPSHOT
  • org.isel.thesis.impads.connectors:spds-redis-connector 0.0.1-SNAPSHOT
spds-benchmark/spds-storm/pom.xml
  • org.isel.thesis.impads:spds-benchmark 0.0.1-SNAPSHOT
  • org.isel.thesis.impads:spds-common 0.0.1-SNAPSHOT
spds-benchmark/spds-storm/spds-storm-gira-topology/pom.xml
  • org.isel.thesis.impads:spds-storm 0.0.1-SNAPSHOT
  • com.fasterxml.jackson.core:jackson-databind 2.9.8
  • org.isel.thesis.impads:giragen-data-adapters 0.0.1-SNAPSHOT
  • org.isel.thesis.impads.connectors:spds-redis-connector 0.0.1-SNAPSHOT
spds-benchmark/spds-storm/spds-storm-rabbitmq-connector/pom.xml
  • org.isel.thesis.impads:spds-storm 0.0.1-SNAPSHOT
pipenv
scripts/ipma-data-to-redis/Pipfile
scripts/redis-list-consumer/Pipfile
pyenv
spds-infrastructure/.python-version
  • python 3.9.5
terraform
spds-infrastructure/terraform/aws/all/main.tf
spds-infrastructure/terraform/aws/flink/main.tf
spds-infrastructure/terraform/aws/kafka/main.tf
spds-infrastructure/terraform/aws/storm/main.tf
  • Check this box to trigger a request for Renovate to run again on this repository

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Invalid schedule: 'Invalid schedule: "every 30 minutes" should not specify minutes'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.