geethaanne / creditcardtransactionmonitor Goto Github PK

This demo was created to show Streaming components usage in a Telecom company.

Author: Guilherme Braccialli

With special thanks to:

• Ali Bajwa who created original Twitter Demo that inspired this demo.
• <a href="https://github.com/randerzander"target="_blank">Randy Gelhausen who created Storm Commons component.

Check more details in this PDF presentation.

• Ingest:
  • Listen for CDR records coming as text file in a directory
• Processing:
  • Monitor dropped calls
  • Monitor networks type change
• Persistence:
  • Hive Streaming ORC (for interactive query)
  • HBase (for granular events alerts)
  • Solr/Banana (for alerts and reports/dashboards)
• Simulation:
  • Generate "fake" CDR files every minute
• Demo setup:
  • Start HDP 2.5 sandbox and run provided scripts to setup demo

1. Setup demo via scripts on vanilla HDP 2.5 sandbox
2. Run demo to monitor Tweets about S&P 500 securities in realtime
3. Observe results in HDFS, Hive, HBase/Phoenix, Solr/Banana

These setup steps are only needed first time and may take upto 30min to execute (depending on your internet connection)

• Download HDP 2.3.2 sandbox from Hortonworks website
• Import the Sandbox into VMWare or VirtualBox and allocate at least 4cpus and 10GB RAM and start the VM
• Connect to the VM via SSH (password hadoop). You can also SSH via browser by clicking: http://sandbox.hortonworks.com:4200

ssh -p 2222 root@localhost

• Download code as root user

cd
git clone https://github.com/gbraccialli/telco-cdr-monitoring.git	

• Setup demo:Run below to setup demo (one time): it will start HBase/Kafka/Storm, install solr, banana and create hive/phoenix tables.

chmod -R +x /root/telco-cdr-monitoring/
cd /root/telco-cdr-monitoring
mvn clean package
scripts/setup_demo.sh

ssh -p 2222 root@localhost

• Start all components:

cd /root/telco-cdr-monitoring
scripts/start_demo.sh

or

• Start individual compoments:
  • Flume

  cd /root/telco-cdr-monitoring
  scripts/start_flume.sh
  

  • Storm

  cd /root/telco-cdr-monitoring
  scripts/start_storm.sh
  

  • CDR Producer

  cd /root/telco-cdr-monitoring
  scripts/start_cdr_producer.sh
  

• To stop Demo, use:

cd /root/telco-cdr-monitoring
scripts/stop_demo.sh

• Dashboards

  • Dashboard Dropped Calls

  • Dashboard Network Type Change

• Storm Topology UI

  • Storm UI

    ![Image](https://github.com/gbraccialli/telco-cdr-monitoring/raw/master/screenshots/storm_ui_2.png)
    

• Hive/Phoenix tables

  • telco_cdr_monitoring_raw
  • telco_cdr_monitoring_phoenix_dropped_call
  • telco_cdr_monitoring_phoenix_network_type_change

• Log files

  • /root/telco-cdr-monitoring/logs/flume.log
  • /root/telco-cdr-monitoring/logs/producer.log
  • /root/telco-cdr-monitoring/logs/storm.log

• CDR Files

  • /root/telco-cdr-monitoring/data