pivotal-open-source-hub / stockinference-spark Goto Github PK

View Code? Open in Web Editor NEW

380.0 77.0 201.0 63.14 MB

Stock inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.

Home Page: http://pivotal-open-source-hub.github.io/StockInference-Spark

License: Apache License 2.0

Shell 3.47% Groovy 0.98% Scala 22.59% R 10.75% Java 60.69% CSS 1.53%

stockinference-spark's Introduction

Stock Prediction with Machine Learning

Summary

Stock Inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.

Requirements

Apache Geode (Incubating) or Pivotal GemFire
Spring XD 1.2+
Apache Spark 1.3.1
Apache Zeppelin (Incubating)
8GB+ RAM (recommended)
Linux or OSX (Windows should be OK but instructions assume *nix shell)
Pre-packaged VM for download: http://bit.ly/1F7uBxT (vagrant/vagrant)

PS: If you are were given a pre-packaged Vagrant VM for this lab you'll only need:

If you prefer to create a demo VM yourself, the instructions are here

First steps with each product

If you're not familiar with Geode/GemFire, Spring XD, Spark ML and Apache Zeppelin, please first follow the product specific labs below:

Starting the demo environment

If you have received a pre-built Virtual Machine, start the VM and access its console using ssh as below:

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'package.box' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Adding box 'package.box' (v0) for provider: virtualbox
    default: Downloading: file:///Users/fmelo/qcon/package.box
==> default: Successfully added box 'package.box' (v0) for 'virtualbox'!
(...)
    default: Running: inline script
==> default: stdin: is not a tty
==> default: QCon Rio 2015 - Pivotal Lab

$ vagrant ssh
Welcome to Ubuntu 15.04 (GNU/Linux 3.19.0-23-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Mon Aug 24 01:13:06 UTC 2015

  System load:  0.0                Processes:           96
  Usage of /:   17.6% of 38.81GB   Users logged in:     0
  Memory usage: 2%                 IP address for eth0: 10.0.2.15
  Swap usage:   0%                 IP address for eth1: 192.168.56.10

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

47 packages can be updated.
27 updates are security updates.


Last login: Sun Aug 23 23:16:30 2015 from 10.0.2.2
_____________________________________________________________________________

Lab variables:
GEODE_HOME=/home/vagrant/incubator-geode/gemfire-assembly/build/install/apache-geode
SPRINGXD_HOME=/home/vagrant/spring-xd-2.0.0.BUILD-SNAPSHOT
ZEPPELIN_HOME=/home/vagrant/incubator-zeppelin
SPARK_HOME=/home/vagrant/spark-1.3.1-bin-hadoop2.6
PROJECT=/home/vagrant/project/StockInference-Spark

_____________________________________________________________________________

[vagrant@stocks-vm, load: 0.00] (Mon Aug 24 - 01:13:10)
~ $

If you're not using the pre-built VM, just create host entries pointing to where your Geode/GemFire ("geode-server") and Spring XD ("xd-server") servers will be available. For example, configuring the demo to run all servers at my localhost:

$ more /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost geode-server xd-server
255.255.255.255 broadcasthost
::1             localhost

If you are using the VM, modify the /etc/hosts file to associate the server names with the external IP your VM is presenting. It will be the address that begins with 192.168.x.x

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:6a:32:e8
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29266 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25649 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10570362 (10.5 MB)  TX bytes:2911832 (2.9 MB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:ae:57:0d
          inet addr:192.168.56.10  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:585718 (585.7 KB)  TX bytes:10272756 (10.2 MB)

The /etc/hosts file changes:

$ more /etc/hosts
#127.0.0.1 localhost stocks-vm geode-server xd-server
127.0.0.1 localhost
192.168.56.10 stocks-vm geode-server xd-server

Also, if you are running within the VM and would like to use an external client you need to modify the setup.gfsh file for that configuration by removing the bind-address entries, and changing the hostname-for-clients address to the external IP of your VM.

$ more ~/project/StockInference-Spark/data/setup.gfsh
# GemFire/Geode setup

start locator --name=locator1 --J=-Dgemfire.http-service-port=7575
start server --name=server1 --J=-Dgemfire.start-dev-rest-api=true --J=-Dgemfire.http-service-port=8888 --hostname-for-cl
ients=192.168.56.10 --locators=geode-server[10334]

connect --locator=geode-server[10334]

create region --name=/Stocks --type=PARTITION
create region --name=/TechIndicators --type=PARTITION
create region --name=/Predictions --type=PARTITION

import data --region=/Stocks --file=../Stocks.gfd --member=server1
import data --region=/TechIndicators --file=../TechIndicators.gfd --member=server1

describe region --name=/Stocks
describe region --name=/TechIndicators
describe region --name=/Predictions

you'll also need to export the GEODE_HOME variable, pointing to your Apache Geode installation

$ export GEODE_HOME=/home/pivotal/apache-geode

To verify if you have the right directory, you should be able to launch gfsh by typing $GEODE_HOME/bin/gfsh

$ $GEODE_HOME/bin/gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  /
 / /__/ / ____/  _____/ / /    / /
/______/_/      /______/_/    /_/    v1.0.0-incubating-SNAPSHOT

Monitor and Manage GemFire
gfsh>exit
Exiting...
$

Creating the Geode / GemFire regions

The demo uses three different regions for storing data:

/Stocks - Stores raw stock trading data, as acquired from Yahoo Finance YQL or using the simulator (which randomly replays data previously ingested from the same Yahoo Finance YQL)
/TechIndicators - Stores technical indicators, which will be used as inputs / features to the Machine Learning model. The indicators are calculated by the R script, which is involked by Spring XD.
/Predictions - Stores the predicted data, as it gets calculated by the Spark MLLib model.

To create the regions, execute the script startGeode.sh under the data folder of this project root path as below:

$ cd project/StockInference-Spark/data
$ ./startGeode.sh

1. Executing - start locator --name=locator1 --J=-Dgemfire.http-service-port=7575

............................................
Locator in /home/vagrant/project/StockInference-Spark/data/locator1 on 192.168.56.10[10334] as locator1 is currently online.
Process ID: 4374
Uptime: 23 seconds
GemFire Version: 1.0.0-incubating-SNAPSHOT
Java Version: 1.8.0_51
Log File: /home/vagrant/project/StockInference-Spark/data/locator1/locator1.log
JVM Arguments: -Dgemfire.enable-cluster-configuration=true -Dgemfire.load-cluster-configuration-from-dir=false -Dgemfire.http-service-port=7575 -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
Class-Path: /home/vagrant/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-1.0.0-incubating-SNAPSHOT.jar:/home/vagrant/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-dependencies.jar

Successfully connected to: [host=192.168.56.10, port=1099]

Cluster configuration service is up and running.

2. Executing - start server --name=server1 --J=-Dgemfire.start-dev-rest-api=true --J=-Dgemfire.http-service-port=8888

..............
Server in /home/vagrant/project/StockInference-Spark/data/server1 on 192.168.56.10[40404] as server1 is currently online.
Process ID: 4546
Uptime: 7 seconds
GemFire Version: 1.0.0-incubating-SNAPSHOT
Java Version: 1.8.0_51
Log File: /home/vagrant/project/StockInference-Spark/data/server1/server1.log
JVM Arguments: -Dgemfire.default.locators=192.168.56.10[10334] -Dgemfire.use-cluster-configuration=true -Dgemfire.start-dev-rest-api=true -Dgemfire.http-service-port=8888 -XX:OnOutOfMemoryError=kill -KILL %p -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
Class-Path: /home/vagrant/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-1.0.0-incubating-SNAPSHOT.jar:/home/vagrant/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-dependencies.jar

3. Executing - create region --name=/Stocks --type=PARTITION

Member  | Status
------- | -------------------------------------
server1 | Region "/Stocks" created on "server1"

4. Executing - create region --name=/TechIndicators --type=PARTITION

Member  | Status
------- | ---------------------------------------------
server1 | Region "/TechIndicators" created on "server1"

5. Executing - create region --name=/Predictions --type=PARTITION

Member  | Status
------- | ------------------------------------------
server1 | Region "/Predictions" created on "server1"

6. Executing - import data --region=/Stocks --file=../Stocks.gfd --member=server1

Data imported from file : /home/vagrant/project/StockInference-Spark/data/Stocks.gfd on host : 192.168.56.10 to region : /Stocks

7. Executing - import data --region=/TechIndicators --file=../TechIndicators.gfd --member=server1

Data imported from file : /home/vagrant/project/StockInference-Spark/data/TechIndicators.gfd on host : 192.168.56.10 to region : /TechIndicators

8. Executing - describe region --name=/Stocks

..........................................................
Name            : Stocks
Data Policy     : partition
Hosting Members : server1

Non-Default Attributes Shared By Hosting Members

 Type  |    Name     | Value
------ | ----------- | ---------
Region | size        | 41540
       | data-policy | PARTITION


9. Executing - describe region --name=/TechIndicators

..........................................................
Name            : TechIndicators
Data Policy     : partition
Hosting Members : server1

Non-Default Attributes Shared By Hosting Members

 Type  |    Name     | Value
------ | ----------- | ---------
Region | size        | 36486
       | data-policy | PARTITION


10. Executing - describe region --name=/Predictions

..........................................................
Name            : Predictions
Data Policy     : partition
Hosting Members : server1

Non-Default Attributes Shared By Hosting Members

 Type  |    Name     | Value
------ | ----------- | ---------
Region | size        | 0
       | data-policy | PARTITION

As you can verify, the script not only created the regions but also imported some data we had captured earlier, in order to train our model. We'll keep re-training it later as we ingest data.

Next, we should deploy the Geode/GemFire functions for the Spark Connector:

~/project/StockInference-Spark/data $ ./deployFunctionVM.sh

(1) Executing - connect

Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=192.168.56.10, port=1099] ..
Successfully connected to: [host=192.168.56.10, port=1099]


(2) Executing - deploy --jar=/home/vagrant/incubator-geode/gemfire-spark-connector/gemfire-functions/target/scala-2.10/gemfire-functions_2.10-0.5.0.jar

Member  |           Deployed JAR           | Deployed JAR Location
------- | -------------------------------- | ------------------------------------------------------------------------------------------------
server1 | gemfire-functions_2.10-0.5.0.jar | /home/vagrant/project/StockInference-Spark/data/server1/vf.gf#gemfire-functions_2.10-0.5.0.jar#1

Training the Machine Learning model

Before executing the Machine Learning model on Spark, we need to train it with existing data. For doing that, run the script below, which involkes the scala class

From the streaming folder, run train.sh . It will take a while for it to run.

~/project/StockInference-Spark $ cd streaming

~/project/StockInference-Spark/streaming $ ./train.sh
15/08/24 03:07:41 INFO SparkContext: Running Spark version 1.3.1
15/08/24 03:07:41 WARN Utils: Your hostname, stocks-vm resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
(...)
15/08/24 03:09:33 INFO DAGScheduler: Stage 2012 (mean at StockInferenceDemo.scala:127) finished in 0.072 s
15/08/24 03:09:33 INFO DAGScheduler: Job 2012 finished: mean at StockInferenceDemo.scala:127, took 0.084467 s
training Mean Squared Error = 4.356799628392633E-4
[info 2015/08/24 03:09:33.493 UTC <Distributed system shutdown hook> tid=0x40] VM is exiting - shutting down distributed system

[info 2015/08/24 03:09:33.498 UTC <Distributed system shutdown hook> tid=0x40] GemFireCache[id = 1756587746; isClosing = true; isShutDownAll = false; created = Mon Aug 24 03:07:48 UTC 2015; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60]: Now closing.

[info 2015/08/24 03:09:33.555 UTC <Distributed system shutdown hook> tid=0x40] Destroying connection pool DEFAULT

Creating the Spring XD streams

Now we'll create the streams in SpringXD that orchestrate all the data flow.

Let's first start Spring XD. From the streaming directory, run the script startSpringXD.sh. This will start the server in background, outputing the logs to the file nohup.out

~/project/StockInference-Spark/streaming $ ./startSpringXD.sh
Starting...
nohup: appending output to ‘nohup.out’

Next, take a look at the file stream-create.xd, containing the streams we'll create on Spring XD. Remember the architecture image

~/project/StockInference-Spark/streaming $ more stream-create.xd
admin config server http://localhost:9393

stream create process_sink --definition "queue:stocks >  transform --script='file:./transform.groovy' | object-to-json | gemfire-json-server --useLocator=true --host=localhost --port
=10334 --regionName=Stocks --keyExpression=payload.getField('entryTimestamp')" --deploy

stream create yahoo_finance_source --definition "trigger --cron='* * 7-13 * * MON-FRI' | http-client --url='''https://query.yahooapis.com/v1/public/yql?q=select Symbol, LastTradeDate
, LastTradeTime, LastTradePriceOnly, DaysHigh, DaysLow, Open from yahoo.finance.quotes where symbol in (\"TSLA\")&format=json&env=store://datatables.org/alltableswithkeys''' --httpMe
thod=GET | splitter --expression=#jsonPath(payload,'$.query.results.quote')  > queue:stocks" --deploy

stream create http_source --definition "http --port=9000 | splitter --expression=#jsonPath(payload,'$') > queue:stocks" --deploy

stream create --name r_process --definition "tap:stream:process_sink.transform > r-parsing: object-to-json | shell --command='Rscript ./tech_indicators.R' | formatting: splitter --ex
pression=#jsonPath(payload,'$') | filter --expression=#jsonPath(payload,'$.rsi').indexOf('NaN')==-1 | object-to-json | gemfire-json-server --useLocator=true --host=locator --port=103
34 --regionName=TechIndicators --keyExpression=payload.getField('entryTimestamp')" --deploy

stream create --name prediction --definition "tap:stream:r_process.object-to-json > shell --command='../evaluate.sh' | gemfire-json-server --regionName=Predictions --host=localhost -
-port=10334 --useLocator=true --keyExpression=payload.getField('entryTimestamp')" --deploy

# stream create --name training --definition "trigger --fixedDelay=300 | shell --command='../train.sh'" --deploy

Please note the yahoo_finance_source stream has a cron based on PDT timezone (7-13 MON-FRI = normal Wall Street operating times). If your environment has another timezone set, you'll need to adjust it accordingly.

Execute the script stream-create.sh to deploy all streams to Spring XD:

~/project/StockInference-Spark/streaming $ ./stream-create.sh
Aug 24, 2015 3:19:48 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Successfully targeted http://localhost:9393
Aug 24, 2015 3:19:53 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Created and deployed new stream 'process_sink'
Aug 24, 2015 3:19:54 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Created and deployed new stream 'yahoo_finance_source'
Aug 24, 2015 3:19:55 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Created and deployed new stream 'http_source'
Aug 24, 2015 3:20:05 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Created and deployed new stream 'r_process'
Aug 24, 2015 3:20:08 AM org.springframework.shell.core.AbstractShell handleExecutionResult
INFO: Created and deployed new stream 'prediction'

At this point, Spring XD is already querying Yahoo Finance for the latest quotes(at the normal market hours), and also listening for possible quotes at port 9000.

Using the Simulator

When you're not on US stock market open hours, we can use the simulator to randomly replay some of the old quotes stored on Geode / GemFire. As the order is random, we can expect a much higher volatility.

Start the simulator from the FinanceStreamSimulator folder:

~/project/StockInference-Spark/FinanceStreamSimulator $ ./startSimulator.sh
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:findMainClass
:run
  /$$$$$$$$ /$$            /$$$$$$   /$$                      /$$$$$$  /$$                         /$$             /$$
| $$_____/|__/           /$$__  $$ | $$                     /$$__  $$|__/                        | $$            | $$
| $$       /$$ /$$$$$$$ | $$  \__//$$$$$$    /$$$$$$       | $$  \__/ /$$ /$$$$$$/$$$$  /$$   /$$| $$  /$$$$$$  /$$$$$$    /$$$$$$   /$$$$$$
| $$$$$   | $$| $$__  $$|  $$$$$$|_  $$_/   /$$__  $$      |  $$$$$$ | $$| $$_  $$_  $$| $$  | $$| $$ |____  $$|_  $$_/   /$$__  $$ /$$__  $$
| $$__/   | $$| $$  \ $$ \____  $$ | $$    | $$  \__/       \____  $$| $$| $$ \ $$ \ $$| $$  | $$| $$  /$$$$$$$  | $$    | $$  \ $$| $$  \__/
| $$      | $$| $$  | $$ /$$  \ $$ | $$ /$$| $$             /$$  \ $$| $$| $$ | $$ | $$| $$  | $$| $$ /$$__  $$  | $$ /$$| $$  | $$| $$
| $$      | $$| $$  | $$|  $$$$$$/ |  $$$$/| $$            |  $$$$$$/| $$| $$ | $$ | $$|  $$$$$$/| $$|  $$$$$$$  |  $$$$/|  $$$$$$/| $$
|__/      |__/|__/  |__/ \______/   \___/  |__/             \______/ |__/|__/ |__/ |__/ \______/ |__/ \_______/   \___/   \______/ |__/


Finance Stream Simulator

Usage:  java -jar FinanceStreamSimulator.jar [--serverUrl=<URL>] [--numberOfMessages=<messages>] [--basePrice=<price>] [--scale=<scale>]


2015-08-24 04:16:38.055  INFO 9954 --- [           main] io.pivotal.demo.SimulatorApp             : Starting SimulatorApp on stocks-vm with PID 9954 (/home/vagrant/project/StockInference-Spark/FinanceStreamSimulator/build/classes/main started by vagrant in /home/vagrant/project/StockInference-Spark/FinanceStreamSimulator)
2015-08-24 04:16:38.096  INFO 9954 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@51e2adc7: startup date [Mon Aug 24 04:16:38 UTC 2015]; root of context hierarchy
2015-08-24 04:16:38.847  INFO 9954 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2015-08-24 04:16:38.852  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : --------------------------------------
2015-08-24 04:16:38.853  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : >>> Geode rest endpoint: http://localhost:8888
2015-08-24 04:16:38.853  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : >>> Endpoint URL: http://localhost:9000
2015-08-24 04:16:38.853  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : >>> Number of messages: 500
2015-08-24 04:16:38.854  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : >>> Symbol: TSLA
2015-08-24 04:16:38.854  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : --------------------------------------
2015-08-24 04:16:39.240  INFO 9954 --- [           main] io.pivotal.demo.ReplaySimulator          : >>> Posting 500 messages 2015-08-24 04:21:20.768  INFO 10187 --- [           main] io.pivotal.demo.ReplaySimulator          : done
2015-08-24 04:21:20.769  INFO 10187 --- [           main] io.pivotal.demo.SimulatorApp             : Started SimulatorApp in 154.355 seconds (JVM running for 154.685)
2015-08-24 04:21:20.772  INFO 10187 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@51e2adc7: startup date [Mon Aug 24 04:18:46 UTC 2015]; root of context hierarchy
2015-08-24 04:21:20.773  INFO 10187 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

BUILD SUCCESSFUL

Total time: 2 mins 38.003 secs

Verifying the calculated predictions

Use GFSH to connect to Geode/GemFire and verify the Predictions region:

$ gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  /
 / /__/ / ____/  _____/ / /    / /
/______/_/      /______/_/    /_/    v1.0.0-incubating-SNAPSHOT

Monitor and Manage GemFire
gfsh> connect --locator=192.168.56.10[10334]
Connecting to Locator at [host=192.168.56.10, port=10334] ..
Connecting to Manager at [host=192.168.56.10, port=1099] ..
Successfully connected to: [host=192.168.56.10, port=1099]

gfsh>describe region --name=/Predictions
..........................................................
Name            : Predictions
Data Policy     : partition
Hosting Members : server1

Non-Default Attributes Shared By Hosting Members

 Type  |    Name     | Value
------ | ----------- | ---------
Region | size        | 500
       | data-policy | PARTITION

If the Value shows as 0, there was an issue creating the stream. To fix: Issue a destory and create to recreate the streams.

~/project/StockInference-Spark/streaming $ ./stream-destroy.sh
~/project/StockInference-Spark/streaming $ ./stream-create.sh

Now check some of the data:

gfsh>query --query="select entryTimestamp, ema, predicted from /Predictions order by entryTimestamp desc"

Result     : true
startCount : 0
endCount   : 20
Rows       : 548

entryTimestamp |       ema        | predicted
-------------- | ---------------- | ------------------
13353015247329 | 243.655200392179 | 244.36126012453593
13352705066802 | 243.304022701552 | 244.4679070450193
13352402037135 | 242.724916646386 | 243.45581253816388
13352098169940 | 242.36154257993  | 243.70099081921597
13351795573541 | 241.695218711637 | 238.62424186981085
13351492002111 | 243.225267310611 | 244.00046225381527
13351189015085 | 242.839771151012 | 242.8768374516088
13350884494287 | 242.82194251399  | 240.2610870520309
13350573308646 | 244.09792973193  | 246.75466201448913
13350268197871 | 242.775025233505 | 240.11608047958853
13349964892135 | 244.100586370887 | 244.7387182131781
13349658440285 | 243.782938889622 | 243.34998938877558
13349354287691 | 243.999147524282 | 243.42056117407438
13349043851852 | 244.287846987753 | 245.56848677338888
13348734159759 | 243.650035229633 | 243.8471030722387
13348413390251 | 243.552265288641 | 244.84555218518653
13348104692176 | 242.908324243737 | 244.77470961584862
13347800579225 | 241.979062973214 | 238.83273528756686
13347496878110 | 243.547743641872 | 243.66095598690484
13347193662872 | 243.491708904    | 242.94782444029957
13346886510828 | 243.763199762867 | 243.0980329260479
(...)

You can check how accurate the predicted value is by comparing the ema and predicted columns

EXTRA: Checking the predictions in real-time using the GUI

ps: If you're using the provided VM, you'll need to clone this repository in your local environment to use the Java Rich UI.

From the directory JavaFXClient, edit the file src/main/resources/client.xml to point the GemFire/Geode native client to where you're running the servers. If you're using the provided VM, this should be the VM IP (192.168.56.10).. otherwise, if you're running it local just use localhost.

JavaFXClient $ more src/main/resources/client.xml
<?xml version="1.0"?>
<!DOCTYPE client-cache PUBLIC
        "-//GemStone Systems, Inc.//GemFire Declarative Caching 8.0//EN"
        "http://www.gemstone.com/dtd/cache8_0.dtd">

<client-cache>
    <pool name="client" subscription-enabled="true">
        <locator host="192.168.56.10" port="10334" /> <!-- CHANGE THE IP ON THIS LINE TO POINT TO GEODE/GEMFIRE -->
    </pool>

    <region name="Stocks">
        <region-attributes refid="PROXY">
            <cache-listener>
                <class-name>io.pivotal.demo.StockListener</class-name>
            </cache-listener>
        </region-attributes>
    </region>

    <region name="Predictions">
        <region-attributes refid="PROXY">
            <cache-listener>
                <class-name>io.pivotal.demo.PredictionListener</class-name>
            </cache-listener>
        </region-attributes>
    </region>
</client-cache>

Start the GUI:

JavaFXClient $ ./gradlew run
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:runfx
(...)

This should open the Java based UI. Now run the simulator again, and watch the predicted and real values being plotted in real-time!

stockinference-spark's People

Contributors

Stargazers

Watchers

Forkers

sufyaan-kazi cloudstrack ebornier-pivotal kubeages madirey todaylxp vincent-poon tormoalto kali786516 tempbottle mt0803 agilemobiledev likeucode kiapper stevenmoks muhaozhang bitisony huyyeegz riskyhe309 justintung wangxiong2015 win2cs mckelvin hx168 yonglehou lenny-r yipchiuhung fdoperezi bikash tengfei1010 bryanyang0528 jiunnguo julianzhang kknet quedexco hsavit1 zmoon111 fubuki copyfun dodgerv petrex easonlv scott0228 qdrk jiayong nissimnabar juanlp guanlongtianzi slimmtl hyperji raghparihar 272029252 tomzhang miguelperalvo aluenkinglee genliu777 xghrbc1001 songoku119 terastars darwinsys zhangxin23 ryanliang129 lukeloveme greatgreatgreg mack007liu nkhuyu garygyao vadimmalykh kenmsj jsonbao mharris021 weidiaoxiang aachyee tommy-kwon zhouyongtao dambor skrishnan602 kitcharoenp gerryqi mfcardenas hurricane1260 pelluru sushengyang ill-look-later quyetdd wangyihaier xiahome threadbuilder meoooh deepseafishing richon0276 kaidix rubythonode mysoniw nareshmiriyala csgwma hoonmokmoon anhmike westsource ramakrishna580

stockinference-spark's Issues

Cannot start GUI

That is very interesting project, and I would like to run GUI.
However, I couldn't start GUI on Mac OSX.
Does anyone have information to fix it ?

Following is command log with debug option.
I extract [ERROR] message, and a part of message is character corruption.
Also, progress was stopped when 75%, so I have interrupted this process.

Best regards.

$ ./gradlew run --debug
...
22:45:44.177 [ERROR] [system.err] Exception in thread "JavaFX Application Thread" Exception in thread "main" java.lang.reflect.InvocationTargetException
22:45:44.178 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
22:45:44.178 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
22:45:44.178 [ERROR] [system.err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
22:45:44.178 [ERROR] [system.err] at java.lang.reflect.Method.invoke(Method.java:497)
22:45:44.179 [ERROR] [system.err] at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
22:45:44.179 [ERROR] [system.err] Caused by: java.lang.NullPointerException
22:45:44.179 [ERROR] [system.err] at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:383)
22:45:44.179 [ERROR] [system.err] at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
22:45:44.179 [ERROR] [system.err] ... 5 more
22:45:44.179 [ERROR] [system.err] java.lang.ExceptionInInitializerError
22:45:44.179 [ERROR] [system.err] at java.lang.Class.forName0(Native Method)
22:45:44.180 [ERROR] [system.err] at java.lang.Class.forName(Class.java:348)
22:45:44.180 [ERROR] [system.err] at com.sun.javafx.application.LauncherImpl.lambda$launchApplicationWithArgs$153(LauncherImpl.java:352)
22:45:44.180 [ERROR] [system.err] at com.sun.javafx.application.LauncherImpl$$Lambda$46/1232367853.run(Unknown Source)
22:45:44.180 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$172(PlatformImpl.java:326)
22:45:44.180 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl$$Lambda$47/186276003.run(Unknown Source)
22:45:44.181 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl.lambda$null$170(PlatformImpl.java:295)
22:45:44.181 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl$$Lambda$49/1932841810.run(Unknown Source)
22:45:44.187 [ERROR] [system.err] at java.security.AccessController.doPrivileged(Native Method)
22:45:44.188 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl.lambda$runLater$171(PlatformImpl.java:294)
22:45:44.188 [ERROR] [system.err] at com.sun.javafx.application.PlatformImpl$$Lambda$48/237061348.run(Unknown Source)
22:45:44.188 [ERROR] [system.err] at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
22:45:44.190 [ERROR] [system.err] Caused by: com.gemstone.gemfire.cache.CacheXmlException: While reading Cache XML file:/Users/xxx/vagrant/stockvm/JavaFXClient/build/resources/main/client.xml. Fatal error while parsing XML, caused by org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 10; 讖溯��"http://apache.org/xml/features/disallow-doctype-decl"縺荊rue縺ｫ險ｭ螳壹＆繧後※縺��ｋ蝣ｴ蜷医�．OCTYPE縺ｯ謖��ｮ壹〒縺阪∪縺帙ｓ縲�
22:45:44.190 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.xmlcache.CacheXml.fatalError(CacheXml.java:889)
22:45:44.190 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser$DefaultHandlerDelegate.fatalError(CacheXmlParser.java:3810)
22:45:44.191 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:180)
22:45:44.191 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
22:45:44.191 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
22:45:44.191 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:919)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
22:45:44.192 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
22:45:44.193 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
22:45:44.193 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
22:45:44.193 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
22:45:44.193 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
22:45:44.193 [ERROR] [system.err] t javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
22:45:44.193 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.parse(CacheXmlParser.java:196)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4244)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1174)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.init(GemFireCacheImpl.java:1018)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:678)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:207)
22:45:44.194 [ERROR] [system.err] at com.gemstone.gemfire.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:162)
22:45:44.195 [ERROR] [system.err] at io.pivotal.demo.FinanceUI.(FinanceUI.java:69)
22:45:44.195 [ERROR] [system.err] ... 12 more
22:45:44.195 [ERROR] [system.err] Caused by: org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 10; 讖溯��"http://apache.org/xml/features/disallow-doctype-decl"縺荊rue縺ｫ險ｭ螳壹＆繧後※縺��ｋ蝣ｴ蜷医�．OCTYPE縺ｯ謖��ｮ壹〒縺阪∪縺帙ｓ縲�
22:45:44.195 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
22:45:44.195 [ERROR] [system.err] at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
22:45:44.195 [ERROR] [system.err] ... 34 more

22:46:07.7 [DEBUG] [org.gradle.process.internal.ExecHandleRunner] Abort requested. Destroying process: command '/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java'.
Building 75% > :runfx

Compiling Java Rich UI errors

Hi,

Nice Project, can't wait to see more documentation (specially around how to change stock market and analyse stocks).

I have built the VM using this repo, the VM is working well with test prediction data.

I am trying to compile the Java GUI on my mac but it is throwing errors.

Please help, I am not good at Java :(

Thanks in advance.
Mehmood

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

Gradle 2.6

Build time: 2015-08-10 13:15:06 UTC
Build number: none
Revision: 233bbf8e47c82f72cb898b3e0a96b85d0aad166e

Groovy: 2.3.10
Ant: Apache Ant(TM) version 1.9.3 compiled on December 23 2013
JVM: 1.8.0_60 (Oracle Corporation 25.60-b23)
OS: Mac OS X 10.10.5 x86_64

Lodhis-MacBook-Pro:JavaFXChart lodhi$ ./gradlew run

Error

23:35:17.991 [ERROR] [org.gradle.BuildExceptionReporter]
23:35:17.991 [ERROR] [org.gradle.BuildExceptionReporter] FAILURE: Build failed with an exception.
23:35:17.992 [ERROR] [org.gradle.BuildExceptionReporter]
23:35:17.992 [ERROR] [org.gradle.BuildExceptionReporter] * What went wrong:
23:35:17.992 [ERROR] [org.gradle.BuildExceptionReporter] Execution failed for task ':runfx'.
23:35:17.992 [ERROR] [org.gradle.BuildExceptionReporter] > Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1
23:35:17.993 [ERROR] [org.gradle.BuildExceptionReporter]
23:35:17.993 [ERROR] [org.gradle.BuildExceptionReporter] * Try:
23:35:17.993 [ERROR] [org.gradle.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.
23:35:17.994 [LIFECYCLE] [org.gradle.BuildResultLogger]
23:35:17.994 [LIFECYCLE] [org.gradle.BuildResultLogger] BUILD FAILED
23:35:17.995 [LIFECYCLE] [org.gradle.BuildResultLogger]
23:35:17.995 [LIFECYCLE] [org.gradle.BuildResultLogger] Total time: 11.347 secs

Can't download VM

Permission required please help??

Where I can find username/password to login to pre-build VM ?

I am at "stocks-vm login:" prompt now.
Searched around the files including readme but could not find.

Thanks in advance !
Bing

Java Rich UI

At the time all I have is available to me is a windows environment from which I am running the VM in. For what its worth, everything works beautifully up until I am trying to Invoke the JavaFX UI. I am pretty sure it boils down to a connection issue after attaching the debugger for a full trace. I had already noticed some differences in how VirtualBox was assigning IPs (mine was in the 10.0.X.X range), and I had to connect to the geode server using "connect --locator=geode-server[10334]" instead of going by IP. This isn't a bug so much as a windows limitation most likely, so i'm more asking for guidance along these lines.

A questtion: why the Pre-packaged VM can not be downloaded?

Pre-packaged VM for download: http://bit.ly/1F7uBxT

Any one has the same problem?

Can't Get any Prediction data?

Hi, all,
I just download and running the vm , but i can't get any data of predictions .

gfsh>connect
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=stocks-vm.localdomain, port=1099] ..
Successfully connected to: [host=stocks-vm.localdomain, port=1099]

gfsh>describe region --name=/Predictions
..........................................................
Name : Predictions
Data Policy : partition
Hosting Members : server1

Non-Default Attributes Shared By Hosting Members

Type	Name	Value
Region	size	0
	data-policy	PARTITION

is there something wrong i?

[info 2016/09/14 01:40:54.023 PDT tid=0x60] Defining: PdxType[
dsid=0typenum=11, name=__GEMFIRE_JSON, fields=[
Symbol:String:0:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=-1
LastTradeDate:String:1:1:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=1
LastTradeTime:String:2:2:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=2
tradeTimestamp:long:3:2:idx0(relativeOffset)=-42:idx1(vlfOffsetIndex)=-1
Open:double:4:2:idx0(relativeOffset)=-34:idx1(vlfOffsetIndex)=-1
LastTradePriceOnly:double:5:2:idx0(relativeOffset)=-26:idx1(vlfOffsetIndex)=-1
DaysHigh:double:6:2:idx0(relativeOffset)=-18:idx1(vlfOffsetIndex)=-1
DaysLow:short:7:2:idx0(relativeOffset)=-10:idx1(vlfOffsetIndex)=-1
entryTimestamp:long:8:2:idx0(relativeOffset)=-8:idx1(vlfOffsetIndex)=-1]]
2016-09-14T01:40:54-0700 2.0.0.SNAP ERROR SimpleAsyncTaskExecutor-1 process.ShellCommandProcessor - Error in order(historicalSet$entry
Timestamp) : argument 1 is not a vector
2016-09-14T01:40:54-0700 2.0.0.SNAP ERROR SimpleAsyncTaskExecutor-1 process.ShellCommandProcessor - Execution halted
2016-09-14T01:40:54-0700 2.0.0.SNAP WARN xd.localbus-1 dispatcher.BroadcastingDispatcher - Suppressing Exception since 'ignoreFailures
' is set to TRUE.
org.springframework.messaging.MessageHandlingException: ; nested exception is java.lang.RuntimeException: Stream closed between payloa
ds
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:9
6) ~[spring-integration-core-4.2.0.M2.jar:na]

run train.sh fail

the log:

15/12/29 14:25:52 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/29 14:25:52 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 172.26.120.12 instead (on interface eth0)
15/12/29 14:25:52 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/12/29 14:25:53 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

[info 2015/12/29 14:25:56.413 CST tid=0x1]

Copyright (C) 1997-2015 Pivotal Software, Inc. All rights reserved. This
product is protected by U.S. and international copyright and intellectual
property laws. Pivotal products are covered by one or more patents listed
at http://www.pivotal.io/patents. Pivotal is a registered trademark
of trademark of Pivotal Software, Inc. in the United States and/or other
jurisdictions. All other marks and names mentioned herein may be
trademarks of their respective companies.

Java version: 1.0.0-incubating-SNAPSHOT yy 122815 2015-12-28 19:30:20 +0800 javac 1.8.0_60
Native version: native code unavailable
Source revision: 1366ff2d4fcbf54bfad684e9ba9822db2a2b0ff5
Source repository: master
Running on: /172.16.88.1, 8 cpu(s), amd64 Linux 3.10.0-327.3.1.el7.x86_64
Process ID: 14136
User: yy
Current dir: /home/yy/StockInference-Spark/streaming
Home dir: /home/yy
Command Line Parameters:
-Xms1G
-Xmx1G
Class Path:
/opt/spark-1.5.2-bin-hadoop2.6/conf/
/opt/spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar
/opt/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar
/opt/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar
/opt/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
Library Path:
/usr/java/packages/lib/amd64
/usr/lib64
/lib64
/lib
/usr/lib
System Properties:
SPARK_SUBMIT = true
awt.toolkit = sun.awt.X11.XToolkit
file.encoding = UTF-8
file.encoding.pkg = sun.io
file.separator = /
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.awt.printerjob = sun.print.PSPrinterJob
java.class.version = 52.0
java.endorsed.dirs = /opt/jdk1.8.0_60/jre/lib/endorsed
java.ext.dirs = /opt/jdk1.8.0_60/jre/lib/ext:/usr/java/packages/lib/ext
java.home = /opt/jdk1.8.0_60/jre
java.io.tmpdir = /tmp
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_60-b27
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.8
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.8.0_60
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.8
java.vm.vendor = Oracle Corporation
java.vm.version = 25.60-b23
line.separator =

log4j.configurationFile = jar:file:/opt/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-1.0.0-incubating-SNAPSHOT.jar!/com/gemstone/gemfire/internal/logging/log4j/log4j2-default.xml
os.version = 3.10.0-327.3.1.el7.x86_64
p2p.useSSL = false
path.separator = :
spark.app.name = io.pivotal.demo.StockInferenceDemo
spark.driver.memory = 1G
spark.jars = file:/home/yy/StockInference-Spark/streaming/../lib/gemfire-spark-connector_2.10-0.5.0.jar,file:/opt/incubator-geode/gemfire-assembly/build/install/apache-geode/lib/gemfire-core-dependencies.jar,file:/home/yy/StockInference-Spark/streaming/../StockInference/build/libs/StockInference-1.0.jar
spark.master = local[*]
spark.submit.deployMode = client
sun.arch.data.model = 64
sun.boot.class.path = /opt/jdk1.8.0_60/jre/lib/resources.jar:/opt/jdk1.8.0_60/jre/lib/rt.jar:/opt/jdk1.8.0_60/jre/lib/sunrsasign.jar:/opt/jdk1.8.0_60/jre/lib/jsse.jar:/opt/jdk1.8.0_60/jre/lib/jce.jar:/opt/jdk1.8.0_60/jre/lib/charsets.jar:/opt/jdk1.8.0_60/jre/lib/jfr.jar:/opt/jdk1.8.0_60/jre/classes
sun.boot.library.path = /opt/jdk1.8.0_60/jre/lib/amd64
sun.cpu.endian = little
sun.cpu.isalist = 
sun.desktop = gnome
sun.io.unicode.encoding = UnicodeLittle
sun.java.command = org.apache.spark.deploy.SparkSubmit --master local[*] --conf spark.driver.memory=1G --class io.pivotal.demo.StockInferenceDemo --executor-memory 1G --jars ../lib/gemfire-spark-connector_2.10-0.5.0.jar,/opt/incubator-geode/gemfire-assembly/build/install/apache-geode//lib/gemfire-core-dependencies.jar ../StockInference/build/libs/StockInference-1.0.jar train
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = UTF-8
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.nio.ch.bugLevel = 
sun.os.patch.level = unknown
user.country = US
user.language = en
user.timezone = Asia/Shanghai

[info 2015/12/29 14:25:56.419 CST

tid=0x1] Running in local mode since mcast-port was 0 and locators was empty.

[info 2015/12/29 14:25:58.429 CST

tid=0x1] Pool DEFAULT started with multiuser-authentication=false

[info 2015/12/29 14:25:58.444 CST tid=0x56] Updating membership port. Port changed from 0 to 36,363.

15/12/29 14:25:59 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.NotSerializableException: com.gemstone.gemfire.cache.query.internal.StructImpl
Serialization stack:
- object not serializable (class: com.gemstone.gemfire.cache.query.internal.StructImpl, value: struct(entryTimestamp:257320529324726,close:243.79,ema:243.777629543721,future_ema:243.77987871759,rsi:55.4265163313406,ema_diff:0.0123704562788305,low_diff:-1.78999999999999,high_diff:10.77))
- element of array (index: 0)
- array (class [Ljava.lang.Object;, size 1)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/12/29 14:25:59 ERROR TaskSetManager: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: com.gemstone.gemfire.cache.query.internal.StructImpl
Serialization stack:
- object not serializable (class: com.gemstone.gemfire.cache.query.internal.StructImpl, value: struct(entryTimestamp:257320529324726,close:243.79,ema:243.777629543721,future_ema:243.77987871759,rsi:55.4265163313406,ema_diff:0.0123704562788305,low_diff:-1.78999999999999,high_diff:10.77))
- element of array (index: 0)
- array (class [Ljava.lang.Object;, size 1); not retrying
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: com.gemstone.gemfire.cache.query.internal.StructImpl
Serialization stack:
- object not serializable (class: com.gemstone.gemfire.cache.query.internal.StructImpl, value: struct(entryTimestamp:257320529324726,close:243.79,ema:243.777629543721,future_ema:243.77987871759,rsi:55.4265163313406,ema_diff:0.0123704562788305,low_diff:-1.78999999999999,high_diff:10.77))
- element of array (index: 0)
- array (class [Ljava.lang.Object;, size 1)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1302)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.RDD.take(RDD.scala:1276)
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1316)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.RDD.first(RDD.scala:1315)
at io.pivotal.gemfire.spark.connector.internal.oql.SchemaBuilder.toSparkSchema(SchemaBuilder.scala:30)
at io.pivotal.gemfire.spark.connector.internal.oql.OQLRelation.schema(RDDConverter.scala:13)
at org.apache.spark.sql.execution.datasources.LogicalRelation.(LogicalRelation.scala:37)
at org.apache.spark.sql.SQLContext.baseRelationToDataFrame(SQLContext.scala:395)
at io.pivotal.gemfire.spark.connector.GemFireSQLContextFunctions.gemfireOQL(GemFireSQLContextFunctions.scala:21)
at io.pivotal.demo.StockInferenceDemo$.train(StockInferenceDemo.scala:66)
at io.pivotal.demo.StockInferenceDemo$.main(StockInferenceDemo.scala:211)
at io.pivotal.demo.StockInferenceDemo.main(StockInferenceDemo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[info 2015/12/29 14:25:59.179 CST tid=0x4a] VM is exiting - shutting down distributed system

[info 2015/12/29 14:25:59.180 CST tid=0x4a] GemFireCache[id = 140778365; isClosing = true; isShutDownAll = false; created = Tue Dec 29 14:25:58 CST 2015; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60]: Now closing.

[info 2015/12/29 14:25:59.194 CST tid=0x4a] Resetting original MemoryPoolMXBean heap threshold bytes 0 on pool PS Old Gen

[info 2015/12/29 14:25:59.212 CST tid=0x4a] Destroying connection pool DEFAULT