Comments (12)
tovbinm
commented on May 18, 2024
Try this:
val outcome = FeatureBuilder.RealNN[Passenger].extract(
v => Option(v.outcome).toRealNN(throw new Exception("Outcome cannot be null"))
).asResponsefrom transmogrifai.
monk1337
commented on May 18, 2024
That issue have been solved , I am running iris model on iris dataset in helloworld , But it's taking a lot of time after splitting the data , It's stuck at this line :
18/09/12 17:29:25 INFO OpCrossValidation: Best set of parameters:
{
rfc_2531cfc708bc-impurity: gini,
rfc_2531cfc708bc-maxBins: 32,
rfc_2531cfc708bc-maxDepth: 3,
rfc_2531cfc708bc-minInfoGain: 0.001,
rfc_2531cfc708bc-minInstancesPerNode: 10,
rfc_2531cfc708bc-numTrees: 50,
rfc_2531cfc708bc-subsamplingRate: 1.0
} for RandomForestClassifier
18/09/12 17:29:25 INFO OpCrossValidation: Best train validation split metric: 0.02941176470588236.
<===========--> 91% EXECUTING [32m 11s]
> :sparkSubmit
is it training or code is stuck somewhere , If it is training then how long it will take normally, and is there any option where i can set epoch , printing result after each epoch something ?
from transmogrifai.
tovbinm
commented on May 18, 2024
Training usually takes about 1-5 minutes depending on how fast your hardware is (32 minutes is definitely too long). You can try browsing to Spark web UI and inspect where is it stuck at http://127.0.0.1:4040
from transmogrifai.
monk1337
commented on May 18, 2024
@tovbinm I am not able to access http://127.0.0.1:4040 , can you check iris code , I tried two times from scratch ( git clone and etc ) but same thing.
from transmogrifai.
tovbinm
commented on May 18, 2024
Spark web UI is only accessible during the execution of the spark job. So after you run the ./gradlew sparkSubmit jump to your browser and try to refresh the http://127.0.0.1:4040 several times.
from transmogrifai.
ajayborra
commented on May 18, 2024
@monk1337 Try creating helloworld/resources/log4.properties file and adding log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=INFO to it. This should help you to find out more details on what is happening during execution of the workflow.
from transmogrifai.
ajayborra
commented on May 18, 2024
Perhaps you can try adding log4j.rootLogger=INFO instead of log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=INFO to helloworld/resources/log4.properties file to get a more detailed log.
from transmogrifai.
monk1337
commented on May 18, 2024
@ajayborra I added log4j.rootLogger=INFO then i am able to see more detail from terminal , after printing the result program is stuck at same point , should i stop the process manually?
"thresholds" : [ 0.0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5, 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.0 ]
},
"(multiEval)_Precision" : 0.9741545893719807,
"(multiEval)_F1" : 0.9740338014501739,
"(multiEval)_Recall" : 0.9739130434782609
}
}
}
18/09/13 13:00:35 INFO SparkContext: Invoking stop() from shutdown hook
18/09/13 13:00:35 INFO OpSparkListener: APP_NAME:op-helloworld:com.salesforce.hw.iris.OpIris,RUN_TYPE:Train,APP:op-helloworld:com.salesforce.hw.iris.OpIris,APP_ID:local-1536823760622,APP_TIME_MS:72583
18/09/13 13:00:35 INFO SparkUI: Stopped Spark web UI at http://monks-air:4040
18/09/13 13:00:35 INFO OpWorkflowRunner: Total run time: 1m12.583s
<===========--> 91% EXECUTING [4m 31s]
> :sparkSubmit
I also wanted to know how to use model for prediction for new data ? Is there any demo which shows how i can use trained model for new data? or for a single row of csv ?
from transmogrifai.
monk1337
commented on May 18, 2024
If i am using score or evaluate command then , I am not able to see any score or validation accuracy , It is stuck at this point :
c/userFiles-876763f8-79c7-4e9f-9bb3-dbcf71f17ada/scopt_2.11-3.5.0.jar to class loader
18/09/13 13:09:51 INFO Executor: Fetching spark://monks-air:53638/jars/carrier-1.72.jar with timestamp 1536824388036
18/09/13 13:09:51 INFO Utils: Fetching spark://monks-air:53638/jars/carrier-1.72.jar to /private/var/folders/3_/q9nh4dx91kq8j618fk9p5tmw0000gn/T/spark-e89b6682-f8a1-4ce3-ac76-c4d91939ffbc/userFiles-876763f8-79c7-4e9f-9bb3-dbcf71f17ada/fetchFileTemp3717913299294748096.tmp
18/09/13 13:09:51 INFO Executor: Adding file:/private/var/folders/3_/q9nh4dx91kq8j618fk9p5tmw0000gn/T/spark-e89b6682-f8a1-4ce3-ac76-c4d91939ffbc/userFiles-876763f8-79c7-4e9f-9bb3-dbcf71f17ada/carrier-1.72.jar to class loader
18/09/13 13:09:51 INFO HadoopRDD: Input split: file:/tmp/iris-model/op-model.json/part-00000:0+55653
18/09/13 13:09:51 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 57116 bytes result sent to driver
18/09/13 13:09:51 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1516 ms on localhost (executor driver) (1/1)
18/09/13 13:09:51 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
18/09/13 13:09:51 INFO DAGScheduler: ResultStage 0 (collect at OpWorkflowModelReader.scala:59) finished in 1.633 s
18/09/13 13:09:51 INFO OpSparkListener: APP_NAME:op-helloworld:com.salesforce.hw.iris.OpIris,RUN_TYPE:Evaluate,APP:op-helloworld:com.salesforce.hw.iris.OpIris,APP_ID:local-1536824388139,JOB_ID:0,RESULT:JobSucceeded,JOB_TIME_MS:1690
18/09/13 13:09:51 INFO DAGScheduler: Job 0 finished: collect at OpWorkflowModelReader.scala:59, took 1.718626 s
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ml.util.DefaultParamsReader$.getAndSetParams(Lorg/apache/spark/ml/param/Params;Lorg/apache/spark/ml/util/DefaultParamsReader$Metadata;)V
at org.apache.spark.ml.SparkDefaultParamsReadWrite$.getAndSetParams(SparkDefaultParamsReadWrite.scala:90)
at com.salesforce.op.stages.OpPipelineStageReader.loadFromJsonString(OpPipelineStageReader.scala:97)
at com.salesforce.op.stages.OpPipelineStageReader.loadFromJson(OpPipelineStageReader.scala:74)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$5.apply(OpWorkflowModelReader.scala:126)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$5.apply(OpWorkflowModelReader.scala:122)
at scala.collection.immutable.List.map(List.scala:273)
at com.salesforce.op.OpWorkflowModelReader.loadStages(OpWorkflowModelReader.scala:122)
at com.salesforce.op.OpWorkflowModelReader.com$salesforce$op$OpWorkflowModelReader$$resolveFeaturesAndStages(OpWorkflowModelReader.scala:109)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1$$anonfun$apply$3$$anonfun$apply$4.apply(OpWorkflowModelReader.scala:87)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1$$anonfun$apply$3$$anonfun$apply$4.apply(OpWorkflowModelReader.scala:87)
at scala.util.Try$.apply(Try.scala:192)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1$$anonfun$apply$3.apply(OpWorkflowModelReader.scala:87)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1$$anonfun$apply$3.apply(OpWorkflowModelReader.scala:86)
at scala.util.Success.flatMap(Try.scala:231)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1.apply(OpWorkflowModelReader.scala:86)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3$$anonfun$apply$1.apply(OpWorkflowModelReader.scala:85)
at scala.util.Success.flatMap(Try.scala:231)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3.apply(OpWorkflowModelReader.scala:85)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$3.apply(OpWorkflowModelReader.scala:84)
at scala.util.Success.flatMap(Try.scala:231)
at com.salesforce.op.OpWorkflowModelReader.loadJson(OpWorkflowModelReader.scala:84)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$2.apply(OpWorkflowModelReader.scala:73)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$loadJson$2.apply(OpWorkflowModelReader.scala:73)
at scala.util.Success.flatMap(Try.scala:231)
at com.salesforce.op.OpWorkflowModelReader.loadJson(OpWorkflowModelReader.scala:73)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$2.apply(OpWorkflowModelReader.scala:60)
at com.salesforce.op.OpWorkflowModelReader$$anonfun$2.apply(OpWorkflowModelReader.scala:60)
at scala.util.Success.flatMap(Try.scala:231)
at com.salesforce.op.OpWorkflowModelReader.load(OpWorkflowModelReader.scala:60)
at com.salesforce.op.OpWorkflow.loadModel(OpWorkflow.scala:456)
at com.salesforce.op.OpWorkflowRunner.evaluate(OpWorkflowRunner.scala:276)
at com.salesforce.op.OpWorkflowRunner.run(OpWorkflowRunner.scala:311)
at com.salesforce.op.OpAppWithRunner.run(OpApp.scala:211)
at com.salesforce.op.OpApp.main(OpApp.scala:182)
at com.salesforce.hw.iris.OpIris.main(OpIris.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/09/13 13:09:52 INFO SparkContext: Invoking stop() from shutdown hook
18/09/13 13:09:52 INFO OpSparkListener: APP_NAME:op-helloworld:com.salesforce.hw.iris.OpIris,RUN_TYPE:Evaluate,APP:op-helloworld:com.salesforce.hw.iris.OpIris,APP_ID:local-1536824388139,APP_TIME_MS:3079
18/09/13 13:09:52 INFO SparkUI: Stopped Spark web UI at http://monks-air:4040
18/09/13 13:09:52 INFO OpWorkflowRunner: Total run time: 3.079s
<===========--> 91% EXECUTING [15m 7s]
> :sparkSubmit
from transmogrifai.
tovbinm
commented on May 18, 2024
@monk1337 are you sure you are using the Spark 2.2.1 version? this type of error usually indicates that.
from transmogrifai.
tovbinm
commented on May 18, 2024
@monk1337 I suggest to move this conversation to https://gitter.im/salesforce-TransmogrifAI-Issues/Issues
from transmogrifai.
ajayborra
commented on May 18, 2024
@monk1337 As @tovbinm mentioned this issue is due to spark version mismatch. Ensure that your spark version at helloworld/build.gradle and echo $SPARK_HOME spark version are matching. If you are still facing issues reach out to us on gitter channel mentioned above.
from transmogrifai.
Related Issues (20)
- Did the documentation site's domain name expire? HOT 2
- cannot be cast to [Lcom.salesforce.op.stages.impl.feature.TextStats; HOT 5
- Model saving and loading behavior changed since #475 HOT 1
- MultiClassClassificationModelsToTry and BinaryClassificationModelsToTry not contains OpMultilayerPerceptronClassifier HOT 2
- Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String at com.salesforce.op.features.types.FeatureTypeSparkConverter$$anonfun$2.apply(FeatureTypeSparkConverter.scala:146) HOT 9
- Testing something HOT 1
- Unnecessary codec factory initialization in readAsString HOT 1
- Release drafter
- UV Computation HOT 2
- Normalize special characters in string
- CDH 6.3.2 not worked,throw NoClassDefFoundError( com.fasterxml.jackson.module.scala.modifiers.EitherModule) HOT 3
- How to use feature selection with no model training and optimization? HOT 8
- Failed to run titanic example, got java.lang.AbstractMethodError HOT 2
- build fails on AArch64, Fedora 33 HOT 1
- Changing imputation for nulls in DateToUnitCircleTransformer
- Make RecordInsightsLOCO perform reasonable calculation on numeric features and fix the name to reflect actual calculation. HOT 1
- The effect of random seeds on results ? HOT 5
- Migrating Documentation Page to Docusaurus 2
- Two cache miss case
- āšā¸ā¸´ā¸
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â¤ī¸ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transmogrifai.