Spark pipeline was not converted properly about jpmml-sparkml HOT 5 CLOSED

This issue is a functional duplicate of jpmml/jpmml-evaluator#84

In brief, your testing data is not "compatible" with the training data - for some categorical feature(s), the testing data contains category values that were not present in training data. The JPMML-Evaluator library then refuses to score such data records, because the prediction would be non-sensical.

from jpmml-sparkml.

Also, what is your Apache Spark ML version, and how is the StringIndexer transformation configured?

For example, in Apache Spark ML version 2.2 it is possible to specify how invalid values should be handled by setting the StringIndexer@handleInvalid attribute. In this case, the value of this attribute should be set to "keep".

from jpmml-sparkml.

I am using Spark ML version 2.2.0
This is my StringIndexer configuration.

def getStringIndexer(columnNames:Array[String]):Array[PipelineStage]={
   var stringIndexers : Array[PipelineStage] = new Array[PipelineStage](0)
    for (i <- 0 until  columnNames.length)
    {
      val indexer = new StringIndexer()
        .setInputCol(columnNames(i))
        .setOutputCol(columnNames(i)+"Indexer")
      stringIndexers = stringIndexers:+indexer
    }
    stringIndexers
  }

from jpmml-sparkml.

This is my StringIndexer configuration.

You're using the default StringIndexer@handleInvalid attribute value, which is "error". Check Apache Spark ML documentation - it is a scoring error (implemented as InvalidResultException in the JPMML-Evaluator library) if the input contains a previously unseen category value.

Please note that the native Apache Spark ML workflow would also fail to make a prediction in this case.

You need to do the following to make unseen categories scorable:

val indexer = new StringIndexer()
  .setInputCol(columnNames(i))
  .setOutputCol(columnNames(i) + "Indexer")
  .setHandleInvalid("keep"); // THIS!

from jpmml-sparkml.

Thanks a lot @vruusmann ! That's really helpful. ^O^

from jpmml-sparkml.