I am using MIM feature selector model. It running perfectly fine and output the features results automatically in a terminal. How can I extract output features with its relevant score? I want to save this output to a file.

val selector = new InfoThSelector()
         .setSelectCriterion("mim")
         .setNumTopFeatures(2)
         .setFeaturesCol("features")
         .setLabelCol("class")
         .setOutputCol("selectedFeatures")

Would it be possible to build and deploy a sources jar along with the execution jar when you deploy to the maven repository? Having the source makes it easy to debug and follow the code during execution.

Its pretty easy, just add the sources plugin section (link below) to the pom file and it'll create the sources jar when you run package.

https://maven.apache.org/plugin-developers/cookbook/attach-source-javadoc-artifacts.html

Hi I'm trying the given example (mllib) but encountered the following error messages:

val featureSelector = InfoThSelector(criterion, nToSelect, nPartitions).fit(data)
org.apache.spark.mllib.feature.InfoThSelector.type does not take parameters
val featureSelector = InfoThSelector(criterion, nToSelect, nPartitions).fit(data)

Any ideas? Would appreciate very much for the help.

Wei

Hi, I am using the spark-infotheoretic-feature-selection jar file. I import the jar file into my project and maven the project,but it occur the error as follows:

I don't know how to fix it? May the jar file conflit with the origin mllib.feature jar file ?

Thank you

Currently feature selection and MDL techniques are built using spark 1.6.1. Is there plan to support spark 2.0.0 or latest versions?.

Thanks for making this lib.

Can you please build a Spark 3 / Scala 2.12 JAR file and upload it to Maven?

Looks like you were using Spark Packages. Think it's best to shift off Spark packages now that it's pretty much abandonware.

It's relatively straightforward to publish directly to Maven with sbt-sonatype. spark-daria is a good example.

Let me know if you'd be willing to go through this whole process. I am happy to help.

Hi @sramirez,

Would you like to make a release of this package on the Spark Packages Maven Repo as well? There is an sbt-plugin called sbt-spark-package that would help you make the release straight from your sbt console. All you need to do is set a couple configurations.

Publishing on the Spark Packages Repo will bump your ranking on the website, and will fill in the How To section, which users can use to include your package in their work.

Please let me know if you have any comments/questions/suggestions!

Best,
Burak

P.S. I bolded as well so that you would at least know that this is not an automatically generated message :)

hello， Have you solved this problem? Can you explain to me how to put the key points and bounding boxes together in the format of coco dataset，Thank you

/path/to/spark-infotheoretic-feature-selection/build.sbt:19: error: not found: value sparkPackageName
sparkPackageName := "sramirez/infotheoretic-feature-selection"
^
[error] Type error in expression

All I did change was sparkVersion and libraryDependencies to 1.4.0

Thanks

Is your algorithm not support for double or float value? Do you have suggestion if I have a very big value data like in million ? because your algorithm not support it

I would like to ask why my code reported an error：Exception in thread "main" java.lang.AbstractMethodError

object FSTest01 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("FS test")
    val sc = new SparkContext(conf)
    val raw = MLUtils.loadLibSVMFile(sc, "a2a.libsvm")
    val data: RDD[LabeledPoint] = raw.map({ case LabeledPoint(label, values) => LabeledPoint(label, Vectors.dense(values.toArray)) })

    val criterion: InfoThCriterionFactory = new InfoThCriterionFactory("mrmr")
    val nToSelect = 5
    val nPartitions = 10

    val featureSelector = new InfoThSelector(criterion, nToSelect, nPartitions).fit(data)

    val redData = data.map { lp =>
      LabeledPoint(lp.label, featureSelector.transform(lp.features))
    }

    println(redData.first().toString())
  }
}

Error reported on the fit line of the code

I just tried all the criterion on the same dataset. But I get the same ranking no matter what i set the SelectCriterion to.

I was wondering if I need to do a particular transform to my data before running the selector.

When running, I first apply a minmaxscaler to all features then multiply by 255 round off to int and finally subtract 128 so they are the byte range.