java.lang.ClassNotFoundException: java.lang.ProcessEnvironment$Variable about spark-bigquery HOT 12 CLOSED

Same error .. just coming to post this same problem.
Code works fine from Google Cloud but when run from my desktop using credentials it fails with the above.

Java.lang.ClassNotFoundException: java.lang.ProcessEnvironment$Variable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at com.samelamin.spark.bigquery.utils.EnvHacker$.setEnv(EnvHacker.scala:15)
	at com.samelamin.spark.bigquery.BigQueryClient$.setGoogleBQEnvVariable(BigQueryClient.scala:51)
	at com.samelamin.spark.bigquery.BigQueryClient$.getInstance(BigQueryClient.scala:34)
	at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19)
	at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19)
	at com.samelamin.spark.bigquery.BigQuerySQLContext.setBigQueryDatasetLocation(BigQuerySQLContext.scala:65)
	at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply$mcV$sp(DfpSdSupplyDemandTest.scala:35)
	at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22)
	at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
	at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
	at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)

from spark-bigquery.

from spark-bigquery.

Scala .. must be something in my desktop config - it works on the google cloud vm.
On gcp box I run ..
spark-shell --packages com.github.samelamin:spark-bigquery_2.11:0.2.2,com.github.databricks:spark-avro:204864b6cf < test.scala

If I run via intellij on my desktop ..
It fails on the last line ... note that I dont have the JsonKey line in the code when I run on gcp.

import com.samelamin.spark.bigquery._
spark.sqlContext.setGcpJsonKeyFile(s"$resources/BIGDATA-CREDENTIALS.json")
spark.sqlContext.setBigQueryProjectId("bbm-production-bigdata")
spark.sqlContext.setBigQueryGcsBucket("bbm-production-bigdata")
spark.sqlContext.setBigQueryDatasetLocation("US")

from spark-bigquery.

from spark-bigquery.

Windows .. I tried setting the GOOGLE_APPLICATION_CREDENTIALS. First time it said it couldnt find the file then I renamed and it found it. Got the above error. So I removed it thinking I could just use the api call instead but still get the error.

using jdk1.8.0_144 and jre1.8.0_111

Are you able to share any intelij settings that might be relevant?
I have your bigquery_2.11-0.2.2.jar
along with the spark2.2.0-bin-hadoop2.7 jars

from spark-bigquery.

Is the json creds file something similar to ..

{
"type": "service_account",
"project_id": "xxxx",
"private_key_id": "xxxxxxa",
"private_key": "-----BEGIN PRIVATE KEY-----\n
blah blah blah

from spark-bigquery.

from spark-bigquery.

Did you say you didnt use an environment variable? or you do? Did you do anything special? I did in windows environment VARIABLE="/path/to/.json"

from spark-bigquery.

from spark-bigquery.

I am closing this issue as I have not heard anything back and I cannot replicate it

from spark-bigquery.

A member of our team here found the issue . it is the hack for read/writing environment variables that doesnt work on windows. He commented out the code as it wasn't apparently required. setting a hadoop environment variable?

com.samelamin.spark.bigquery.utils.EnvHacker

from spark-bigquery.

from spark-bigquery.