Quantcast
Channel: Unable load a CSV file as dataframe in spark - Stack Overflow
Viewing all articles
Browse latest Browse all 2

Unable load a CSV file as dataframe in spark

$
0
0

I am trying to load a CSV file in the data frame and my objective is to display the first row as the column name of the CSV file. but while using the below code, I am getting the error

Exception in thread "main" java.lang.AbstractMethodError    at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:278)

Code:

def main(args : Array[String]): Unit = { val spark : SparkSession = SparkSession      .builder()      .master("local")      .appName("SparkSessioncsvExample")      .config("spark.some.config.option", "some-value")      .getOrCreate()   val df = spark.read      .format("csv")      .option("header", "true") //reading the headers      .load("D:/Scala/C2ImportCalEventSample.csv")}

But I'm able to load the file with the code:

val df = spark.sparkContext        .textFile("D:/Scala/C2ImportCalEventSample1.csv")        //.flatMap(header='true')        .map(line => line.split(","))        // .map(line => line.map()        .toDF()

but in the second code file is getting successfully loaded but the first row is not getting as column_name of the data frame.

spark version is: spark-2.3.2  scala 2.11.3  jdk1.8.0_20sbt-1.2.7

Thanks any anyone who can help me on this.


Viewing all articles
Browse latest Browse all 2

Latest Images

Trending Articles



Latest Images