I am trying to load a CSV file in the data frame and my objective is to display the first row as the column name of the CSV file. but while using the below code, I am getting the error
Exception in thread "main" java.lang.AbstractMethodError at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:278)
Code:
def main(args : Array[String]): Unit = { val spark : SparkSession = SparkSession .builder() .master("local") .appName("SparkSessioncsvExample") .config("spark.some.config.option", "some-value") .getOrCreate() val df = spark.read .format("csv") .option("header", "true") //reading the headers .load("D:/Scala/C2ImportCalEventSample.csv")}
But I'm able to load the file with the code:
val df = spark.sparkContext .textFile("D:/Scala/C2ImportCalEventSample1.csv") //.flatMap(header='true') .map(line => line.split(",")) // .map(line => line.map() .toDF()
but in the second code file is getting successfully loaded but the first row is not getting as column_name of the data frame.
spark version is: spark-2.3.2 scala 2.11.3 jdk1.8.0_20sbt-1.2.7
Thanks any anyone who can help me on this.