Apache Spark

How to use map, filter, groupByKey Problem statement: Get daily revenue by product with CLOSED and COMPLETED orders. Save the output to a text, CSV, TSV and parguet file. The example is using spark-shell for executing all the steps. 1. Read all the files from local system, The same can be read from HDFS as well.  For simplicity of the example I am reading it from local file system. val orders = sc.textFile("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db/orders") val order_items = sc.textFile("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db/order_items") val products = sc.textFile("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db/products") All the data files can be found at GitHub. https://github.com/dgadiraju/data/tree/master/retail_db 2. Check the content of each RDD. This will help to understand the data structure and position of each element. orders.take(3).f...

How to use DataFrame registerTempTable in spark SQL. Problem statement: Get daily revenue by product with CLOSED and COMPLETED orders. Save the output to a text, CSV, TSV and parguet file. 1. Load all the json file into spark DataSet using sqlContext. All the data files can be found at GitHub. https://github.com/dgadiraju/data/tree/master/retail_db_json scala> val products = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/products/*") scala> val orders = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/orders/*") scala> val order_items = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/order_items/*") scala> products.limit(2).show +-------------------+-------------------+----------+--------------------+--------------------+-------------+ |product_category_id|product_d...

How to use (inner) JOIN and group by in spark SQL. Problem statement: Get daily revenue by product with CLOSED and COMPLETED orders. Save the output to a text, CSV, TSV and parguet file. 1. Load all the json file into spark DataSet using sqlContext. All the data files can be found at GitHub. https://github.com/dgadiraju/data/tree/master/retail_db_json scala> val products = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/products/*") scala> val orders = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/orders/*") scala> val order_items = sqlContext.read.json("file:///home/cloudera/workspace/scala/rajudata/data-master/retail_db_json/order_items/*") scala> products.limit(2).show +-------------------+-------------------+----------+--------------------+--------------------+-------------+ |product_category_id|product_desc...

How to read a tab separated text file and display the details in a table using spark and Scala 1. Read a text (tab separated) file which needs to be processed. scala> val election2014 = sc.textFile("file:///home/hadoop/workspace/scala/rajudata/data-master/electionresults/ls2014.tsv") election2014: org.apache.spark.rdd.RDD[String] = file:///home/hadoop/workspace/scala/rajudata/data-master/electionresults/ls2014.tsv MapPartitionsRDD[59] at textFile at <console>:24 Note: You can get the data file at below location. https://github.com/dgadiraju/data/tree/master/electionresults 2. The input text file is having a header. Which needs to be filtered before actual processing. scala> election2014.first res72: String = state        constituency        candidate_name        sex        age  ...