Excersize

From Ivan Kirilenkov, 10 Months ago, written in Python, viewed 322 times.
URL http://codebin.org/view/b4ca0c27 Embed
Download Paste or View Raw
  1. from pyspark.sql import SparkSession
  2.  
  3. APP_NAME = "DataFrames"
  4. SPARK_URL = "local[*]"
  5.  
  6. spark = SparkSession.builder.appName(APP_NAME) \
  7.         .config('spark.ui.showConsoleProgress', 'false') \
  8.         .getOrCreate()
  9.  
  10. taxi = spark.read.load('/datasets/pickups_terminal_5.csv',
  11.                        format='csv', header='true', inferSchema='true')
  12.  
  13. taxi = taxi.fillna(0)
  14.  
  15. taxi.registerTempTable("taxi")
  16.  
  17. result = spark.sql("SELECT pickups FROM taxi order by pickups desc")
  18.  
  19. print(result.show(5))

Reply to "Excersize"

Here you can reply to the paste above