Azure DataBricks Getting Started

Comments

Post a Comment

Popular posts from this blog

The auxService:mapreduce_shuffle does not exist

Error and Probable Fix: 1. Error : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2015-05-02 13:57:39,982 Stage-1 map = 0%,  reduce = 0% 2015-05-02 13:57:49,511 Stage-1 map = 100%,  reduce = 100% Ended Job = job_1430542224940_0001 with errors Error during job, obtaining debugging information... Examining task ID: task_1430542224940_0001_m_000000 (and more) from job job_1430542224940_0001 Task with the most failures(4): ----- Task ID:   task_1430542224940_0001_m_000000 URL:   http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1430542224940_0001&tipid=task_1430542224940_0001_m_000000 ----- Diagnostic Messages for this Task: Container launch failed for container_1430542224940_0001_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun...
Read more

Zeppelin and Anaconda

Set Anaconda As Default Python Interpreter In Zeppelin Click  anonymous  in top right corner. Click  Interpreter . Scroll down to the  python  interpreter. Click  Edit . Locate  zeppelin.python . Set value to  /home/hadoop/anaconda/bin/python Now find the  spark  interpreter. Locate  zeppelin.pyspark.python . Set value to  /home/hadoop/anaconda/bin/python https://dziganto.github.io/zeppelin/spark/zeppelinhub/emr/anaconda/tensorflow/shiro/s3/theano/bootstrap%20script/EMR-From-Scratch/ This works..tested.. conda install -c calex sklearn-pandas P.S:  zeppelin 0.7.3 don't support spark 2.3. spark 2.3 is supported by zeppelin 0.8 which will be released soon zeppelin 0.7.3 doesn't support python 3.6.. now testing with spark 2.1 and python 3.5...so far so good.. Check:::::: zeppelin home/conf should have zeppelin-env.sh. Here you can specify spark home and zeppelin port Starting Apach...
Read more

Swagger ( An easy way to document your Rest API) Tutorial

Image
Hi, In this blog, I will share how you can use Swagger to document your Rest services/api. Swagger is an open source software that can come handy and is easy to use/configure if you are looking to document your Rest APIs. During development with multiple developers in a distributed team it can also come handy to have a visibility on Rest Services developed by Team. This article will focus on basic configuration.For sample JSON Rest web service have referred to an example from http://avaldes.com/spring-restful-web-service-example-with-json-and-jackson-using-spring-tool-suite/. It's a good site with info. Step-1: Make sure you have a Restful webservice up and running. I tried using an example from http://avaldes.com/spring-restful-web-service-example-with-json-and-jackson-using-spring-tool-suite/ It can be downloaded from http://avaldes.com/?wpdmdl=4550 so before we move forward,check http://localhost:8080/<context-name>/issuers is returning a JSON response. Step-2...
Read more