Azure DataBricks Getting Started

Comments

Post a Comment

Popular posts from this blog

The auxService:mapreduce_shuffle does not exist

Error and Probable Fix: 1. Error : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2015-05-02 13:57:39,982 Stage-1 map = 0%,  reduce = 0% 2015-05-02 13:57:49,511 Stage-1 map = 100%,  reduce = 100% Ended Job = job_1430542224940_0001 with errors Error during job, obtaining debugging information... Examining task ID: task_1430542224940_0001_m_000000 (and more) from job job_1430542224940_0001 Task with the most failures(4): ----- Task ID:   task_1430542224940_0001_m_000000 URL:   http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1430542224940_0001&tipid=task_1430542224940_0001_m_000000 ----- Diagnostic Messages for this Task: Container launch failed for container_1430542224940_0001_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun...
Read more

Running Map Reduce Program using Eclipse

Hi, The below post lists out my inital experience to setup and code/run Map Reduce Program using Eclipse  and Hadoop 2.5.0. The easiest way is to create a  maven project and use appropriate pom.xml to setup mapreduce project in Eclipse. Have tried to list the way if one tries to create java project and add all needed libraries (jar) to build path. Dependency Jars org-apache-commons-logging.jar org.apache-commons-collections.jar apache-commons-lang.jar hadoop-common-2.5.0.jar hadoop-common-2.5.0-tests.jar hadoop-nfs-2.5.0.jar commons-configuration-1.7.jar hadoop-auth-2.5.0.jar hadoop-hdfs-2.5.0.jar hadoop-hdfs-2.5.0-tests.jar hadoop-hdfs-nfs-2.5.0.jar hadoop-mapreduce-client-app-2.5.0.jar hadoop-mapreduce-client-common-2.5.0.jar hadoop-mapreduce-client-core-2.5.0.jar hadoop-mapreduce-client-hs-2.5.0.jar hadoop-mapreduce-client-hs-plugins-2.5.0.jar hadoop-mapreduce-client-jobclient-2.5.0.jar hadoop-mapreduce-client-jobclient-2.5.0-tests.jar hadoop-mapre...
Read more

Linear Regression using R

Image
Iteration-0 Consolidated  2 good posts..This is just for personal reading..references use only. ------------------------------------------------------------------------ https://www.r-bloggers.com/simple-linear-regression-2/ One of the most frequent used techniques in statistics is linear regression where we investigate the potential relationship between a variable of interest (often called the response variable but there are many other names in use) and a set of one of more variables (known as the independent variables or some other term). Unsurprisingly there are flexible facilities in  R  for fitting a range of linear models from the simple case of a single variable to more complex relationships. For this example we will use some data from the book Mathematical Statistics with Applications by Mendenhall, Wackerly and Scheaffer (Fourth Edition – Duxbury 1990). This data is for a study in central Florida where 15 alligators were captured and two measurements w...
Read more