TechMusings (BigData,Hadoop,Pig,Hive,DataScience,IoT,EAI,SOA,J2EE)

Hi, The below post lists out my inital experience to setup and code/run Map Reduce Program using Eclipse  and Hadoop 2.5.0. The easiest way is to create a  maven project and use appropriate pom.xml to setup mapreduce project in Eclipse. Have tried to list the way if one tries to create java project and add all needed libraries (jar) to build path. Dependency Jars org-apache-commons-logging.jar org.apache-commons-collections.jar apache-commons-lang.jar hadoop-common-2.5.0.jar hadoop-common-2.5.0-tests.jar hadoop-nfs-2.5.0.jar commons-configuration-1.7.jar hadoop-auth-2.5.0.jar hadoop-hdfs-2.5.0.jar hadoop-hdfs-2.5.0-tests.jar hadoop-hdfs-nfs-2.5.0.jar hadoop-mapreduce-client-app-2.5.0.jar hadoop-mapreduce-client-common-2.5.0.jar hadoop-mapreduce-client-core-2.5.0.jar hadoop-mapreduce-client-hs-2.5.0.jar hadoop-mapreduce-client-hs-plugins-2.5.0.jar hadoop-mapreduce-client-jobclient-2.5.0.jar hadoop-mapreduce-client-jobclient-2.5.0-tests.jar hadoop-mapreduce-clie

Hi, In this blog, I will share how you can use Swagger to document your Rest services/api. Swagger is an open source software that can come handy and is easy to use/configure if you are looking to document your Rest APIs. During development with multiple developers in a distributed team it can also come handy to have a visibility on Rest Services developed by Team. This article will focus on basic configuration.For sample JSON Rest web service have referred to an example from http://avaldes.com/spring-restful-web-service-example-with-json-and-jackson-using-spring-tool-suite/. It's a good site with info. Step-1: Make sure you have a Restful webservice up and running. I tried using an example from http://avaldes.com/spring-restful-web-service-example-with-json-and-jackson-using-spring-tool-suite/ It can be downloaded from http://avaldes.com/?wpdmdl=4550 so before we move forward,check http://localhost:8080/<context-name>/issuers is returning a JSON response. Step-2

Error and Probable Fix: 1. Error : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2015-05-02 13:57:39,982 Stage-1 map = 0%,  reduce = 0% 2015-05-02 13:57:49,511 Stage-1 map = 100%,  reduce = 100% Ended Job = job_1430542224940_0001 with errors Error during job, obtaining debugging information... Examining task ID: task_1430542224940_0001_m_000000 (and more) from job job_1430542224940_0001 Task with the most failures(4): ----- Task ID:   task_1430542224940_0001_m_000000 URL:   http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1430542224940_0001&tipid=task_1430542224940_0001_m_000000 ----- Diagnostic Messages for this Task: Container launch failed for container_1430542224940_0001_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeCons