TechMusings (BigData,Hadoop,Pig,Hive,DataScience,IoT,EAI,SOA,J2EE)

Why a framework? To standardize the automation development effort and implement best practices. To provide a single point for customization and manage changes. Reusable code. Types of frameworks? refer softwaretestinghelp.com for details. Components that one can usually code as part of framework... Input Folders Output  Folders Recording Scripts Src Code -modules Utility Components Reporting Some Basic skills in programming ? Read data from xls [opening an xls..selecting a sheet..selecting value of a cell..] Create folders based on date of execution and run number..write results to folders utility functions to manage data...like date format change, removing of trail spaces..etc. selection of test cases for execution batch creation emailing failure handling execution results and summarized view of execution results Integration with daily code ..Continuous integration and build. Browser and OS independent. (refer few others http://w...

In apache site, if you simply follow..below in ubuntu it didn't work.. https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/SingleCluster.html Setup passphraseless ssh Now check that you can ssh to the localhost without a passphrase: $ ssh localhost If you cannot ssh to localhost without a passphrase, execute the following commands: $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys $ chmod 0600 ~/.ssh/authorized_keys So check for 2 things.. a) U have a SSH server installed..do it first. b) Instead of using dsa use rsa $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys $ chmod 0600 ~/.ssh/authorized_keys

Iteration-0 Consolidated  2 good posts..This is just for personal reading..references use only. ------------------------------------------------------------------------ https://www.r-bloggers.com/simple-linear-regression-2/ One of the most frequent used techniques in statistics is linear regression where we investigate the potential relationship between a variable of interest (often called the response variable but there are many other names in use) and a set of one of more variables (known as the independent variables or some other term). Unsurprisingly there are flexible facilities in  R  for fitting a range of linear models from the simple case of a single variable to more complex relationships. For this example we will use some data from the book Mathematical Statistics with Applications by Mendenhall, Wackerly and Scheaffer (Fourth Edition – Duxbury 1990). This data is for a study in central Florida where 15 alligators were captured and two measurements w...

Installing R in Ubuntu 14 LTS sudo apt-get install r-base Let's verify that installation was  correct. Start an R session. ~$ R R version 3.0.2 (2013-09-25) -- "Frisbee Sailing" Copyright (C) 2013 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details.   Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > OOPS!!!.. This installed an old one... i.e /etc/apt/sources.list had a old reference to R  ...

1. Download and configure JDK1.8 2. Download and configure Hadoop 2.6 3. Download and configure Spark for Hadoop2.6     Start master and access the url via http://localhost:8080.     Start slave pass the spark://localhost:<port> seen in the master 4.Download and configure Kafka (http://kafka.apache.org/documentation.html#quickstart) https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz 5. Start producer/broker..test using a message and check if spark gets the message. a) a uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance. > bin/zookeeper-server-start.sh config/zookeeper.properties   b) Now start the Kafka server: > bin/kafka-server-start.sh config/server.properties [2016-07-13 04:54:07,557] INFO [Kafka Server 0], started (kafka.server.Kafka...