Showing posts with label ubuntu. Show all posts
Showing posts with label ubuntu. Show all posts

Saturday, 20 October 2012

Compiling and Deploying a simple servlet in tomcat : UBUNTU


Hi there,

These are the steps required to compile and deploy a simple servlet in tomcat 7 that is installed on ubuntu.

So initially tomcat has to be installed and the JAVA_HOME variable has to be set.

Next write a simple java servlet progam. For example,




In order to compile this successfully, we have to include the  servlet-api.jar. If working_dir is the directory where tomcat is installed , it is present in working_dir/lib.

So this can be done as,

$export CLASSPATH=$CLASSPATH:working_dir/lib/servlet-api.jar


Once this is done, you can compile it normally,

$javac Hello.java

Now that  you have the class file, time to deploy on tomcat!!!

Supposing the name of my project is sample, I will have the following hierarchy in tomcat,
                                   
                                    working_dir
                                           |
                                           ----------------webapps
                                                                 |
                                                                 ----------------sample
                                                                                    |
                                                                                    -------------WEB-INF
                                                                                                      |
                                                                                                      |-------classes
                                                                                                      |-------lib
                                                                                                      |-------web.xml

Here the folder classes contains the class files of our servlet whereas the library folder contains the libraries that are specific to the project.
The web.xml is the configuration of this particular project.

So place the Hello.class in the classes folder. Open web.xml and type in the following,




The servlet name Hello is mapped to the url-pattern /hello, hence we can access using the url pattern, http://localhost:8080/sample/Hello. We also associate it with the class Hello.class.

Now start the tomcat server,

$working_dir/bin/startup.sh

Having done this,go to your browser and type in 

http://localhost:8080/sample/Hello

That is about it

aps.

Sunday, 15 July 2012

Installing Apache Mahout on Hadoop Cluster

             Mahout is an open source scalable machine learning library by Apache. To process huge amounts of data, you have to run mahout on top of hadoop cluster. Mahout can run as a stand-alone application as well. To run mahout on hadoop cluster it is sufficient to install mahout only on Hadoop master node.
             Mahout can be installed from its source distributions. It's obvious to build the source using ant like build tool. Here we use Apache maven to build the source by dynamically downloading dependencies and plug-ins from appropriate repositories.

Installing Maven on Ubuntu:

      1) Download the latest version of apache-maven from the link http://maven.apache.org/download.html and extract it.

      2) Set the MAVEN_HOME variable in the /etc/environment file.
                   
                     
PATH = $JAVA_HOME/bin:$MAVEN_HOME /bin

MAVEN_HOME = /your location/apache-maven-*

SUDO GEDIT /ETC/ENVIRONMENT


                     

                     
      3) Add the above two lines in the environment file and save the file. Logout and login for the changes to take effect or use source command
                   
SOURCE /ETC/ENVIRONMENT


      4) Maven is installed and Check your maven version.
                     
MVN --VERSION.    


Mahout Installation:

      1) Download and extract the mahout-distribution-* from the link http://www.apache.org/dyn/closer.cgi/mahout/  . Alternatively you can checkout mahout distributions from the trunk.
                     
 svn co http://svn.apache.org/repos/asf/mahout/trunk
                      cd mahout-distributions-*
   
     2) Now build the package using maven
                     
MVN CLEAN INSTALL

              [avoid testing of files by using the option -DskipTests]  
  
     3) Mahout is installed and you can check by the command
                     
/BIN/MAHOUT



     4) Before running mahout on top of hadoop ensure hadoop is installed and running in any of the modes. To run mahout on top of hadoop set the environment variables HADOOP_HOME and HADOOP_CONF_DIR and comment the MAHOUT_LOCAL variable in the mahout script file.
                   
 SUDO GEDIT /BIN/MAHOUT

     5) Add the following lines on the top
                   
HADOOP_HOME =         /your location/hadoop
        HADOOP_CONF_DIR = /your location/hadoop/conf
         
     6) Comment the following line which makes the mahout application to run locally.
                   
#MAHOUT_LOCAL =/your location/mahout-distribution-*

         
Simple Program in Mahout:

     1) 20news-group with naive bayes classifier algorithm. the test and train data set from the link http://people .csail.mit.edu/jrennie/20Newsgroups/20news- bydate.tar.gz.
       
     2) Extract the tar file. Now you need to prepare the data set to be accepted by the bayes algorithm. You need to scan each directory and transform each file into relevant format. The mahout program prepare20newsgroups does this. Use the command to convert train and test data
           
bin/mahout prepare20newsgroups -p /your location/20news-bydate-train -o /your location/20news-train -a org.apache.mahout.vectorizer.DefaultAnalyzer -c UTF-8

           
 bin/mahout prepare20newsgroups -p /your location/20news-bydate-test -o /your/location/20news-test -a org.apache.mahout.vectorizer.DefaultAnalyzer -c UTF-8

     3) Now train the classifier to create the model.
           
bin/mahout trainclassifier -i /your location/20news-train -o /your location/20news-model -type bayes -ng 1 -source
hdfs          

                       
           
    4) The model generated is tested with the test data set to measure the accuracy. A confusion matrix is generated for the classification.
             
bin/mahout testclassifier -d /your location/20news-test -m /your location/20news-model -type bayes -ng 1 -source hdfs -method parallel
                       
                          [parallel - runs on map-reduce process , sequential - runs on local system ]
           
     5) The confusion matrix generated

   
 Thats it !!!! Mahout is configured to run on hadoop and you can play with it.




>>Venk@_7harun<<


 

Saturday, 14 July 2012

How to create a Hadoop project in eclipse



Most of us prefer IDE for the development process as it makes life easier. So here we will see  how to create a hadoop project on eclipse.
Hadoop on Eclipse:
1) Follow the installation setup of hadoop single node cluster on Ubuntu (My previous blog).
2) Download Hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar from the following site



(I prefer this plugin and it works well for me).
3) Paste that jar file into /usr/lib/eclipse/plugins/ 
4) Run the eclipse via terminal

     $ cd /usr/lib/eclipse
     $ ./eclipse –clean

5) Create a new map-reduce project
     
a) File->new->map/reduce project
    
 b) Select configure Hadoop install directory or specify hadoop library location and give the pathname of hadoop folder in which you configured single node cluster.

6) From window->show view -> map-reduce locations.
     
a) In that map-reduce location, you have to set hadoop location by using New hadoop location icon (blue color elephant symbol). Hadoop location can be any name as you like.
     b) In General
           1) Give Map/Reduce Master as    
              Host -> localhost
              port -> 54311
           2) Give HDFS Master as
              Host -> localhost
              port -> 54310
           3) Refer: /conf/*-site.xml files for the values.
7) Refresh the dfs locations in project explorer tab. 
      a) You can upload the file into dfs location and also download it.
      b) This location will be helpful for input – output operation for hadoop map-reduce programs.
8) Finally you can run your project on hadoop.

Now create your own map/reduce program on eclipse and enjoy with hadoop.

Cheers,
Kiran.




Installation setup of Hadoop single node cluster on Ubuntu

Installation of hadoop for single node cluster on Ubuntu.

Many people are eager to work in big data, nosql and map reduce framework in the present day scenario.  I was also given the opportunity during  my summer internship to work  in big data. We used hadoop technology to achieve map/reduce framework and I would like to write about the installation setup of hadoop single node cluster on ubuntu.

Do the following commands in your linux machine and enjoy with hadoop!!!

Install Java 6 

$sudo add-apt-repository ppa:ferramroberto/java
$sudo apt-get update
$sudo apt-get install sun-java6-jdk
$sudo update-java-alternatives -s java-6-sun (To config java information).
Check the version – > $java -version

Install SSH

$sudo apt-get install openssh-client
$sudo apt-get install openssh-server

Configuring SSH


$ssh-keygen -t rsa -P ""




$cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
$ssh localhost

Download the Hadoop source package in the following link http://www.apache.org/dyn/closer.cgi/hadoop/core and do the following changes.

Extract the hadoop package from the downloaded tar file.
Change the Hadoop source package permission (chmod -R 777 (hadoop_package_name). 
NOTE: Some times it wont works for all internal files so u have to right click the folder and change the permission for all the internal files.

Set the relevant environmental variables.

$export HADOOP_HOME=(path of ur hadoop package).
$export JAVA_HOME=/usr/lib/jvm/java-6-sun

In Hadoop/conf/hadoop-env.sh

Update the following line as
# The java implementation to use. Required.

$export JAVA_HOME=/usr/lib/jvm/java-6-sun

In Hadoop/conf/core-site.xml

Do the following changes

In Hadoop/conf/mapred-site.xml

Do the Following changes 
 

In Hadoop/conf/hdfs-site.xml

Do the following changes

Getting Start with Hadoop

To format ur Namenode

$.../Hadoop/bin/hadoop namenode –format

To start ur single node cluster

$…/Hadoop/bin/start-all.sh
              (or)
$.../Hadoop/bin/start-dfs.sh
$.../Hadoop/bin/start-mapred.sh



Running Sample application on mapreduce – (Word count program)

Download the sample plain txt file in the following links and save it as …./input/

http://www.gutenberg.org/ebooks/20417
http://www.gutenberg.org/ebooks/5000
http://www.gutenberg.org/ebooks/4300

Move the input folder to Hadoop distributed filesystem

Here /user/hduser/input is the path that will be created in the HDFS.

$bin/hadoop dfs -copyFromLocal …/input /user/hduser/input

To run a word count program

Here * represents version of your Hadoop and the jar file.

$bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/input /user/hduser/output


Retrieve job results from hdfs

$bin/hadoop dfs -cat /user/hduser/output/part-r-00000


Hadoop web interfaces

$http://localhost:50070/ - Web UI of the NameNode (Browse the hdfs filesystem, log files).

$http://localhost:50030/ - Web UI of the JobTracker.

$http://localhost:50060/ - Web UI of the TaskTracker


Once this is over you can work with the single node cluster of Hadoop on Ubuntu. Have fun!!!

Cheers,
Kiran.