Configuring freeIPA Kerberos in Cloudera Manager : Cloudera Manager + freeIPA + Kerberos

1. Create a user in free IPA using command line or WebUI
• Suppose this user is Hadoop
2. Now add this user to admin and trust user group in IPA user groups.
3. Generate a Kerberos ticket for this created user, run this command at the server where Cloudera Manager is hosted or installed.
• ipa-getkeytab -s <your ipa server name> -p <hadoop@your-realm.name> -k keytabfile.keytab 
• e.g.:
• ipa-getkeytab -s ipa.server.com -p hadoop@SERVER.COM -k cdh.keytab
• kinit with the the user hadoop
• kinit hadoop@SERVER.COM
• It will ask you the password; pass the password set while creating the keytab.
• It will now ask you to change the password, set a new password
• Use kdestroy to remove the hadoop user cache with old password.
• again do kinit with new password
• kinit hadoop@SERVER.COM
• Again get the keytab files using command, in order to get the latest keytab file with update password.
• ipa-getkeytab -s ipa.server.com -p hadoop@SERVER.COM -k cdh.keytab
4. Copy this keytab file /etc/cloudera-scm-server directory of any path which should be accessible to Cloudera Manager.
5. Once you get the cdh.keytab file change its ownership to cloudera-scm user
• chown cloudera-scm: cloudera-scm cdh.keytab
6. Download the custom keytab retrieval script file from: 
• https://github.com/ndunnage/hadoop-tools/blob/master/kerberos/gen_credentials_ipa.sh. 
• (Credit: ndunnage) Which is tested and running fine.
7. Save this script to some location which should be accessible to Cloudera Manager for example:
• /etc/cloudera-scm-server/
• Change the permission of this file
• chown cloudera-scm: cloudera-scm gen_credentials_ipa.sh
• chmod 775 gen_credentials_ipa.sh
8. The parameter which you need to change in this file follows:
• IPA_SERVER=ipa.server.com
• REALM=SERVER.COM
• KEYTAB_FILE=/etc/cloudera-scm-server/cdh.keytab
• CM_PRINC=hadoop@$REALM
• There is a line near line number 39 in this script which says
• ipa service-add $PRINCIPAL
• Make change to this like add --force at the end of this line(This is use to override the restriction in freeIPA) then line will look like
• ipa service-add $PRINCIPAL  --force
9. Login the server freeIPA is installed/login to freeIPA for example ipa.server.com
• Start kadmin.local as
•  kadmin.local -x ipa-setup-override-restrictions
•  This will login as root/admin@SERVER.COM
10.  Add a new principle for Cloudera-manager here
•  addprinc –pw <passwordforuser> cloudera-scm/admin@SERVER.COM
• e.g.:
• addprinc –pw cdhpassword Cloudera-scm/admin@SERVER.COM
11. Once this principle is added successfully, go to Cloudera manager Web UI <your-clouderamanage- server:7180>
12.  Go to administration security/ or from the cluster dropdown menu choose enable Kerberos
13. In the resulting page enter the required details, and in the page where it says KDC account manager credentials enter
• First text box Username : cloudera-scm/admin
•  2nd Text box Your REALM : SERVER.COM
• Third Box Password : passwordforuser
14.  Once this step successes move forward to generate other service principal which Cloudera Manager will generate automatically if this step is successful.

Hope this will help.

Change desktop screen size using terminal linux inside VM

--To--

Change desktop screen size using terminal linux inside VM

xrandr -s 1440x900

xrandr -s <Resolution which you want to change>

you can use

xrandr --verbose 

option to get the available screen resolution available.

Option with xrandr :

Install GNOME Desktop Ubuntu

Using following command we can install and enable desktop GUI for Ubuntu server version if needed.


sudo apt-get install ubuntu-gnome-desktop
sudo service gdm restart

org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

Add following entries to yarn-site.xml if not present and restar yarn service.


<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>


<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Enable passwordless sudoers for a user

We can add following line in /etc/sudoers to enable the user to be passwordless login from other machines.


<username>  ALL=(ALL:ALL) NOPASSWD: ALL

Comparison between RDBMS and Map-Reduce

	RDBMS	Map-Reduce
Scalability	Scale UP	Scale Out
Data Size	GB	PB and more
Read/Write	Batch and Interactive	Batch
Update Type	Write many Read many	Write once Read many
Integrity	High	Low
Structure	Structural data/ Schema first Write later	Non Structural data/ Write first Schema later
Query	SQL	No SQL and SQL support too with add on tools
Response time	Faster for Less data/ slow once size increases	Faster for more data in comparison
Note: Slowly with the new sub projects being developed for Hadoop this gap is being filled up as people are developing an abstract layer on top of Map-Reduce and YARN frameworks which takes SQL and in turn convert it to Map-Reduce/Yarn.

Extract All Tar Files in a directory in Linux

This will first list the files containing extension tar.gz, and then awk will get the file names which is column 9 in

ls -lrth command, and

NF > 2 will remove the blank line and

tar -xvzf will extract files names contained in variable $i,

Like this we can experiment various operations like renaming all files with specific extension of so and can fiddled with to achieve various goals.


for i in `ls -lrth *.tar.gz |awk  'NF>2 {print $9}'`; do tar -xvzf $i; done

All about hadoop Balancer.

Hadoop Data Balancing

Hadoop Balancer:

This is tool provided to balance the disk uses throughout the Hadoop cluster. I may happen sometime that some of the nodes in the cluster becomes over utilized or underutilized, which occurs due to addition of new nodes where newly added nodes may be underutilized or if there are less number of nodes result in overutilization. We can run balancer from more than 1 machine in the cluster to increase the speed of balancing but it will increase bandwidth uses to very high.

This tool requires administrator right on the Hadoop cluster to run.

Syntax of the balancer:

bin/start-balancer.sh [-threshold <threshold>]

Where start-balancer.sh files resides in the bin directory of the Hadoop folder. And the threshold is the parameter which decides target of balance, this lies in fraction between 0,1 the default value is 10% if nothing is passed as the threshold value.

This process does the transferring of blocks between the nodes resulting network activity and if a production cluster must be used cautiously, as it result in some block missing error or late reply from the cluster.

This process can be stopped any time if required using following command:

WPF Architecture

WPF architecture is multilayered architecture.  It has three layers mainly Managed code, Unmanaged code and  Core Operating system. We can call these layers as set of assemblies that built up the entire framework.

The major components are below Presentation Framework, Presentation Core and Media Integration(Milcore)  are the major components of wpf architecture. 

 

Limit Disk uses in Datanodes Hadoop

There may in some scenario when disk attached to data node may go over utilize and you become unable to perform any operation on the data node due to no space left on the system. so we have a option of defining a limit of space which can be used by data node daemons.
Using following configuration:

<property>
 <name>dfs.datanode.du.reserved</name>
 <value>182400</value>
 <description>Reserved space in bytes per volume. This defines to leave this much space free for non dfs use.</description>
</property>

Reading sequence file/ compressed file / or TextRecordInputStream file from hadoop

As we know sequence file are binary files with key/value pair specially build for hadoop. If we want to read file on hadoop we have option of cat which will not show the output of sequence or compressed file correctly in this case we can use -text option with hadoop command which follows:

hadoop fs -text   <file path name/filename>

which will show they correct output of the sequence file content.