Monday, June 19, 2017

Get only matching words using Grep in a command Linux, Shell command

We can use Grep to get the fields or string which contains the specified character, word or string.

cat filename|grep -oh "\w*@\w*"

so suppose a test file contains:

2017-06-19 01:04:49 trigproc man-db:amd64 2.7.5-1 <none>
2017-06-19 01:04:49 status half-configured man-db:amd64 2.7.5-1
2017-06-19 01:04:50 status installed man-db:amd64 2.7.5-1
2017-06-19 01:04:50 configure vim-runtime:all 2:8.0.0095-1ubuntu3 <none>
2017-06-19 01:04:50 status unpacked vim-runtime:all 2:8.0.0095-1ubuntu3
2017-06-19 01:04:50 status half-configured vim-runtime:all 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status installed vim-runtime:all 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 trigproc gnome-menus:amd64 3.13.3-6ubuntu4 <none>
2017-06-19 01:04:51 status half-configured gnome-menus:amd64 3.13.3-6ubuntu4
2017-06-19 01:04:51 status installed gnome-menus:amd64 3.13.3-6ubuntu4
2017-06-19 01:04:51 trigproc hicolor-icon-theme:all 0.15-1 <none>
2017-06-19 01:04:51 status half-configured hicolor-icon-theme:all 0.15-1
2017-06-19 01:04:51 status installed vim-common:all 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status installed hicolor-icon-theme:all 0.15-1
2017-06-19 01:04:51 configure vim-tiny:amd64 2:8.0.0095-1ubuntu3 <none>
2017-06-19 01:04:51 status unpacked vim-tiny:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status unpacked vim-tiny:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status half-configured vim-tiny:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status installed vim-tiny:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 configure vim:amd64 2:8.0.0095-1ubuntu3 <none>
2017-06-19 01:04:51 status unpacked vim:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status half-configured vim:amd64 2:8.0.0095-1ubuntu3
2017-06-19 01:04:51 status installed vim:amd64 2:8.0.0095-1ubuntu3


in a file /tmp/test then if we want to get all word with ubun we can give the command as

grep -oh "\w*ubun\w*" /tmp/test

will give this word from all lines.

Wednesday, May 17, 2017

Find file older than n days with full absolute path

This command will find the files log, out and txt file as in this example which is older than n days which is defined in parameter -mtime and print the absolute path of the files found.

find `pwd` -maxdepth 1 -mtime +n -type f \( -name "*.log*" -o -name "*.out*" -o -name "*.txt" \)

Examples:

Find and print absolute path of the files which are 7 days or 30 days old

find `pwd` -maxdepth 1 -mtime +7 -type f \( -name "*.log*" -o -name "*.out*" -o -name "*.txt" \)

find `pwd` -maxdepth 1 -mtime +30 -type f \( -name "*.log*" -o -name "*.out*" -o -name "*.txt" \)

We can play with -mtime parameter to get other desired result

Friday, May 5, 2017

List or See MySQL command or query history: MySql Command Or Query History : Command Line

MySql store the history of command or queries in .mysql_history file in the user directory, suppose you want to see the history of commands or SQL queries which <username> has executed we can either login to that user or from another user we can type:

cat ~/.mysql_history

Suppose current user is logged in as root or any user if it has Sudo permission then we can give command

cat /home/username/.mysql_history

Wednesday, May 3, 2017

Configuring freeIPA Kerberos in Cloudera Manager : Cloudera Manager + freeIPA + Kerberos

  1. Create a user in free IPA using command line or WebUI
    • Suppose this user is Hadoop
  2. Now add this user to admin and trust user group in IPA user groups.
  3. Generate a Kerberos ticket for this created user, run this command at the server where Cloudera Manager is hosted or installed.
    • ipa-getkeytab -s <your ipa server name> -p <hadoop@your-realm.name> -k keytabfile.keytab 
    • e.g.:
      • ipa-getkeytab -s ipa.server.com -p hadoop@SERVER.COM -k cdh.keytab
    • kinit with the the user hadoop
      • kinit hadoop@SERVER.COM
      • It will ask you the password; pass the password set while creating the keytab.
      • It will now ask you to change the password, set a new password
      • Use kdestroy to remove the hadoop user cache with old password.
      • again do kinit with new password
      • kinit hadoop@SERVER.COM
      • Again get the keytab files using command, in order to get the latest keytab file with update password.
      • ipa-getkeytab -s ipa.server.com -p hadoop@SERVER.COM -k cdh.keytab
  4. Copy this keytab file /etc/cloudera-scm-server directory of any path which should be accessible to Cloudera Manager.
  5. Once you get the cdh.keytab file change its ownership to cloudera-scm user
    • chown cloudera-scm: cloudera-scm cdh.keytab
  6. Download the custom keytab retrieval script file from: 
  7. Save this script to some location which should be accessible to Cloudera Manager for example:
  8. The parameter which you need to change in this file follows:
    • IPA_SERVER=ipa.server.com
    • REALM=SERVER.COM
    • KEYTAB_FILE=/etc/cloudera-scm-server/cdh.keytab
    • CM_PRINC=hadoop@$REALM
    • There is a line near line number 39 in this script which says
      • ipa service-add $PRINCIPAL
    • Make change to this like add --force at the end of this line(This is use to override the restriction in freeIPA) then line will look like
      • ipa service-add $PRINCIPAL  --force
  9. Login the server freeIPA is installed/login to freeIPA for example ipa.server.com
      • Start kadmin.local as
      •  kadmin.local -x ipa-setup-override-restrictions
      •  This will login as root/admin@SERVER.COM
  10.  Add a new principle for Cloudera-manager here
      •  addprinc –pw <passwordforuser> cloudera-scm/admin@SERVER.COM
      • e.g.:
      • addprinc –pw cdhpassword Cloudera-scm/admin@SERVER.COM
  11. Once this principle is added successfully, go to Cloudera manager Web UI <your-clouderamanage- server:7180>
  12.  Go to administration security/ or from the cluster dropdown menu choose enable Kerberos
  13. In the resulting page enter the required details, and in the page where it says KDC account manager credentials enter
      • First text box Username : cloudera-scm/admin
      •  2nd Text box Your REALM : SERVER.COM
      • Third Box Password : passwordforuser
  14.  Once this step successes move forward to generate other service principal which Cloudera Manager will generate automatically if this step is successful.


Hope this will help.


Friday, September 4, 2015

Change desktop screen size using terminal linux inside VM



--To--






Change desktop screen size using terminal linux inside VM

xrandr -s 1440x900

xrandr -s <Resolution which you want to change>

you can use

xrandr --verbose 

option to get the available screen resolution available.

Option with xrandr :


Wednesday, September 2, 2015

Install GNOME Desktop Ubuntu

Using following command we can install and enable desktop GUI for Ubuntu server version if needed.


sudo apt-get install ubuntu-gnome-desktop
sudo service gdm restart

Tuesday, September 1, 2015

org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

Add following entries to yarn-site.xml if not present and restar yarn service.


<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>


<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Sunday, August 30, 2015

Enable passwordless sudoers for a user

We can add following line in /etc/sudoers to enable the user to be passwordless login from other machines.


<username>  ALL=(ALL:ALL) NOPASSWD: ALL

Monday, August 24, 2015

Comparison between RDBMS and Map-Reduce



RDBMS
Map-Reduce
Scalability
Scale UP
Scale Out
Data Size
GB
PB and more
Read/Write
Batch and Interactive
Batch
Update Type
Write many Read many
Write once Read many
Integrity
High
Low
Structure
Structural data/ Schema first Write later
Non Structural data/ Write first Schema later
Query
SQL
No SQL and SQL support too with add on tools
Response time
Faster for Less data/ slow once size increases
Faster for more data in comparison
Note: Slowly with the new sub projects being developed for Hadoop this gap is being filled up as people are developing an abstract layer on top of Map-Reduce and YARN frameworks which takes SQL and in turn convert it to Map-Reduce/Yarn.

Tuesday, June 16, 2015

Extract All Tar Files in a directory in Linux

This will first list the files containing extension tar.gz, and then awk will get the file names which is column 9 in

ls -lrth command, and

NF > 2 will remove the blank line and

tar -xvzf will extract files names contained in variable $i,

Like this we can experiment various operations like renaming all files with specific extension of so and can fiddled with to achieve various goals.


for i in `ls -lrth *.tar.gz |awk  'NF>2 {print $9}'`; do tar -xvzf $i; done