Reset root password in Linux

1. Reboot the system

2. Press any key while your linux box boot up, it will bring you the the Grub loader
There will be image name, select that

3. Press 'e'

4. It will enter into grub boot screen

5. It will be containing following list

• · Image name
• · Kernel Version
• · MBR

6. Select Kernel line and press 'e' it will take you to edit mode

7. Put your curser to the last of the resulting line and type (single/1/s)

8. Save it

9. After saving it you will again to the step 5

10. Here you press ‘b’ this will boot up the system(by selecting the kernel version line)

11. This will take you to the single user mode

12. Then type ‘passwd’ it will not ask you the old password, will ask you to type new password,

13. Type the new password its done.

View file sise sorted, file size/folder size sorted in size

du -sm * | sort -nr --> Decending

du -sm * | sort -n  --> Ascending

du -sc * | sort -nr --> Total size of current folder

du -sm * | sort -nr | head -10 --> First 10 : Top 10 Largest Items

du -sm * | sort -nr | tail -10 --> Last 10

ls -ltr | grep [0-9][0-9][0-9]M/G  -->  will show files withree digit number in MB or GB

FAILED FSError: java.io.IOException: No Space left on device

 

FAILED FSError: java.io.IOException: No Space left on device

org.apache.hadoop.util.DiskChecker$DiskErrorException : Could not find any valid local directory for taskTracker/jobcache

When you run hive queries and you queries fails, just go the the job tracker of Hadoop : mostly on 50030 port on your Hadoop nodes, and check the failed or killed jobs, you may find these errors.

The reason behind will be following:

Shell script: Using FOR loop

Here's a simple shell script to print numbers using for loop.

Just got through:

# cat test.sh

for ((i=0;i<=10;i++))
do

 echo $i

done


Output:

Do you need easier way to view or manipulate the data saved in the - HBase

Here i found two projects that will connect to HBase and will let you to view and manipulate data in HBase. Just though of sharing with you all.

Automatically Finding files of specific size of type or modified date and adding to tar

Refer following command for your specific need:

#All files of size greater thab 1Gb

find / -type f -size +1048576 | xargs tar -czf myfile.tgz

#All files modified one day before

find / -type f -mtime -1 | xargs tar -czf myfile.tgz

#All files modified one day before and size more than 1Gb

find / -type f -mtime -1 -size +1048576 | xargs tar -czf myfile.tgz

------------------------ OR ------------------------------------

Password-less authentication userful for Hadoop Administrators and linux users

Although Hadoop never requires any password-less authentication to communicate between nodes, but from Hadoop administrator perspective it provide a great flexibility while managing multiple nodes together.

a. Generate the SSH private/public key pair for hadoop user in Namenode

i. ssh-keygen –t rsa

Linux : Prefix a string to all rows in a file - sed

Consider, you have a file like below:

# cat test.txt
12324
234523
3431
343
34345
3434
43234

Now, if you want to prefix a particular string to each line in the file, how will you do it? This is a common challenge for DBA/Developers working with Linux environment.

Linux: Comparing 2 files - 'comm'

Many a times, i encountered the situation wherein I would be having 2 files with some values in each. And I need to get common records in both the files and also, unique records for each files.

So, this is how  I do - one of the easiest way.

Below are the files to be compared:

MySQL Replication: Exclude an SQL from replicating

If you're using MySQL Replication, in rare cases you may require to skip a particular SQL statement from replicating - which is to be executed on Master and should not be on the Slave. Though, its not a good practice.

However, here's how to do it. 

MySQL has a variable sql_log_bin, by default its value will be 'ON' or '1', when we enable replication. That means, it replicates all the SQL statements from Master to Slave.
So, if we want to disable this for a particular SQL, do as below:

SET SESSION sql_log_bin=0; sql statement;SET SESSION sql_log_bin=1;


Example:

 
SET SESSION sql_log_bin=0; DELETE FROM tab1 WHERE col1='NULL'; 
SET SESSION sql_log_bin=1;

 However, after MySQL 5.5.5 version, no need to use SESSION. By default its a SESSION variable and it won't affect other user's transactions. 

Note: Executing this on Master would cause data inconsistency. Don't try this unless you are sure about what you are doing!

Top Command Options for Admnistrators

There are lot of option with top command but some parameters which i found very useful are following :


1. Type top on terminal you will get interface something like :




So mostly What do we need to see??

Memory wise sorted processes
CPU utilization wise sorted process and 
The process name which is using the memory and the CPU Right?

Linux: Add numbers in a column in a file

In windows we use Excel to add all the numbers in a column. But how about Unix/Linux?
We encountered such an requirement and we got the answer too.
Actually solution in much more simpler than the problem..!

Mission accomplished by 'awk' command. Its merely a very powerful and useful command.
Here is an example for it:

Convert image to byte array and byte array to image file

This code will convert an image file to byte array and will write it into a binary file, and will read the binary file and converti it to image file.

import java.awt.image.BufferedImage;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.BufferedInputStream;
import javax.imageio.ImageIO;


public class conveter {

 /**
  * @param args
  */
 public static void main(String[] args) {
  try {
    
   byte[] imageInByte;
   BufferedImage originalImage = ImageIO.read(new File(
     "Chrysanthemum.jpg"));
 

MySQL Error: Error_code: 1032

Here's something about MySQL replication error. The exact error could be as below:

Could not execute Update_rows event on table DBName.tableName; Can't find record in 'tableName', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.01, end_log_pos 218

If SQL Thread on one of your slave stops with this error code, that just mean Master and this Slave are not in sync!! There must be an inconsistency. 

In our case, slave got an Update event form master, but that particular record was not available in the slave. (Of course, that was deleted on Master n Slave separately)
You would get this error only if the binlog_format is set to ROW_BASED or MIXED mode.

So, now check that particular binlog at that position where replication stopped with this error. 
When you convert the binlog with mysqlbinlog command, you may see some junk characters  where you were expecting some DMLs which caused the error(In our case its an Update statement).

So you can use below command to translate binlog completely:


mysqlbinlog --base64-output=DECODE-ROWS --verbose  mysql-bin.01 >mysql-bin.01.txt

mysql: unknown variable 'bind=localhost'

If you are getting this error open /etc/mysql/my.conf and comment the line

bind =127.0.0.1

or

bind =localhost


what ever in your case and you will be able to get mysql prompt.

Changing Replication factor in hadoop on fly

Changing Replication factor in Hadoop on fly, the may be many time when you need to define replication factor of a specific file or directory on the fly, so following command will help you to change it.

The command used for this is "setrep"

How will you do it?

Solution :

Custom Checkpoint, Performing check pointing in hadoop hadoop, Edit log size hadoop, edit log size more hoadoop

Some time it may happen that checkpoint will not happen, and edit size becomes more, of if check point is not happening the is a chance of data loss, if something bad happens, for this

Verify Secondry namenode is up and running fine

• check for services ps -ef | grep secondary\
• check if log rotation of secondary name node is happening properly

 Solution :

Find and remove files older than 'n' days !

Sometimes we schedule a particular daily job in cron, which generates daily log files. Though the size is smaller, we can't keep those files. In that case we need a shell command to clean up and here it is:

To delete all the files whose data modified before 'n' days:

find -mtime +n -delete    


To delete all the files whose data modified in last 'n' days:

find -mtime -n -delete    


To delete files of a particular patterns, say 'archive_date.csv'

find -name 'archive_*.csv' -mtime +n -delete    


To list out all files older than 'n' days along with date and time, just like 'ls' command:

find -mtime +n -ls   


These are simple commands, but very useful. 
Modify as per your need.
Thanks!