Archive

Archive for January, 2015

Timezone Update in Ubuntu 12.04 LTS

January 29, 2015 Leave a comment
Changing Timezone in Ubuntu server.
Step 1: Check
 ahmed@server:~# date
Thu Jan 29 02:38:55 EST 2015
ahmed@server:~# more /etc/timezone
US/Eastern
Step 2: Change Timezone Setting
 ahmed@server:~# dpkg-reconfigure tzdata

Current default time zone: 'Asia/Kolkata'
Local time is now: Thu Jan 29 13:09:54 IST 2015.
Universal Time is now: Thu Jan 29 07:39:54 UTC 2015.

ahmed@server:~# date
Thu Jan 29 13:09:57 IST 2015
ahmed@server:~#
Step 3: Also make sure to restart cron service as this will still have the old time.
 ahmed@server:~# service crond restart
ahmed@server:~# service nginx restart
And we are done.
Categories: HOWTOs

Hadoop `sysctl.conf` parameters

January 28, 2015 Leave a comment
Hadoop sysctl.conf parameters.
File System
  1. fs.file-max Increase size of file handles and inode cache
 [ahmed@server ~]# echo 'fs.file-max = 943718' >> /etc/sysctl.conf
Swappiness : Do less swapping
  1. vm.dirty_ratio setting virtual memory ratio.
  2. vm.swappiness How often swap should be used. 0 is least, 60 default.
 [ahmed@server ~]# echo 'vm.dirty_ratio=10' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'vm.swappiness=0' >> /etc/sysctl.conf
Connection Settings
  1. net.core.netdev_max_backlog Increase number of incoming connections backlog.
  2. net.core.somaxconn Increase number of incoming connections.
 [ahmed@server ~]# echo 'net.core.netdev_max_backlog = 4000' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.core.somaxconn = 4000' >> /etc/sysctl.conf
TCP settings
  1. net.ipv4.tcp_sack Disable select acknowledgments
  2. net.ipv4.tcp_dsack Allows TCP to send “duplicate” SACKs.
  3. net.ipv4.tcp_keepalive_time How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours.
  4. net.ipv4.tcp_keepalive_probes How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.
  5. net.ipv4.tcp_keepalive_intvl How frequently the probes are send out. Multiplied by tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries.
  6. net.ipv4.tcp_fin_timeout Time to hold socket in state FIN-WAIT-2, if it was closed by our side. Peer can be broken and never close its side, or even died unexpectedly. Default value is 60sec. Usual value used in 2.2 was 180 seconds, you may restore it, but remember that if your machine is even underloaded WEB server, you risk to overflow memory with kilotons of dead sockets, FIN-WAIT-2 sockets are less dangerous than FIN-WAIT-1, because they eat maximum 1.5K of memory, but they tend to live longer. Cf.tcp_max_orphans.
  7. net.ipv4.tcp_rmem The three values setting the minimum, initial, and maximum size of the Memory Receive Buffer per connection. They define the actual memory usage, not just TCP window size.
  8. net.ipv4.tcp_wmem The same as tcp_rmem, but just for Memory Send Buffer per connection.
  9. net.ipv4.tcp_retries2 This value influences the timeout of an alive TCP connection, when RTO retransmissions remain unacknowledged. Given a value of N, a hypothetical TCP connection following exponential backoff with an initial RTO of TCP_RTO_MIN would retransmit N times before killing the connection at the (N+1)th RTO. The default value of 15 yields a hypothetical timeout of 924.6 seconds and is a lower bound for the effective timeout. TCP will effectively time out at the first RTO which exceeds the hypothetical timeout. RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.
  10. net.ipv4.tcp_synack_retries Number of times SYNACKs for a passive TCP connection attempt will be retransmitted. Should not be higher than 255. Default value is 5, which corresponds to ~180seconds.
 [ahmed@server ~]# echo 'net.ipv4.tcp_sack = 0' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_dsack = 0' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_time = 600' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_probes = 5' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_intvl = 15' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_fin_timeout = 30' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_rmem = 32768 436600 4194304' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_wmem = 32768 436600 4194304' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_retries2 = 10' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_synack_retries = 3' >> /etc/sysctl.conf
Disable IPv6 Defaults. We dont use these anyway.
 [ahmed@server ~]# echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv6.conf.default.disable_ipv6 = 1' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv6.conf.lo.disable_ipv6 = 1' >> /etc/sysctl.conf
Execute below command to make it permanent.
 [ahmed@server ~]# sysctl -p
Next update limits.
 [ahmed@server ~]# echo '* - nofile 65536' >>/etc/security/limits.conf
[ahmed@server ~]# echo '* - nproc 65536' >>/etc/security/limits.conf
More Details on IPv4
 http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/ip-sysctl.txt
http://www.cyberciti.biz/files/linux-kernel/Documentation/sysctl/
Categories: HOWTOs

Simple Script to Create Users on CentOS 6.5

January 27, 2015 Leave a comment
Simple Script to create a user.
Default passwd is set as username@123
As we are setting the chage, user needs to change the passwd on first login.
usage() {
echo -e "
usage: $0
Optional parameters:\n
--sudo-user, -s\t\tCreate a sudo User
--norm-user, -n\t\tCreate a Normal User.
--help, -h\t\tDisplay this Message.
"
exit 1
}

create_normal_user()
{
USRNAME=$1
# create user
useradd $USRNAME
echo ${USRNAME}@123 | passwd ${USRNAME} --stdin

# make sure user changes his passwd on first login
chage -d 0 ${USRNAME}
}

create_sudo_user()
{
SUDO_USRNAME=$1
create_normal_user ${SUDO_USRNAME}
echo -e "${SUDO_USRNAME}\tALL=(ALL)\tALL" >> /etc/sudoers
}

while true ; do
case "$1" in
-s)
create_sudo_user $2
exit 1
;;
-n)
create_normal_user $2
exit 1
;;
--sudo-user)
create_sudo_user $2
exit 1
;;
--norm-user)
create_normal_user $2
exit 1
;;
-h)
usage
exit 1
;;
--help)
usage
exit 1
;;
*)
echo "Unknown option: $1"
usage
exit 1
esac
done
Categories: HOWTOs

Setting SUN Java for Ubuntu.

January 27, 2015 Leave a comment
First create directory for Java
 ahmed@ahmed-server:~/sun-java# mkdir -p /usr/lib/jvm/
ahmed@ahmed-server:~/sun-java# tar xvzf jdk1.7.0_75.tgz -C /usr/lib/jvm/
Setting Alternatives
 ahmed@ahmed-server:~/sun-java# sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_75/bin/java" 1
ahmed@ahmed-server:~/sun-java# sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_75/bin/javac" 1
ahmed@ahmed-server:~/sun-java# sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_75/bin/javaws" 1
update-alternatives: using /usr/lib/jvm/jdk1.7.0_75/bin/javaws to provide /usr/bin/javaws (javaws) in auto mode.
Make sure we have the right permission.
 ahmed@ahmed-server:~/sun-java# sudo chmod a+x /usr/bin/java
ahmed@ahmed-server:~/sun-java# sudo chmod a+x /usr/bin/javac
ahmed@ahmed-server:~/sun-java# sudo chmod a+x /usr/bin/javaws
ahmed@ahmed-server:/usr/lib/jvm# sudo chown -R root:root /usr/lib/jvm/jdk1.7.0_75
Configuration of Alternatives
 ahmed@ahmed-server:/usr/lib/jvm# sudo update-alternatives --config java
There are 3 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 auto mode
1 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 manual mode
2 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1051 manual mode
3 /usr/lib/jvm/jdk1.7.0_75/bin/java 1 manual mode

Press enter to keep the current choice[*], or type selection number: 3
update-alternatives: using /usr/lib/jvm/jdk1.7.0_75/bin/java to provide /usr/bin/java (java) in manual mode.

ahmed@ahmed-server:/usr/lib/jvm# sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1051 auto mode
1 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1051 manual mode
2 /usr/lib/jvm/jdk1.7.0_75/bin/javac 1 manual mode

Press enter to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/jvm/jdk1.7.0_75/bin/javac to provide /usr/bin/javac (javac) in manual mode.
ahmed@ahmed-server:/usr/lib/jvm# sudo update-alternatives --config javawc
update-alternatives: error: no alternatives for javawc.
Checking version
 ahmed@ahmed-server:/usr/lib/jvm# java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
ahmed@ahmed-server:/usr/lib/jvm# javac -version
javac 1.7.0_75
ahmed@ahmed-server:/usr/lib/jvm#
Categories: HOWTOs

Installing and Initial setup of Tsung Load Testing CentOS

January 21, 2015 Leave a comment

Installing and Initial setup of Tsung Load Testing CentOS

Installation on Centos

 [ahmed@server ~]$ yum install erlang 
[ahmed@server ~]$ tar -xvzf v1.5.1.tar.gz -C /opt
[ahmed@server ~]$ cd /opt/tsung-1.5.1
[ahmed@server ~]$ ./configure
[ahmed@server ~]$ make
[ahmed@server ~]$ make install
Some Version information.
 [ahmed@server ~]$ tsung -v
Tsung version 1.5.1
[ahmed@server ~]$ tsung
Usage: tsung start|stop|debug|status
Options:
-f set configuration file (default is ~/.tsung/tsung.xml)
(use - for standard input)
-l set log directory where YYYYMMDD-HHMM dirs are created (default is ~/.tsung/log/)
-i set controller id (default is empty)
-r set remote connector (default is ssh)
-s enable erlang smp on client nodes
-p set maximum erlang processes per vm (default is 250000)
-m write monitoring output on this file (default is tsung.log)
(use - for standard output)
-F use long names (FQDN) for erlang nodes
-w warmup delay (default is 1 sec)
-v print version information and exit
-6 use IPv6 for Tsung internal communications
-x list of requests tag to be excluded from the run (separated by comma)
-h display this help and exit
[ahmed@server ~]$
Sample Test load.xml update load_test_machine and web_server_to_test as per the servers. 
 




















<http url="/postdata/Information"
method="POST"
contents_from_file="test.json" />




Test Sample Json
 {
"name":"ahmed",
"age":30
}

Next Execute the command to start the service .

 tsung -f load.xml start 
This will start the service which will start hitting the server.
All logs will be available in ${HOME}/.tsung/log
More information
 https://engineering.helpshift.com/2014/tsung/
Categories: HOWTOs

Load Testing ‘siege’ – Install and Usage

January 20, 2015 Leave a comment

Installation Procedure.

First install prerequisites gcc/make on server. If Ubuntu
 sudo apt-get install gcc make
On Centos/Redhat
 sudo yum install gcc make
Step to Installation.
 $ ./configure
$ make
$ sudo make install
Files Installed.
 siege          -->    SIEGE_HOME/bin/siege
bombardment --> SIEGE_HOME/bin/bombardment
siege2csv --> SIEGE_HOME/bin/siege2csv
.siegerc --> $HOME/.siegerc
siege.1 --> SIEGE_HOME/man/man1/siege.1
bombardment.1 --> SIEGE_HOME/man/man1/bombardment.1
siege2csv.1 --> SIEGE_HOME/man/man1/siege2csv.1
layingsiege.1 --> SIEGE_HOME/man/man1/layingsiege.1
urls_text.1 --> SIEGE_HOME/man/man1/urls_txt.1
urls.txt --> SIEGE_HOME/etc/urls.txt
Usage for siege command.
 root@SIDCLB:~# siege
SIEGE 3.0.9
Usage: siege [options]
siege [options] URL
siege -g URL
Options:
-V, --version VERSION, prints the version number.
-h, --help HELP, prints this section.
-C, --config CONFIGURATION, show the current config.
-v, --verbose VERBOSE, prints notification to screen.
-q, --quiet QUIET turns verbose off and suppresses output.
-g, --get GET, pull down HTTP headers and display the
transaction. Great for application debugging.
-c, --concurrent=NUM CONCURRENT users, default is 10
-i, --internet INTERNET user simulation, hits URLs randomly.
-b, --benchmark BENCHMARK: no delays between requests.
-t, --time=NUMm TIMED testing where "m" is modifier S, M, or H
ex: --time=1H, one hour test.
-r, --reps=NUM REPS, number of times to run the test.
-f, --file=FILE FILE, select a specific URLS FILE.
-R, --rc=FILE RC, specify an siegerc file
-l, --log[=FILE] LOG to FILE. If FILE is not specified, the
default is used: PREFIX/var/siege.log
-m, --mark="text" MARK, mark the log file with a string.
-d, --delay=NUM Time DELAY, random delay before each requst
between 1 and NUM. (NOT COUNTED IN STATS)
-H, --header="text" Add a header to request (can be many)
-A, --user-agent="text" Sets User-Agent in request
-T, --content-type="text" Sets Content-Type in request
For more detailed information, consult the man pages:
 $ man siege
$ man layingsiege
$ man siege.config
All the siege man pages are also available online:
 http://www.joedog.org/siege/docs/man/index.html
OR, read the html manual, doc/manual.html The manual is also available online:
 http://www.joedog.org/siege/docs/manual.html 

Usage Procedure

Basic Commands

To test the GET request, you would run
 siege http://example.com/ -c 100 -r 100
To test the POST request, you would run
 siege -H 'Content-Type:application/json' "http://example.com/ POST < ./data.json" -c 10 -r 1000
Categories: HOWTOs

How To Configure Swappiness Linux

January 15, 2015 Leave a comment
How To Configure Swappiness
To find the current swappiness settings, type:
cat /proc/sys/vm/swappiness
60
Swapiness can be a value from 0 to 100.
  1. Swappiness near 100 means that the operating system will swap often and usually, too soon.
  2. Although swap provides extra resources, RAM is much faster than swap space. Any time something is moved from RAM to swap, it slows down.
  3. If we have more RAM on the server its better to leave the swappiness to a lower value like 10.
  4. Also if we are using the server as a Hadoop Datanode and has enough RAM on hand then we set it lower value.
A swappiness value of 0 means that the operating will only rely on swap when it absolutely needs to. We can adjust the swappiness with the sysctl command:
sysctl vm.swappiness=10
vm.swappiness=10
If we check the system swappiness again, we can confirm that the setting was applied:
cat /proc/sys/vm/swappiness
10
To make changes permanent, you can add the setting to the /etc/sysctl.conf file:
sudo nano /etc/sysctl.conf
Add the below line.
# Search for the vm.swappiness setting.  Uncomment and change it as necessary.
vm.swappiness=10
Categories: HOWTOs