get all the urls in html file (local or on server).

To use this, you will need the lynx tool, so install that first.

sudo yum install lynx

Now, to get list of all the URLs in local html file or some URL, just execute this:

lynx -dump -listonly

 

Enhanced by Zemanta

cgroups – use to control your cpu and memory

cgroups is a kernel feature and with userspace utilities, we can use the feature to control the cpu and memory for per process. So, lets first install the required tools.

sudo yum install libcgroup-tools

Now, we need to enable the service.

sudo systemctl enable cgconfig.service
sudo systemctl enable cgred.service

cgconfig.service is to enable configuration for cgroups and
cgred.service is to enable configuration for cgroups for processes depending on the name.

Now, we will need to configure the cgroups and configure the groups for the processes. So, lets open up the configuration files and do some configuration:

Here we will use firefox as the group name and use firefox as an example to restrict the memory to 700Mb and cpu to 10%. Since we do not have any other group configured right now, so max being 1024 – 100 shares would be around 10%.

First, open up the /etc/cgconfig.conf file and add the following. You would need to replace below with your username.

group firefox {
perm {
task {
uid = ;
gid = ;
}
admin {
uid = ;
gid = ;
}
}

cpu{
cpu.shares=”102″;
}
cpuset{
cpuset.cpus=0;
cpuset.mems=0;
}
memory {
memory.limit_in_bytes=”700M”;
memory.max_usage_in_bytes=”0″;
}
}

Now, we edit the /etc/cgrules.conf file and add the following:

:sandbox cpu,memory firefox
::firefox cpu,memory firefox

All done. Now, time to test.

First restart the services.

sudo systemctl restart cgconfig.service
sudo systemctl restart cgred.service

Check, if the cgroup is created. Go to directory:

/sys/fs/cgroup/memory or /sys/fs/cgroup/cpu

and check directory firefox exists.

Note: You can check the cgroup mount directory with mount command to see where your cgroups are mounted.

Now, start firefox, check the pid and check the file /proc//cgroup and you should see something like this:

11:hugetlb:/
10:perf_event:/
9:blkio:/
8:net_cls:/
7:freezer:/
6:devices:/
5:memory:/firefox
4:cpuacct,cpu:/firefox
3:cpuset:/
2:name=systemd:/user.slice/user-1000.slice/session-1.scope

and this is it.

Enhanced by Zemanta

Apache server stats with small and useful bash script.

Just copy this script to your web-server cgi-bin directory and enjoy.

The script with show the common errors like 404 Error, Internal Server Error and others. It will show the User agent distribution using simple commands like grep, uniq, awk and so on.

You would need to change the tfile – which is temporary file and also the access.log path in the next line.

Just re-direct the output to some file with html extenstion. You could even put this in the cron which re-directs the output to some html in server document root.

#!/bin/bash
tfile=/tmp/apache.log
sudo grep $(date '+%d/%b/%Y') /var/log/apache2/access.log >$tfile
thits=$(grep -c . $tfile)
echo "Content-type: text/html"
echo ""
echo "Report for $(date '+%d/%b/%Y')

Report for $(hostname) on $(date ‘+%d/%b/%Y’)

"

echo "Total hits :: $thits"
echo ""
echo "User agent distribution :: "
#awk -F" '{print $6}' $tfile  | sort | uniq -c | sort -fr

awk -F" '{print $6}' $tfile | sed 's/(([^;]+; [^;]+)[^)]*)/(1)/' |sort |uniq -c|sort -fr

echo "
User response code: "
awk 'BEGIN{
a[200]="OK";
a[206]="Partial Content";
a[301]="Moved Permanently";
a[302]="Found";
a[304]="Not Modified";
a[401]="Unauthorised (password required)";
a[403]="Forbidden";
a[404]="Not Found";
a[500]="Internal Server Error";
}
{print $9 " => "a[$9]""}' $tfile | sort | uniq -c | sort -nr

echo "
404 Error Summary::"
awk '($9 ~ /404/)' $tfile | awk '{print $9,$7}' | sort | uniq -c|sort -nr|head -5

echo "
IP Visit counts :: "
awk '{print $1}' $tfile | grep -o "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" | sort -n | uniq -c | sort -nr|while read count ip
do
	name=$(echo $ip|/usr/bin/logresolve)
	printf "%5st%-15st%sn" $count $ip $name
done

# cat $tfile |grep -o "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" | sort -nr | uniq -c | sort -n

echo "
Top Agents :: "
cat $tfile | awk -F" '{print $6}'| sort -n | uniq -c | sort -nr |head -5 

echo "
Top urls ::"
cat $tfile  | awk -F" '{print $2}'| sort -n | uniq -c | sort -nr |head -5

echo -n "
Total Bytes ::: "
cat $tfile | awk '{ sum += $10 } END { if ( sum > 1024*1024) {print sum/(1024*1024)"Mb"}else if ( sum > 1024) {print sum/1024"Kb";}else print sum }'

echo -n "
Total Seconds :: "
cat $tfile  | awk '{ sum += $13 } END { print sum }'
# sed 's/.*GET (.*) HTTP.*/1/g' $tfile|awk -F/ '{if ( NF > 3 ) print $2"/"$3"/"$4; else print $2;}'|sort|uniq -c

Apache Server Stats – Download the script in zip file format.

And here is a sample ::

Report for <server> on <date>

Total hits :: 

User agent distribution :: 

User response code: 

404 Error Summary::

IP Visit counts :: 

Top Agents :: 

Top urls ::

Total Bytes ::: 

Total Seconds ::

Hope that it will be useful.

Enhanced by Zemanta