Archive

Archive for October, 2014

Fetch Monitoring Data from Item to CSV file.

October 20, 2014 Leave a comment
Have done some minor changes as per my need.
This script gets monitoring data from Items from Zabbix server and creates a CSV file. For a range of time given.
Time range is given in ‘yyyy-mm-dd hh:mm:ss’ format.
For the range we give start datetime and end datetime,
if only start datetime is specified then time period will be start datetime to now()
Script can do 2 things:
  1. Single key data into csv
  2. Multiple keys read from a file and creates a single csv, with all the keys (creates a .csv file).

Single key data into csv

This option will connect to Zabbix and collection a single key monitoring data.
CSV file is created with the key_name.csv format.
python -s 127.0.0.1 -h host-in-zabbix -u admin -p zabbix -k key-in-zabbix -t1 '2014-10-16 12:00:00' -v 1 

Multiple keys read from a file.

This option will connect to Zabbix and collection a multiple key monitoring data.
CSV file is created with the hostname.csv format.
A Text file or a zabbix export xml file can be use to get multiple keys from zabbix.
python -s 127.0.0.1 -h host-in-zabbix -u admin -p zabbix -f sample_key_file.txt -t1 '2014-10-16 12:00:00' -v 1 
python -s 127.0.0.1 -h host-in-zabbix -u admin -p zabbix -f sample_zabbix_export_file.xml -t1 '2014-10-16 12:00:00' -v 1

Sample Output CSV File.

CSV file created is in below format.
key timestamp value
TestKeyFromZabbix_VS.dlBytes 2014-10-14 12:00:00 0
TestKeyFromZabbix_VS.dlBytes 2014-10-14 12:05:00 0
TestKeyFromZabbix_VS.dlBytes 2014-10-14 12:10:00 0
TestKeyFromZabbix_VS.dlBytes 2014-10-14 12:15:00 3517
Here is a raw output. Rather a semicolon separated values (SSV).
key;timestamp;value
TestKeyFromZabbix_VS.dlBytes;2014-10-14 12:00:00;0
TestKeyFromZabbix_VS.dlBytes;2014-10-14 12:05:00;0
TestKeyFromZabbix_VS.dlBytes;2014-10-14 12:10:00;0
TestKeyFromZabbix_VS.dlBytes;2014-10-14 12:15:00;3517

Usage

usage: fetch_monitoring_data_to_csv.py [-h] -s SERVER_IP -n HOSTNAME
(-k KEY | -f FILE) -u USERNAME -p
PASSWORD [-o OUTPUT] -t1 DATETIME_START
[-t2 DATETIME_END] [-v DEBUG_LEVEL]

Fetch history from aggregator and save it into CSV file

optional arguments:
-h, --help show this help message and exit
-s SERVER_IP, --server-ip SERVER_IP
Server IP address
-n HOSTNAME, --hostname HOSTNAME
Hostname of the VM
-k KEY, --key KEY Zabbix Item key
-f FILE, --file FILE Zabbix Item key File. XML export file from Zabbix.
Text file with key in each line.Each Key will create
its own csv file.
-u USERNAME, --username USERNAME
Zabbix username
-p PASSWORD, --password PASSWORD
Zabbix password
-o OUTPUT, --output OUTPUT
Output file name, default key.csv (will remove all
special chars)
-t1 DATETIME_START, --datetime-start DATETIME_START
Start datetime, use this pattern '2014-10-15 19:12:00'
if only t1 specified then time period will be t1 to
now()
-t2 DATETIME_END, --datetime-end DATETIME_END
end datetime, use this pattern '2014-10-15 19:12:00'
-v DEBUG_LEVEL, --debug-level DEBUG_LEVEL
log level, default 0

Code Usage

Single key data into csv

from fetch_monitoring_data_to_csv import fetch_monitoring_data_to_csv

fetch_monitoring_data_to_csv('admin','zabbix', "http://127.0.0.1/zabbix", 'BLR-IN-DEVICE',
'key', '', '2014-10-14 12:00:00', '', 1)

Multiple keys read from a file.

from fetch_monitoring_data_to_csv import fetch_multi_key_monitoring_data_to_csv

fetch_multi_key_monitoring_data_to_csv('admin','zabbix', "http://127.0.0.1/zabbix", 'BLR-IN-DEVICE',
'file.txt', '', '2014-10-14 12:00:00', '', 1)

Code Location

Categories: HOWTOs

Zabbix Item Creation using OID and Name CSV files.

October 20, 2014 Leave a comment
This script is to create a xml import file from a csv file.
We use 2 csv file for this.
  1. First is the OID csv file. This OID file will contain all the range of OIDs which can be configured on the device. OID CSV will be similar to the CSV sample given below.
  2. Information (Name) configured in those OIDs above, with index. This file contains all the configuration information done on the device. Will give the index of the configured variable.

CSV format as below

Below is the OID csv file. (We will called it OID CSV File)
module_name oid_name oid datatype start end description
voip_destination status 1.3.6.1.2.1.2.2.18.0 string 0 1000 some description
voip_destination flags 1.3.6.1.2.1.2.2.18.1 string 0 1000 some description
voip_destination type 1.3.6.1.2.1.2.2.18.2 string 0 1000 some description
Here is the sample csv output for the above file.
module_name,oid_name,oid,datatype,start,end,description
voip_destination,status,1.3.6.1.2.1.2.2.18.0,string,0,1000,some description
voip_destination,flags,1.3.6.1.2.1.2.2.18.1,string,0,1000,some description
voip_destination,type,1.3.6.1.2.1.2.2.18.2,string,0,1000,some description
Below is name (configured) oid information, in the OIDs above. (We will called it NAME CSV File).
We use the name_table.module-name below to join with the table above with oid_table.module_name.
module_name index name
voip_destination 0 test_data_44test
linkset 1 test_data_44tes
ip 0 test_datatest
Here is the sample csv output for the above file.
module_name,TEST_DATAdex,TEST_DATAmTEST
Linkset,0,TEST_DATA_44TEST
Linkset,1,TEST_DATA_44TEST
IP,0,TEST_DATATEST
IP,1,TEST_DATAlTEST
IP,2,TEST_DATAteTEST

USAGE

usage: zabbix_items_from_csv.py [-h] -o CSV_OID -c CSV_NAME -n HOST_NAME -g
HOST_GROUP -i HOST_INTERFACE -a
HOST_APPLICATION [-y] [-f]

This script is to Generate xml import file for Zabbix from CSV files.We need
two CSV files.1. OID file, gives all the OIDs in the device.2. Name file,
gives all names configured for the above OIDs in the Device.

Example : python -o oid_list_with_range_processed.csv -c oid_names_configured.csv -n
GGSN-1-LONDON -g GGSN-GROUP -i 127.0.0.1 -a GGSN-APP-OIDS -y

optional arguments:
-h, --help show this help message and exit
-o CSV_OID, --csv-oid CSV_OID
OID file, Gives all OIDs on the device
-c CSV_NAME, --csv-name CSV_NAME
Name file, gives all names configured for the above
OIDs in the Device.
-n HOST_NAME, --host-name HOST_NAME
Host name as given in Zabbix server.
-g HOST_GROUP, --host-group HOST_GROUP
Host Group which the host belongs to, as in Zabbix
server.
-i HOST_INTERFACE, --host-interface HOST_INTERFACE
SNMP Interface configured on Zabbix server. (Assuming
Single Interface in Configured)
-a HOST_APPLICATION, --host-application HOST_APPLICATION
Application Name in the Zabbix Server. (To organize
all items being imported)
-y, --only-name Create xml items which are present in the name file.
i.e create items which are configured in the device
already, Rest of the OIDs are Ignored. [Default :
False]
-f, --include-first-line
Include First line (Header) in the CSV file input,
[Default : False]

Code Usage :

import zabbix_items_from_csv

#-----------------------
# Limit reading to first 10 Lines.
#-----------------------
complete_list_dict_device_1 = zabbix_items_from_csv.reader_csv_file('oid_list_with_range_processed.csv', 10)
complete_list_name_dict_device_1 = zabbix_items_from_csv.read_csv_name_module('oid_names.csv')
dict_device_1 = zabbix_items_from_csv.merge_csv_data(complete_list_dict_device_1, complete_list_name_dict_device_1, True)
xml_tree_string_device_1 = zabbix_items_from_csv.\
generate_items_xml_file_complete(dict_device_1,
'BLR-DEVICE_1', 'BLR-DEVICE_1',
'10.12.51.11', 'DEVICE_1')
zabbix_items_from_csv.xml_pretty_me('BLR-DEVICE_1.xml', xml_tree_string_device_1)
#-----------------------
xml_tree_string_device_1 = zabbix_items_from_csv.\
generate_items_xml_file_complete(dict_device_1,
'CHN-DEVICE_1', 'CHN-DEVICE_1',
'10.12.51.11', 'DEVICE_1')
zabbix_items_from_csv.xml_pretty_me('CHN-DEVICE_1.xml', xml_tree_string_device_1)


#-----------------------
# Read all the lines from the csv file.
#-----------------------
complete_list_dict_device_2 = zabbix_items_from_csv.reader_csv_file('oid_list_with_range_processed.csv')
complete_list_name_dict_device_2 = zabbix_items_from_csv.read_csv_name_module('oid_names.csv')
dict_device_2 = zabbix_items_from_csv.merge_csv_data(complete_list_dict_device_2, complete_list_name_dict_device_2)
xml_tree_string_device_2 = zabbix_items_from_csv.\
generate_items_xml_file_complete(dict_device_2,
'BLR-DEVICE_2', 'BLR-DEVICE_2',
'12.12.54.66', 'DEVICE_2')
zabbix_items_from_csv.xml_pretty_me('BLR-DEVICE_2.xml', xml_tree_string_device_2)
#-----------------------
xml_tree_string_device_2 = zabbix_items_from_csv.\
generate_items_xml_file_complete(dict_device_2,
'CHN-DEVICE_2', 'CHN-DEVICE_2',
'12.12.52.74', 'DEVICE_2')
zabbix_items_from_csv.xml_pretty_me('CHN-DEVICE_2.xml', xml_tree_string_device_2)

Code Location

Categories: HOWTOs

Zabbix Import File from iReasoning Export CSV.

October 20, 2014 Leave a comment
Module used to generate Zabbix import xml for snmp traps
Below are steps to create a Zabbix Import for Traps.
These traps are captured from the snmptrap function in Zabbix.

Step 1:

Select the Alarms from the iReasoning MIB browser and Export them as CSV.
This Script expect MIB information to be in CSV format.
Export from a iReasoning Browser will generate CSV as below.
"Name","Full Name","OID","Type","Access","Indexes","MIB Module","Description"
We use this information to create snmptrap items and corresponding Trigger in the XML.
Which can be imported directly.

Step 2:

Below are the details to create the xml file from the CSV create above.
python zabbix_snmp_trap_import_from_csv.py    

example: python zabbix_snmp_trap_import_from_csv.py \
-e export_csv_from_ireasoning_mib_browser.csv \
-n GGSN-1-LONDON -g GGSN-GROUP -i 127.0.0.1

usage: zabbix_snmp_trap_import_from_csv.py [-h] -e EXPORT_CSV -n HOST_NAME -g HOST_GROUP -i HOST_INTERFACE
Select the Alarms from the iReasoning MIB browser and Export them as CSV. This
Script expect MIB information to be in CSV format. Export from a iReasoning
Browser will generate CSV as below.
"Name","Full Name","OID","Type","Access","Indexes","MIB Module","Description".
We use this information to create snmptrap items and corresponding Trigger in the XML.
Which can be imported directly.
optional arguments:
-h, --help show this help message and exit
-e EXPORT_CSV, --export-csv EXPORT_CSV
OID file, Gives all OIDs on the device
-n HOST_NAME, --host-name HOST_NAME
Host name as given in Zabbix server.
-g HOST_GROUP, --host-group HOST_GROUP
Host Group which the host belongs to, as in Zabbix
server.
-i HOST_INTERFACE, --host-interface HOST_INTERFACE
SNMP Interface configured on Zabbix server. (Assuming
Single Interface in Configured)

Code location

Categories: HOWTOs

Send KPI Data to Zabbix from XML KPI File

October 20, 2014 Leave a comment
Send Zabbix metric by reading from a XML file.
This needs package zbxsend from : https://github.com/zubayr/zbxsend
  1. Install zbxsend
  2. Install xmltodict
  3. Create items in zabbix using auto_zabbix_export_generator.py script.
  4. Import items to zabbix.
  5. Running the script zabbix_send_from_kpi_xml.py to send the data to zabbix server.

Zabbix Item Creation

Before we send the Data to Zabbix. We need to create Keys (Items) in Zabbix.
Use the script called auto_zabbix_export_generator.py. Script can be found the same location as the sender script.
This script will create a xml file, which can be imported into the zabbix directly.
Below are the usage for the zabbix item xml import generator.

To Generate XML Export file for Zabbix from a Properties file.

This properties file can be generated from the sender script zabbix_send_from_kpi_xml.py as in the option (1) below.
Once created then we can use this file to create specific KPI items import file, for the zabbix server which we want to monitor.
 python ggsn_auto_zabbix_export_generator.py  [ALL]  
example: python ggsn_auto_zabbix_export_generator.py KPI_SOURCE_FILE.xml ALL ZABBIX-TMP DEVICE_X1_2

Create All KPI Key values from the Zabbix server.

This option is used to create import xml for all KPIs items which are present in the kpi xml.
Which we use to create items in the zabbix server.
 python ggsn_auto_zabbix_export_generator.py  [FILE]   
example: python ggsn_auto_zabbix_export_generator.py KPI_SOURCE_FILE.xml FILE KPI_SOURCE_FILE.xml.properties ZABBIX-TMP DEVICE_X1_2
NOTE: items key is created based on the Module_SubCategory from the kpi xml file.

Parameter Information:

       : Is the XML KPI file for Device.
: This is the file which is generated using option (1). '#' is comment in properties file.
: Host name which needs to be created on zabbix.
: Group name to which host needs to be assigned.
[ALL] : Static value. Use as-is.
[FILE] : Static Value. Use as-is.

Below is the Usage for Zabbix KPI Sender.

1. To Generate Properties file.

This is to generate a properties file, which can be used to send specific KPIs.
Instead of sending all KPI values. Putting a # before the line to comment the line.
python zabbix_send_from_kpi_xml.py --kpi-xml  --generate-properties
example: python zabbix_send_from_kpi_xml.py --kpi-xml KPI_SOURCE_FILE.xml --generate-properties --debug

2. Send all KPI from the xml to server.

Using this option will send all the KPI values from the XML. Most of the time this is better.
python zabbix_send_from_kpi_xml.py --kpi-xml  --zabbix-server-ip  --zabbix-server-port  --host-name  --all --debug
example: python zabbix_send_from_kpi_xml.py --kpi-xml KPI_SOURCE_FILE.xml --zabbix-server-ip 127.0.0.1 --zabbix-server-port 10051 --host-name ams-ggsn --all --debug

3. Send KPI from properties file.

Using this option script expects the properties file to be provided, which was create in option (1).
python zabbix_send_from_kpi_xml.py --kpi-xml  --zabbix-server-ip  --zabbix-server-port  --host-name  --properties-file  --debug
example: python zabbix_send_from_kpi_xml.py --kpi-xml KPI_SOURCE_FILE.xml --zabbix-server-ip 127.0.0.1 --zabbix-server-port 10051 --host-name ams-ggsn --properties-file KPI_SOURCE_FILE.xml.properties --debug

Parameter Information

Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/zubayr/zbxsend.
Install Packages – pip install zbxsend and pip install xmltodict
python zabbix_send_from_kpi_xml.py        
[-h] -x KPI_XML [-i ZABBIX_SERVER_IP] [-s ZABBIX_SERVER_PORT]
[-n HOST_NAME] [-p PROPERTIES_FILE] [-a] [-g] [-d] [-v]

optional arguments:
-h, --help show this help message and exit
-x KPI_XML, --kpi-xml KPI_XML
Is the XML KPI file for Device.
-i ZABBIX_SERVER_IP, --zabbix-server-ip ZABBIX_SERVER_IP
IP address of the Zabbix Server to which this data
needs to be sent.
-s ZABBIX_SERVER_PORT, --zabbix-server-port ZABBIX_SERVER_PORT
Port which Zabbix server is listening (Default :
10051).
-n HOST_NAME, --host-name HOST_NAME
Host name as given in Zabbix server.
-p PROPERTIES_FILE, --properties-file PROPERTIES_FILE
This is the file which is generated using option (1).
'#' is comment in properties file.
-a, --all Application Name in the Zabbix Server. (To organize
all items being imported)
-g, --generate-properties
Include First line (Header) in the CSV file input,
[Default : False]
-d, --debug Running Debug mode - More Verbose

Code location

Categories: HOWTOs

SFTP Data Collector

October 20, 2014 Leave a comment
SFTP Data Collector
Easy way to collect files recursively over a sftp server is to connect to the server over scp and do scp -r.
Problem was that the device we were connecting to did not support recursive 😦 over a regEx expression.

Example:

scp -r ahmed@remote-host:/home/ahmed/*file_123*
Was not working. Here is a simple SFTP Data Collector script.
This script can be used, if the source device/server is unable to get file recursively.

Step in this script:

  1. get listing of the files present.
  2. select files required from the list using a reg_ex or a pattern.
  3. Download select files. Below is the command usage.

Usage:

usage: sftp_data_collector.py [-h] -sh SRC_HOST_NAME -su SRC_USERNAME
(-sp SRC_PASSWORD | -es) -dh DEST_HOST_NAME -du
DEST_USERNAME (-dp DEST_PASSWORD | -ed)
[-c SRC_DIRECTORY] [-y DEST_DIRECTORY]
(-t YYYYMMDD_HH | -p PATTERN_IN_FILE) [-d]
[--version]

SFTP Data Collector.
----------------------

This script can be used, if the source device/server is unable to "get" file recursively.
To use password from environment, set values for :

PASS_SRC for source.
PASS_DEST for destination.

Steps in this script:

1. "get" listing of the files present.
2. select files required from the list using a reg_ex or a pattern. (currently this is yyyymmdd_hh)
3. Download select files.

----------------------

optional arguments:
-h, --help show this help message and exit
-sh SRC_HOST_NAME, --src-host-name SRC_HOST_NAME
Source Host name to get Files from.
-su SRC_USERNAME, --src-username SRC_USERNAME
Source Host - Username.
-sp SRC_PASSWORD, --src-password SRC_PASSWORD
Source Host - Password.
-es, --src-env-password
Source Host - Password. Pick From Environment Variable
PASS_SRC
-dh DEST_HOST_NAME, --dest-host-name DEST_HOST_NAME
Destination Host name to send Files to.
-du DEST_USERNAME, --dest-username DEST_USERNAME
Destination Host - Username.
-dp DEST_PASSWORD, --dest-password DEST_PASSWORD
Destination Host - Password.
-ed, --dest-env-password
Destination Host - Password. Pick From Environment
Variable PASS_DEST
-c SRC_DIRECTORY, --cd-src-directory SRC_DIRECTORY
Source Directory, If not provided then "."
-y DEST_DIRECTORY, --cd-dest-directory DEST_DIRECTORY
Destination Directory, If not provided then "."
-t YYYYMMDD_HH, --date-hour YYYYMMDD_HH
Enter date_hour in yyyymmdd_hh format,File has
date_hour pattern in the filename
-p PATTERN_IN_FILE, --pattern-in-file PATTERN_IN_FILE
Enter pattern in filename which needs to be collected
from sFTP server.
-d, --debug Running Debug mode - More Verbose
--version show program's version number and exit

Code Usage:

Call function with parameters. Might want to do a check for the pattern here.
sftp_data_collector.get_file_from_src (src_host_name, dest_host_name, src_username, src_passwd,
dest_username, dest_passwd, pattern,
cd_src_directory_args=current_dir_src, cd_dest_directory_args=current_dir_dest)

Code Location:

Code can be found on github : https://github.com/zubayr/sftp_data_collector
Categories: HOWTOs