System troubleshooting is the major task for admin in day to day activities.They are facing difficulties to identify system performance issue on Linux.
There are many tools available in Linux to identify system performance issue but each tool (like top, vmstat, ps, iostat, etc) has it own features and we can’t get all in one place to identify the root causes.
Few tools are available in Linux to get all these details in one place. We already covered such kind of topic in past called sosreport.
Suggested Read :
(#) What Is sosreport? And How It Helps To Troubleshoot Issue In Linux
(#) Sysdig – Linux System Exploration/Diagnostic And Troubleshooting Tool
What is OSWatcher (OSWbb)
Oracle OSWatcher Black Box (OSWbb) collects and archives operating system and network metrics that you can use to diagnose performance issues. It’s a set of shell scripts OSWbb operates as a set of background processes on the server and gathers data on a regular basis, invoking below Unix utilities.
Make sure the blow packages should be installed on your system as prerequisites. If not, install missing packages using distribution package manager.
- VMSTAT
- IOSTAT
- MPSTAT
- IFCONFIG
- NETSTAT
- TOP
- TRACEROUTE
oswbb is comprised of two separate components:
oswbb:An Unix shell script data collector which collects and stores the data.oswbba:A java utility which will analyze the data automatically and provide advice and produce graphs and an html document.
How to install OSWatcher (OSWbb) in Linux
This is straight forward. Just download the latest available package version from oracle website and extract to descried location.
- Log on to My Oracle Support (MOS) at http://support.oracle.com
- Download the file oswbbXXX.tar, which is available at https://support.oracle.com/epmos/main/downloadattachmentprocessor?attachid=301137.1:OSW_file
Move/Copy the file to the directory where you want to install OSWbb and extract the file.
# tar xvf oswbb812.tar
It creates a directory called oswbb, which contains all the directories and files that are associated with OSWbb.
# ls -lh /opt/oswbb total 516K drwx------ 2 oracle oinstall 4.0K Sep 6 2017 analysis drwx------ 13 oracle oinstall 4.0K Mar 1 18:03 archive drwx------ 13 oracle oinstall 4.0K Oct 20 13:49 archive-old -rwxr-xr-x 1 oracle oinstall 67 Jan 15 2014 call_du.sh -rwxr-xr-x 1 oracle oinstall 68 Oct 7 2013 call_sar.sh -rwxr-xr-x 1 oracle oinstall 71 Jan 7 2014 call_uptime.sh drwx------ 2 oracle oinstall 4.0K May 2 2017 data drwx------ 4 oracle oinstall 4.0K Sep 5 2017 docs -rwxr-xr-x 1 oracle oinstall 665 Feb 6 2017 Example_extras.txt -rwxr-xr-x 1 oracle oinstall 1.9K Oct 7 2013 Exampleprivate.net -rwxr-xr-x 1 oracle oinstall 3.9K Apr 17 2017 genprvnet.sh drwx------ 2 oracle oinstall 4.0K Sep 6 2017 gif -rwxr-xr-x 1 oracle oinstall 772 May 8 2014 ifconfigsub.sh -rwxr-xr-x 1 oracle oinstall 743 Oct 7 2013 iosub.sh drwx------ 2 root root 4.0K Mar 16 19:59 locks -rwxr-xr-x 1 oracle oinstall 1.5K Jan 8 2014 ltop.sh -rwxr-xr-x 1 oracle oinstall 542 Oct 7 2013 mpsub.sh -rwxr-xr-x 1 oracle oinstall 740 Oct 7 2013 nfssub.sh -rwxr-xr-x 1 oracle oinstall 6.7K Jul 12 2017 OSWatcherFM.sh -rwxr-xr-x 1 oracle oinstall 47K Aug 14 2017 OSWatcher.sh -rw-r--r-- 1 oracle oinstall 10K Sep 6 2017 oswbb810.tar -rwxr-xr-x 1 oracle oinstall 310K Sep 6 2017 oswbba.jar -rwxr-xr-x 1 oracle oinstall 414 Oct 7 2013 oswib.sh -rwxr-xr-x 1 oracle oinstall 435 Jan 8 2014 oswnet.sh -rwxr-xr-x 1 oracle oinstall 825 Oct 7 2013 oswrds.sh -rwxr-xr-x 1 oracle oinstall 524 Oct 7 2013 oswsub.sh -rwxr-xr-x 1 oracle oinstall 1.5K Aug 24 2017 psmemsub.sh -rwxr-xr-x 1 oracle oinstall 557 Jul 12 2017 sarsub.sh drwx------ 7 oracle oinstall 4.0K Sep 6 2017 src -rwxr-xr-x 1 oracle oinstall 2.6K Feb 26 2015 startOSWbb.sh -rwxr-xr-x 1 oracle oinstall 751 Aug 14 2017 stopOSWbb.sh -rwxr-xr-x 1 oracle oinstall 834 Feb 6 2017 tar_up_full_archive.sh -rwxr-xr-x 1 oracle oinstall 5.3K Feb 6 2017 tar_up_partial_archive.sh drwx------ 2 root root 4.0K Mar 16 19:59 tmp -rwxr-xr-x 1 oracle oinstall 527 Feb 7 2017 topaix.sh -rwxr-xr-x 1 oracle oinstall 545 Feb 23 2015 vmsub.sh -rwxr-xr-x 1 oracle oinstall 1.5K Feb 26 2015 xtop.sh
To start OSWbb, run the startOSWbb.sh script from the oswbb directory. This script accepts two parameters, which is Frequency & Duration.
Frequency :How frequent (in seconds) data needs to be collected.Duration :How many hours data needs to be archived
If you do not input any arguments, the script runs with default values of 30 and 48. It collects data every 30 seconds and store the last 48 hours of data in archive files.
# ./startOSWbb.sh [Frequency Duration]
The below example collects and stores the last 10 hours of data to archive files at 60 second intervals.
# ./startOSWbb.sh 60 10 Setting the archive log directory to/opt/oswbb/archive Testing for discovery of OS Utilities... VMSTAT found on your system. IOSTAT found on your system. MPSTAT found on your system. IFCONFIG found on your system. NETSTAT found on your system. TOP found on your system. TRACEROUTE found on your system. Discovery of CPU CORE COUNT CPU CORE COUNT will be used by oswbba to automatically look for cpu problems CPU CORE COUNT = 6 VCPUS/THREADS = 24 Discovery completed. Starting OSWatcher v8.1.0 on Tue Feb 27 15:14:54 MST 2018 With SnapshotInterval = 60 With ArchiveInterval = 10 OSWatcher - Written by Carl Davis, Center of Expertise, Oracle Corporation For questions on install/usage please go to MOS (Note:301137.1) If you need further assistance or have comments or enhancement requests you can email me [email protected] Data is stored in directory: /opt/oswbb/archive Starting Data Collection... oswbb heartbeat:Tue Feb 27 15:14:59 MST 2018 oswbb heartbeat:Tue Feb 27 15:15:59 MST 2018 oswbb heartbeat:Tue Feb 27 15:17:00 MST 2018 oswbb heartbeat:Tue Feb 27 15:18:00 MST 2018 oswbb heartbeat:Tue Feb 27 15:19:00 MST 2018 oswbb heartbeat:Tue Feb 27 15:20:00 MST 2018 oswbb heartbeat:Tue Feb 27 15:21:00 MST 2018 oswbb heartbeat:Tue Feb 27 15:22:00 MST 2018 . .
To stop OSWbb prematurely, run the stopOSWbb.sh script from the oswbb directory.
# ./stopOSWbb.sh
How to view the collected data
Collected data will be stored under archive directory, it is created when OSWbb is started for the first time. OSWbb stores data in hourly archive files named system_name_utility_name_timestamp.dat, and each entry in a file is preceded by the characters *** and a timestamp.
# ls -lh /opt/oswbb/archive total 44K drwx------ 2 root root 4.0K Mar 1 18:06 oswcpuinfo drwx------ 2 root root 4.0K Mar 16 19:01 oswifconfig drwx------ 2 root root 4.0K Mar 16 19:01 oswiostat drwx------ 2 root root 4.0K Mar 16 19:01 oswmeminfo drwx------ 2 root root 4.0K Mar 16 19:01 oswmpstat drwx------ 2 root root 4.0K Mar 16 19:01 oswnetstat drwx------ 2 root root 4.0K Mar 1 18:03 oswprvtnet drwx------ 2 root root 4.0K Mar 16 19:01 oswps drwx------ 2 root root 4.0K Mar 16 19:01 oswslabinfo drwx------ 2 root root 4.0K Mar 16 19:01 oswtop drwx------ 2 root root 4.0K Mar 16 19:01 oswvmstat
To check any data, just navigate to corresponding directory and view the corresponding file. For testing purpose, we printed vmstat file output.
# more oracle.2daygeek.com_vmstat_18.02.28.2300.dat Linux OSWbb v8.1.0 oracle.2daygeek.com SNAP_INTERVAL 60 CPU_CORES 6 VCPUS 24 OSWBB_ARCHIVE_DEST /opt/oswbb/archive zzz ***Wed Feb 28 23:00:09 MST 2018 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 23 18027636 873140 11044 477948 101 138 5013 1964 93 13 14 2 67 17 0 2 17 18052168 878900 11044 481844 7636 24704 88672 58333 28214 41906 6 4 59 31 0 2 16 18065140 873880 11580 486796 5692 13868 71832 81249 27917 42840 8 3 61 29 0 zzz ***Wed Feb 28 23:01:09 MST 2018 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 21 18481600 868868 10896 507336 102 138 5037 1966 96 19 14 2 67 17 0 5 12 18495272 880956 10884 502060 3472 14188 296964 76683 25768 44284 6 3 61 30 0 2 13 18495680 874532 11000 504336 4620 1532 215072 43823 22749 40949 6 2 64 29 0 zzz ***Wed Feb 28 23:02:09 MST 2018 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 7 21 18840580 866384 10756 479816 102 139 5054 1967 0 25 14 2 67 17 0 5 10 18845460 870052 10868 483252 5624 5696 191656 75508 24249 42834 7 3 60 31 0 2 9 18855152 881676 10844 478320 5620 10556 248988 22601 21635 37511 5 2 66 28 0 zzz ***Wed Feb 28 23:03:10 MST 2018 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 8 12 15170720 1074652 18680 489808 103 139 5068 1967 3 31 14 2 67 17 0 2 11 15169920 1055024 19356 500372 5868 0 185516 202669 24888 45625 5 2 67 27 0 2 16 15169420 1037896 19640 509260 3944 0 99680 268204 25285 45869 4 2 67 27 0 zzz ***Wed Feb 28 23:04:10 MST 2018 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 8 8 15347816 879160 7268 595280 103 139 5058 1973 7 36 14 2 67 17 0 5 5 15346984 865644 7536 602356 6164 0 51172 52905 24542 39722 14 1 67 18 0 --More--(8%)
OSWatcher Analyzer
oswbba is a graphing and analysis utility which comes bundled with OSWatcher. This allows the user to graph and analyze data collected from running OSWatcher. Analyzing this data automatically avoids the time consuming task of manually inspecting all the files that OSWatcher collects. This utility is a stand-alone java program which requires java to be installed.
# java -jar oswbba.jar -i /opt/oswbb/archive/ Starting OSW Analyzer V8.1.2 OSWatcher Analyzer Written by Oracle Center of Expertise Copyright (c) 2017 by Oracle Corporation Parsing Data. Please Wait... Scanning file headers for version and platform info... Parsing file oracle.2daygeek.com_iostat_18.04.19.0300.dat ... Parsing file oracle.2daygeek.com_iostat_18.04.19.0400.dat ... This directory already exists. Rewriting... Parsing file oracle.2daygeek.com_vmstat_18.04.19.0300.dat ... Parsing file oracle.2daygeek.com_vmstat_18.04.19.0400.dat ... Parsing file oracle.2daygeek.com_netstat_18.04.19.0300.dat ... Parsing file oracle.2daygeek.com_netstat_18.04.19.0400.dat ... Parsing file oracle.2daygeek.com_top_18.04.19.0300.dat ... Parsing file oracle.2daygeek.com_top_18.04.19.0400.dat ... Parsing file oracle.2daygeek.com_ps_18.04.19.0300.dat ... Parsing file oracle.2daygeek.com_ps_18.04.19.0400.dat ... Parsing Completed. Enter 1 to Display CPU Process Queue Graphs Enter 2 to Display CPU Utilization Graphs Enter 3 to Display CPU Other Graphs Enter 4 to Display Memory Graphs Enter 5 to Display Disk IO Graphs Enter GC to Generate All CPU Gif Files Enter GM to Generate All Memory Gif Files Enter GD to Generate All Disk Gif Files Enter GN to Generate All Network Gif Files Enter L to Specify Alternate Location of Gif Directory Enter Z to Zoom Graph Time Scale (Does not change analysis dataset) Enter B to Returns to Baseline Graph Time Scale (Does not change analysis dataset) Enter R to Remove Currently Displayed Graphs Enter X to Export Parsed Data to Flat File Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale) Enter A to Analyze Data Enter D to Generate DashBoard Enter Q to Quit Program Please Select an Option:
