mex-docs

Writing Metrics to CSV

This example shows how to export metrics from the MobiledgeX API and write them to a CSV file, suitable for import to the spreadsheet or analysis program of your choice.

Metrics pulled will include:

CPU
MEM
Disk
Network
Connections

Exclusions

The example script provided is not suited for production use, and is intended solely as a proof of concept. Additionally, please be aware of the following additional limitations of the script:

Samples are taken at 10-second intervals.
Metrics being sampled are CPU, disk, network, and memory.
The script assumes you have logged into the MobiledgeX console with mcctl and have an active JWT token.
The script relies on data being returned in json format from the mcctl utility.
Please see the script header for additional information.

Assumptions

You have the mcctl utility installed.
You have an account with access to the application you wish to monitor.
You have an InfluxDB installation with a database named mex.
You can read/write to/from the InfluxDB database.

Script Flow

The script flow is very simple:

Pull data from the MobiledgeX API.
Transform the data into CSV.
Write the data out to *.csv files for each metric class.
Sleep for 30 seconds.
Return to step 1.

Usage

The first step is to define the variables for your use case.

Data Output

The following variables define the CSV files to write to; note that the script will clear out any existing data at startup.

cpuCSV
memCSV
diskCSV
networkCSV
connectionsCSV

Application Detail

The following variables define the application to be monitored.

theregion
theapporg
theappname
theappvers

As written, the script runs at 30 second intervals and loops 80 times, which will result in a run time of roughly 40 minutes for data collection. Note that this is a rough estimate only, since the script does not take into account the amount of time the metrics collection takes. If you wish to change this, the relevant line to modify is for KOUNT in {1..80} ; do

Once you have adjusted the script appropriately, your next steps are:

Using mcctl, login to console.mobiledgex.net which will generate a JWT token for you to use.
Run the script.

Once the script has completed it’s run you can use the CSV files for analysis as required. Note that the script can be interrupted at any time; the data is written to the files as it is received (ie, there is no buffering in the script).

Data Collection Script

#!/bin/env bash
###########################################################################
#
# This is a simple shell script to show the process of pulling data from the MeX
# metrics API endpoint, converting it to CSV, and writing it to a file.
#
# This script is intended as a demonstration of how this process can be
# accomplished. This is not intended to be a script that can be productionized without
# major rewriting.
#
# Version 1.0
# JAS
# 20-Aug-2020
###########################################################################

##
# Output Files
##
export cpuCSV=cpu.csv
export memCSV=mem.csv
export diskCSV=disk.csv
export networkCSV=network.csv
export connectionsCSV=connections.csv

##
# Application Information
##
export theregion=EU
export theapporg=demoorg
export theappname=healthcheck
export theappvers=2.0

##
# Put Headers into the CSV
##
echo "time, app, ver, cluster, clusterorg, cloudlet, cloudletorg, apporg, pod, cpu" > $cpuCSV
echo "time, app, ver, cluster, clusterorg, cloudlet, cloudletorg, apporg, pod, sendBytes, recvBytes" > $network.csv
echo "time, app, ver, cluster, clusterorg, cloudlet, cloudletorg, apporg, pod, mem"  > $memCSV
echo "time, app, ver, cluster, clusterorg, cloudlet, cloudletorg, apporg, pod, disk"  > $diskCSV
echo "time, app, ver, cluster, clusterorg, cloudlet, cloudletorg, apporg, pod, port, active, handled, accepts, bytesSent, bytesRecvd, P0, P0, P25, P50, P75, P90, P95, P99, P99.5, P99.9, P100" > $connectionsCSV

###
# Loop 80 Times; at 30 second loops this is roughly 40 minutes.
# Note that this will run longer than 40 minutes, since the time
# spent in the API calls.
###
for KOUNT in {1..80} ; do

echo Stats Being pulled at $(date)
# CPU
mcctl --addr https://console.mobiledgex.net  --output-format json metrics app  region=$theregion app-org=$theapporg  appname=$theappname appvers=$theappvers last=1 selector=cpu| jq -r  '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) | {"time": .[].time | tostring, "app": .[].app | tostring, "ver": .[].ver | tostring, "cluster": .[].cluster | tostring, "clusterorg": .[].clusterorg | tostring, "cloudlet": .[].cloudlet | tostring, "cloudletorg": .[].cloudletorg | tostring, "apporg": .[].apporg | tostring, "pod": .[].pod | tostring, "cpu": .[].cpu | tostring} | to_entries|map(.value)|@csv' >> $cpuCSV

# MEM
mcctl --addr https://console.mobiledgex.net  --output-format json metrics app  region=$theregion app-org=$theapporg  appname=$theappname appvers=$theappvers last=1 selector=mem | jq -r  '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) | {"time": .[].time | tostring, "app": .[].app | tostring, "ver": .[].ver | tostring, "cluster": .[].cluster | tostring, "clusterorg": .[].clusterorg | tostring, "cloudlet": .[].cloudlet | tostring, "cloudletorg": .[].cloudletorg | tostring, "apporg": .[].apporg | tostring, "pod": .[].pod | tostring, "mem": .[].mem | tostring} | to_entries|map(.value)|@csv' >> $memCSV

# Disk
mcctl --addr https://console.mobiledgex.net  --output-format json metrics app  region=$theregion app-org=$theapporg  appname=$theappname appvers=$theappvers last=1 selector=disk| jq -r  '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) | {"time": .[].time | tostring, "app": .[].app | tostring, "ver": .[].ver | tostring, "cluster": .[].cluster | tostring, "clusterorg": .[].clusterorg | tostring, "cloudlet": .[].cloudlet | tostring, "cloudletorg": .[].cloudletorg | tostring, "apporg": .[].apporg | tostring, "pod": .[].pod | tostring, "disk": .[].disk | tostring} | to_entries|map(.value)|@csv' >> $diskCSV

# Network
mcctl --addr https://console.mobiledgex.net  --output-format json metrics app  region=$theregion app-org=$theapporg  appname=$theappname appvers=$theappvers last=1 selector=network | jq -r  '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) | {"time": .[].time | tostring, "app": .[].app | tostring, "ver": .[].ver | tostring, "cluster": .[].cluster | tostring, "clusterorg": .[].clusterorg | tostring, "cloudlet": .[].cloudlet | tostring, "cloudletorg": .[].cloudletorg | tostring, "apporg": .[].apporg | tostring, "pod": .[].pod  | tostring, "sendBytes": .[].sendBytes | tostring, "recvBytes": .[].recvBytes | tostring} | to_entries|map(.value)|@csv' >> $networkCSV

# Connections
mcctl --addr https://console.mobiledgex.net  --output-format json metrics app  region=$theregion app-org=$theapporg  appname=$theappname appvers=$theappvers last=1 selector=connections | jq -r  '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) | {"time": .[].time | tostring, "app": .[].app | tostring, "ver": .[].ver | tostring, "cluster": .[].cluster | tostring, "clusterorg": .[].clusterorg | tostring, "cloudlet": .[].cloudlet | tostring, "cloudletorg": .[].cloudletorg | tostring, "apporg": .[].apporg | tostring, "pod": .[].pod  | tostring, "port": .[].port | tostring, "active": .[].active | tostring, "handled": .[].handled | tostring, "accepts": .[].accepts | tostring, "bytesSent": .[].bytesSent | tostring, "bytesRecvd": .[].bytesRecvd | tostring, "P0": .[].PO | tostring | tostring, "P0": .[].P0 | tostring, "P25": .[].P25 | tostring, "P50": .[].P50 | tostring, "P75": .[].P75 | tostring, "P90": .[].P90 | tostring, "P95": .[].P95 | tostring, "P99": .[].P99 | tostring, "P99.5": .[]."P99.5" | tostring, "P99.9": .[]."P99.9" | tostring, "P100": .[].P100 | tostring} | to_entries|map(.value)|@csv' >> $connectionsCSV

echo "Sleeping for 15 seconds"
sleep 15
echo "Finished Loop $KOUNT"

done