The MobiledgeX platform provides the ability to retrieve metrics on your applications and clusters via both the Web Console and the MobiledgeX API. MobiledgeX controls the granularity and retention policy for these metrics. If you wish more control over your metrics, you can write an ETL pipeline to move the metrics that you are interested into your own TSDB.
This example uses InfluxDB as a TSDB to store application metric data.
The example script provided is not suited for production use, and is intended solely as a proof of concept. Additionally, please be aware of the following additional limitations of the script:
mcctl and have an active JWT tokenmcctl utility.Please see the script header for additional information.
mcctl utility installed.mex.The script flow is very simple:
The mcctl command is used to pull data from the MobiledgeX API.
mcctl --addr https://console.mobiledgex.net --output-format json metrics app \ region=$REGION app-org=$APPORG appname=$APPNAME appvers=$APPVER last=1"
You will need to replace REGION, APPORG, APPNAME, and APPVER with the data that corresponds to the application you wish to monitor. The use of last=1 restricts the data returned to the most recently collected metrics. This can be omitted, in which case the API will return multiple rows (unique by timestamp). You can also specify start and end times for metrics. For this example, we will just be using the last collected set of metrics.
The data from the above will be returned in json format, and will be presented as follows:
{
"data": [
{
"Series": [
{
"columns": [
"time",
"app",
"ver",
"cluster",
"clusterorg",
"cloudlet",
"cloudletorg",
"apporg",
"pod",
"cpu"
],
"name": "appinst-cpu",
"values": [
[
"2020-08-11T14:51:54.687583518Z",
"compose-file-test",
"10",
"autoclustercompose-file-test",
"demoorg",
"hamburg-main",
"TDG",
"demoorg",
"compose-file-test",
0
]
]
}
]
}
]
}
The structure is as follows:
data: This is the top-level key that all returned data will be presented beneath.
Series: This is the level below data and contains information on the metrics you have requested.
columns: An array of the columns that are being presented. This occurs once in the series.name: The name of the metric being returned. This can occur several times in the series, depending on the metrics selected.values: An array of the values that correspond to the keys specified in the columns section. This can occur several times in the series, depending on the time/intervals selected.To load this data into a TSDB we will need to transform it into a format that the DB understands. For our example, we will be changing this data into InfluxDB’s Line Protocol. To do this, we will need to parse the JSON output. To accomplish this, we will be using the jq utility, along with awk. This could also be accomplished using other JSON and text processing tools if desired.
Note: This document is not intended to guide the usage of jq. The example presented here has been tested and works correctly with the MobiledgeX API’s JSON output. This particular example is parsing memory information.
In its simplest form line protocol provides the name of the metric, a list of one or more key/value paris of tags, a list of one or more key/value pairs of measurements, and an optional timestamp. The syntax is defined as:
<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]
For our purposes we will be constructing a very basic data payload. The following is an example of what that payload will look like for the memory metric:
mem.app=compose-file-test,ver=10 mem="1990197"
We will use the jq utility to convert our data; the following line will take as input the data returned from the MobiledgeX API and parse the JSON to prepare it for final transformation:
jq -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | \
map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]})) |\
{measurement: "mem", mem: .[].mem | tostring, app: .[].app, ver: .[].ver, \
timestamp: .[].time }| \
to_entries|map(.value)|@csv'
Breaking down that command, we are doing the following:
jq to provide the output in raw format (-r) so we can parse the output with awk.column array and array(s) of values (Lines 1-2).This provides us with the following output:
"mem","1990197","compose-file-test","10","2020-08-11T15:15:59.135953533Z"
The next step is finalizing the conversion. To do this we need to manipulate the data into the Line Protocol format. We will be using awk to complete the transformation:
awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}'
Breaking down that command, we are doing the following:
, as our separator character.The final output to be sent to InfluxDB is:
mem.app=compose-file-test,ver=10 mem="1990197"
The reason we are allowing the InfluxDB installation to generate a timestamp rather than using the value returned from the API is due to the way that the MobiledgeX API provides the timestamp, and the way that InfluxDB requires timestamps to be presented.
The MobiledgeX API provides timestamps in RFC3339 format, whereas InfluxDB wants the timestamps to be in [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time#:~:text=Unix%20time%20(also%20known%20as,UTC%20on%201%20January%201970.) format. Although it is possible to convert between these two (for example, using the GNU date program), this has not been done in this POC script to keep the complexity low.
The InfluxDB API can be used to load the processed data into InfluxDB. The format for inserting data into InfluxDB using curl is:
curl -i -XPOST 'http://localhost:8086/write?db=mex'
--data-binary 'measurement-name.tag1=value1,tag2=value2 value=123 1434055562000000000'
Breaking down the command, we are doing the following:
--data-binary flag to enables us to pass data without it being interpreted.-i flag shows us the return headers from the server (useful in debugging).For this test, we are going to be inserting the following data:
mem.app=compose-file-test,ver=10 mem="1990197"
To do this, we can write the following cURL command:
$ curl -i -XPOST 'http://localhost:8086/write?db=mex' --data-binary 'mem.app=compose-file-test,ver=10 mem="1990197"'
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: de38aed6-dc1d-11ea-8002-acde48001122
X-Influxdb-Build: OSS
X-Influxdb-Version: v1.8.1
X-Request-Id: de38aed6-dc1d-11ea-8002-acde48001122
Date: Tue, 11 Aug 2020 21:59:02 GMT
The 204 return code indicates that the data was accepted.
There are several ways to verify the data being added to InfluxDB. Visualization tools such as Grafana or Chronograf can be used, as can the influx CLI utility. For this example, we are going to use the CLI.
$ influx
Connected to http://localhost:8086 version v1.8.1
InfluxDB shell version: v1.8.1
> use mex;
Using database mex
> SELECT * FROM "mex"."autogen"."mem.app=compose-file-test" WHERE "ver"='10' limit 1;
name: mem.app=compose-file-test
time mem ver
---- --- ---
1597096338602015000 1990197 10
>
The following script uses all of the components that have been discussed in this document. Again, please note that this is intended as a proof of concept demonstration only and is not intended for production usage.
#!/usr/bin/env bash
###########################################################################
#
# This is a simple shell script to show the process of pulling data from the MeX
# metrics API endpoint and pushing them into a local influxdb data store.
#
# This script is intended as a demonstration of how this process can be
# accomplished. This is not intended to be a script that can be productionized without
# major rewriting.
#
# This script makes the following assumptions:
# 1\. You are able to use the `mcctl` program to access the MeX API.
# 2\. You have authenticated the `mcctl` program and saved an access token;
# this script does not authenticate.
# 3\. You have an influxdb server running on the standard port (8086)
# 4\. There is no security on the influxdb database.
# 5\. You have an existing database called `mex` without security.
#
# The script performs the following tasks:
# 1\. Connects to the api and pulls the most recent update for the given metric.
# 2\. Transforms the returned data using `jq` and `awk` to create influxdb line
# protocol compatible output.
# 3\. Writes the resulting data into the influxdb `mex` database using `curl`
#
# Notes:
# 1\. Influxdb does not accept RFC3339 formatted dates as returned by the MeX API;
# Because of this the example allows influxdb to generate a timestamp. In an
# actual production implementation you would want to use the MeX provided
# timestamp, which can be converted to epoch time using either the GNU `date`
# command, or programatically.
#
###########################################################################
# General Variables
MCCTL=/usr/local/bin/mcctl
JQ=/usr/local/bin/jq
INFLUXDB=mex
INFLUXURI=http://localhost:8086
# MeX Vars
APPNAME=compose-file-test
APPVER="1.0"
APPORG=demoorg
REGION=EU
CONSOLE="https://console.mobiledgex.net"
MCCTLCONS="$MCCTL --addr $CONSOLE --output-format json metrics app region=$REGION app-org=$APPORG appname=$APPNAME appvers=$APPVER last=1"
#cURL
CURLC="curl -X POST -d @- http://localhost:8086/write?db=mex"
# CPU
$MCCTLCONS selector=cpu | $JQ -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]}))| {measurement: "cpu", cpu: .[].cpu | tostring, app: .[].app, ver: .[].ver, timestamp: .[].time }| to_entries|map(.value)|@csv' | awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}' | $CURLRC
# MEM
$MCCTLCONS selector=mem | $JQ -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]}))| {measurement: "mem", mem: .[].mem | tostring, app: .[].app, ver: .[].ver, timestamp: .[].time }| to_entries|map(.value)|@csv' | awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}' | $CURLRC
# NET
$MCCTLCONS selector=network | $JQ -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]}))| {measurement: "recvBytes", recvBytes: .[].recvBytes | tostring, app: .[].app, ver: .[].ver, timestamp: .[].time }| to_entries|map(.value)|@csv' | awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}' | $CURLRC
$MCCTLCONS selector=network | $JQ -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]}))| {measurement: "sendBytes", sendBytes: .[].sendBytes | tostring, app: .[].app, ver: .[].ver, timestamp: .[].time }| to_entries|map(.value)|@csv' | awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}' | $CURLRC
# Disk
$MCCTLCONS selector=disk | $JQ -r '.data[0].Series[0] | (.columns | map(.)) as $headers| .values | map(. as $row | $headers | with_entries({"key": .value, "value": $row[.key]}))| {measurement: "disk", disk: .[].disk | tostring, app: .[].app, ver: .[].ver, timestamp: .[].time }| to_entries|map(.value)|@csv' | awk -F, '{gsub("\"","",$0);printf("%s.app=%s,ver=%s mem=\"%s\"\n",$1,$3,$4,$2)}' | $CURLRC
The same techniques shown here can be used to write data from the MobiledgeX metrics API to any other datastore, provided can create an ETL pipeline to load data into your datastore of choice.