Hello,

ist it possible also to include the temperatures and power usage for managed systems ?
See here : POWER8 Energy (Watts & Temperature) via the HMC REST API
Best regards,
Carsten

I want to separate influxdb from grafana by running each service on a dedicated container.

influxdb docker:

docker run -d --name influxdb -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 adil/influxdb

grafana

docker run --rm -v /Users/adil/000_aix/data:/var/lib/grafana \
    -e "GF_SERVER_ROOT_URL=http://grafana.adil.me"  \
    --link influxdb:influxdb -p 3000:3000 adil/grafana

everything works fine, connection between grafana and influxdb is ok:

I can import data:

➜  ~ nmon2influxdb -s 192.168.99.100 import aix731_test.nmon
###########
File aix731_test.nmon imported !
➜  ~  nmon2influxdb dashboard file --gurl "http://192.168.99.100:3000" aix731_test.nmon
Dashboard uploaded to grafana
➜ nmon2influxdb dashboard file --gurl "http://192.168.99.100:3000" -f aix731_test.nmon
Writing GRAFANA dashboard: aix731_test_dashboard

But It seems that graphs were not generated:

I have tested with the direct access too, but the same issue as with proxy mode.

Improve the default aix dashboard.

For example, by adding the new table panel from grafana 2.6 :

Hello,

Is it possible to make an evolution of this tool to integrate CSV mode nmon with ";" delim ?

I have to enable inluxdb ssl for data transfer encryption, but nmob2influxdb can't connect to the url when adding https to the influxdb_server in ~/.nmon2influxdb.cfg

/data/nmon/bin/nmon2influxdb import LPAR71_TEST_170404_0000.nmon
2017/04/04 15:09:22 Using configuration file /root/.nmon2influxdb.cfg
2017/04/04 15:09:22 Get http://https/ping: dial tcp: lookup https: no such host

my config

grep influxdb_server  /root/.nmon2influxdb.cfg
influxdb_server="https://influxserver.smrc.domain.org"

Would be nice to be able to import all nmon files in one folder :

nmon2influxdb import /mydirectory

Hi. Thanks for the earlier help. You've written a nice tool.
I'm new to reading Go and identifying the logic. Is there a way to add tags to the individual measurements? Is there a set of identifiers in an NMON file that are used to dynamically create measurements? I see that you use name to hold the metric type and host to hold the hostname.

We are using Telegraf to add tags identifying the data center, business class, application service, etc for our Windows and Linux hosts.

What are your recommendations for a similar use case for AIX hosts?

Thanks!
Alan

TOP metrics are not skipped when specified in import_skip_metrics.

Bug and fix reported by Thomas P.

Thanks :)

This is not really an issue, I wanted to graph the total CPU usage (between 0-100%)

The CPU_ALL table doesn't have the a value for cpu usage:

> SELECT * FROM CPU_ALL LIMIT 5
name: CPU_ALL
time                host            name          value
----                ----            ----          -----
1450783583000000000 AIX_71_TEST     Idle%         99.1
1450783583000000000 AIX_71_TEST     Sys%          0.3
1450783583000000000 AIX_71_TEST     PhysicalCPUs  128
1450783583000000000 AIX_71_TEST     Wait%         0
1450783583000000000 AIX_71_TEST     User%         0.6

My guess is the total CPU usage= SUM (User+Sys+Wait)

I've tried this but couldn't generate a graph:

SELECT sum("value") FROM "CPU_ALL" WHERE "host" =~ /^$host$/ AND "name" = 'Sys%' AND "name" = 'User%' AND "name" = 'Wait%' AND $timeFilter GROUP BY time($interval) fill(null)

is there a way to generate a graph based on the above formula?

Great tool!

It'd be useful to be able to process a folder of nmon files, not one at a time.

Default dashboard should include FCREAD, FCWRITE, FCXFERIN and FCXFEROUT graphs.

I have built a new server to test influxdb 1.2.0-1 using the same data I'm already using for influx 0.9

but seems grafana dashbord can't load data from influxdb 1.2

PS: I'm using gravana 4.1.1-1484211277

nmon2influxdb crash when the HMC cannot get properly the system statistics.

You see this message in the JSON data:

	"sampleType": "ManagedSystem",
				"sampleInfo": {
					"timeStamp": "2017-01-04T08:32:30-0500",
					"status": 1,
					"errorInfo": [
						{
							"errId": "4002",
							"errMsg": "Processing did not happen because of error in managed system's LTM collection",
							"uuid": "067fe31c-d3d5-314a-a065-ec216a49d9c2",
							"reportedBy": "ManagedSystem",
							"occurrenceCount": 1
						}
					]
				}
			}

And the error is:

nmon2influxdb hmc import -m system1
Using configuration file /Users/adejoux/.nmon2influxdb.cfg
Updating  autogen retention policy to keep only the last 40d days. Timestamp based.
Fetching latest 2 hours performance metrics. See hmc_samples parameter.
Getting list of managed systems

system1

goroutine 1 [running]:
panic(0x3d8220, 0xc420012110)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/adejoux/nmon2influxdb/hmc.Import(0xc4201a0500)
	/Users/adejoux/go/src/github.com/adejoux/nmon2influxdb/hmc/import.go:68 +0x7757
github.com/codegangsta/cli.Command.Run(0x457df1, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x45d546, 0x13, 0x0, ...)
	/Users/adejoux/go/src/github.com/codegangsta/cli/command.go:174 +0xbbe
github.com/codegangsta/cli.(*App).RunAsSubcommand(0xc4201a0280, 0xc4201a0140, 0x0, 0x0)
	/Users/adejoux/go/src/github.com/codegangsta/cli/app.go:298 +0x9f8
github.com/codegangsta/cli.Command.startApp(0x456d4d, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x45acc5, 0xd, 0x0, ...)
	/Users/adejoux/go/src/github.com/codegangsta/cli/command.go:249 +0x564
github.com/codegangsta/cli.Command.Run(0x456d4d, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x45acc5, 0xd, 0x0, ...)
	/Users/adejoux/go/src/github.com/codegangsta/cli/command.go:65 +0x16a3
github.com/codegangsta/cli.(*App).Run(0xc4201a0000, 0xc42000c190, 0x5, 0x5, 0x0, 0x0)
	/Users/adejoux/go/src/github.com/codegangsta/cli/app.go:187 +0xa0c
main.main()
	/Users/adejoux/go/src/github.com/adejoux/nmon2influxdb/main.go:286 +0x20e6

I asked HMC developers what are the erro codes to watch.

The disk graph is missed in my dashboard

this is my .nmon2influxdb.cfg:

debug=true
timezone="Asia/Qatar"
influxdb_user="root"
influxdb_password="root"
influxdb_server="localhost"
influxdb_port="8086"
influxdb_database="nmon"
grafana_user="admin"
grafana_password="admin"
grafana_url="http://localhost:3000"
grafana_datasource="nmon2influxdb"
import_skip_disks=false
import_all_cpus=false
import_build_dashboard=true
dashboard_write_file=true
stats_limit=20
stats_sort="mean"
stats_filter=""
stats_from=""
stats_to=""
stats_host=""

the nmon data was uploaded successfully as well the dashboard :

 nmon2influxdb import aix71_test_15122*
#######################################
File aix71_test1_151223_0000.nmon imported : 398927 points !
Dashboard uploaded to grafana
###############
File aix71_test1_151224_0000.nmon imported : 157759 points !
Dashboard uploaded to grafana

Hello Alain,
thank you very much for providing us nmon2influxdb! It workes very fine! One little question: We have a large System where we want to import the TOP-Data. nmon2influxdb import worked without errors. But on the generated dashboard the Graph is incompletly drawn. (only on part in the middle of the graph) Checking the nmon-file against pgraph the data is correctly reported. Is there a possibility to do some debugging ? Or do you have some idea for that behaviour ? I checked influxdb.log/grafana.log without success.
Thank you in advance !

P.S. "click" on a Process-Name in the footer of the TOP-Graph displays the process correctly over the complete timewindow. It seems the "stacked" Start-Graph is not 100% correct.

Would be nice to load metrics direclty from the HMC.

Would be nice to support a configuration file to avoid typing all options each time.

go buil fails, the build shows the following errorslog

[root@c1bffe16ee9d nmon2influxdb]# go build
# _/nmon2influxdb
./nmon_list.go:28: influxdb.ListMeasurement undefined (type *influxdbclient.InfluxDB has no field or method ListMeasurement)
./nmon_stats.go:47: undefined: influxdbclient.BuildStats

my Go env is:

[root@c1bffe16ee9d nmon2influxdb]# go version
go version go1.4.2 linux/amd64
[root@c1bffe16ee9d nmon2influxdb]# go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/lib/golang"
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

I have installed Ubuntu 14.04 on influxDB v0.10.2 and Grafana, I download is complete, but go build look like fails.Installation process is as follows:

tony@ubuntu:/go/src$ go get -u github.com/adejoux/nmon2influxdb
package github.com/influxdb/influxdb/client: code in directory /home/tony/go/src/github.com/influxdb/influxdb/client expects import "github.com/influxdata/influxdb/client"
tony@ubuntu:/go/src$ cd $GOPATH/src/github.com/adejoux/nmon2influxdb
tony@ubuntu:/go/src/github.com/adejoux/nmon2influxdb$ go build
../influxdbclient/api.go:3:8: code in directory /home/tony/go/src/github.com/influxdb/influxdb/client expects import "github.com/influxdata/influxdb/client"
tony@ubuntu:$ nmon2influxdb import ubuntu186_160322_1610.nmon
nmon2influxdb: command not found

Hello Sorry to bother again :), i have discover some error when value is "nan" on JFS (when using NFS for example) :

JFSFILE,T0001,27.6,4.9,42.6,41.4,66.9,30.4,2.5,55.3,10.2,54.3,6.6,48.3,7.6,2.9,39.8,55.0,17.9,4.1,34.1,4.5,29.2,0.0,35.0,40.8,40.3,3.1,29.3,77.7,8.9,3.4,30.3,49.4,45.6,87.4,2.7,3.3,5.7,nan,76.3,0.0

2017/04/24 13:45:59 Error: NaN is an unsupported value for field value
2017/04/24 13:45:59 Error: NaN is an unsupported value for field value
2017/04/24 13:45:59 Error: NaN is an unsupported value for field value
2017/04/24 13:45:59 Error: NaN is an unsupported value for field value

it crashed the import of file.

Hi Alain,
I am using nmon2influxdb 2.1.0 in RH7 server to load nmon perfdata from AIX server to influxdb server. The configuration file generated under /home/user is working perfectly.
Later on I added hmc parameters in the same configuration file based on our requirement and facing below issues:

1. Ran the command "nmon2influxdb hmc import" and is throwing below error:
   ---field corresponding to hmc_database' is not defined in **main.Config'
   ---field corresponding to hmc_user' is not defined in **main.Config'
   ---field corresponding to hmc_managed_system' is not defined in **main.Config'
   --- field corresponding to hmc_password' is not defined in **main.Config'
   However, I have created hmc_database = "hmc_reports" , "hmc_user", "hmc_password","hmc_managed_system" accordingly in the main config file. Any clue on it?

2. Do we need to create separate users to run nmon and hmc configurations separately or it is good idea to use combined configuration file and run via same user?
   In my case I am using combined and getting error on loading nmon files too after introducing hmc parameters which was good earlier.

Thanks,
Madhu

It should be possible to import only HMC system metrics skipping partitions.
It should be possible to specify the number of samples to fetch instead of fetching the last two hours at each execution.

Based on @Adilroot comments in #2.

By default, nmon2influxdb create a datasource with direct access to InfluxDB :

So users on Windows or Mac OS need to specify the influxdb ip in command line :

nmon2influxdb-windows-amd64.exe -s 192.168.99.100 dashboard file --gurl http://192.168.99.100:3000 vios1_150706_1552.nmon

By using proxy mode, it would not be needed anymore :

Would be nice to have a windows binary version.

Here a cross compiled version to test.
nmoN2influxdb.exe.zip

Hi

I copied compiled version of nmon2influx from https://www.dropbox.com/s/zylemeaas1w2o2g/nmon2influx

When I try put docker IP :
../nmon2influx -host="192.168.59.104:8086" -file="nmonfile.nmon"

I have error:

flag provided but not defined: -host
Usage of ../nmon2influx:
  -admin="admin": influxdb administor user
  -file="nmonfile": nmon file
  -pass="password": influxdb administor password

When I try default
../nmon2influx -file="nmonfile.nmon"
I have error:

panic: runtime error: slice bounds out of range
goroutine 16 [running]:
runtime.panic(0x6c93a0, 0x8974ef)
    /usr/lib/golang/src/pkg/runtime/panic.c:279 +0xf5
main.main()
    /home/adejoux/devel/go/src/github.com/adejoux/nmon2influx/nmon2influx.go:254 +0xa03
goroutine 19 [finalizer wait]:
runtime.park(0x4164d0, 0x89b548, 0x899789)
    /usr/lib/golang/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x89b548, 0x899789)
    /usr/lib/golang/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/lib/golang/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()

can you check if compiled nmon2influx use -host ?

Best regards
robert

With version 2.1.0, it's not possible to generate a default configuration file anymore:

adejoux@alains-mbp:~|⇒  nmon2influxdb hmc import
panic: runtime error: comparing uncomparable type nmon2influxdblib.Inputs

goroutine 1 [running]:
panic(0x3d8220, 0xc42011e570)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/naoina/toml.marshal(0xc420031c00, 0x37b, 0x400, 0x0, 0x0, 0x448580, 0xc420095440, 0x99, 0xc420090000, 0xc42004ef50, ...)
	/Users/adejoux/go/src/github.com/naoina/toml/encode.go:92 +0x532
github.com/naoina/toml.Marshal(0x448580, 0xc420095440, 0xc420095440, 0x448580, 0xc420095440, 0x1000, 0xc4201183c0)
	/Users/adejoux/go/src/github.com/naoina/toml/encode.go:44 +0x95
github.com/adejoux/nmon2influxdb/nmon2influxdblib.(*Config).BuildCfgFile(0xc420095200, 0xc420118360, 0x21)
	/Users/adejoux/go/src/github.com/adejoux/nmon2influxdb/nmon2influxdblib/config.go:148 +0x12a
github.com/adejoux/nmon2influxdb/nmon2influxdblib.(*Config).LoadCfgFile(0xc420095200, 0xc420118360, 0x21)
	/Users/adejoux/go/src/github.com/adejoux/nmon2influxdb/nmon2influxdblib/config.go:163 +0x394
main.main()
	/Users/adejoux/go/src/github.com/adejoux/nmon2influxdb/main.go:21 +0xcb

It's not possible to omit empty values for Inputs struct in nmon2influxdblib.

Hello Alain,
I have issue on tagging. Below is the part of my configuration file:
In our case, there are number of servers on different locations belongs to "REGION" (NA, EMEA, APAC). Hostnames are held on "Match" section (aixhost1|aixhost2|aixhost3|aixhost4.......and so on).
Is there any possibility that any host (aixhost1|aixhost2|aixhost3|aixhost4....) could pick any values (NA|AMEA|APAC) based on their location?
[[input]]
Measurement="CPU_ALL"
Name="host"
Match="^aixhost1|aixhost2|aixhost3|aixhost4$"
[[input.tag]]
Name="BUSINESSCLASS"
Value="GOLD"
[[input.tag]]
Name="HOSTTYPE"
Value="PHYSICAL"
[[input.tag]]
Name="OSTYPE"
Value="AIX"
[[input.tag]]
Name="REGION"
Value="NA"

Thanks,
Madhu

debug is not consistent and hard to use.

Hi. Thank you for sharing your tool. I attempted to import the test nmon files into my database and I get a message that 17956 points are imported but it is followed by an error stating that there is not default RP.

I created an RP with the name default and made it the default RP, but I still get the error. No measurements are created.

Any thoughts?

Thanks!
File linux_150818_1024.nmon imported : 17956 points !
2016/11/02 16:22:57 {"error":"retention policy not found: default"}

Should be possible to add a option to have the dashboard uploaded to Grafana immediately instead of needing to run another command.

Thanks for the custom tagging feature. In our use case all of the measurements for a given LPAR would have the same tags. For example, the same tag set will apply for the CPU_ALL measurement as for the DISKAVGRIO measurement. Is it feasible to add a [[global input]] section? That way each measurement would not require a specific [[input]] section.

There doesn't seem to be anywhere to point nmon2influxdb at my grafana instance? In this example I'm running it's local and seems to work automagically, but it would be good to make this configurable and/or clarify in the documentation that it will look for grafana locally.

We receive a "cannot execute binary file" error with the 2.1.0 release. 2.0.1 does execute.

Our system type an version:
uname -a
Linux LUSRCDNNV01.fossil.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Mon Feb 29 13:22:02 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
and
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Is this a Go library issue? I'm not able to run a yum update until a change is resolved.

Thanks!

By default, individual cpu is not imported(option import_cpu_all is set to false) but it could be interesting to display individual cpu chart to perform deeper analysis.

The work you have added is a major leap forward. The ability to sample data via the HMC is exactly what we needed. Thx!

It appears that when nmon2influxdb runs against the same directory it loads all of the data in the directory rather than the delta from the last run. Is that purposeful? The earlier version used the nmon2influxdb_log db to checkpoint the prior run and start from there.

If the current design will process all of the files in the directory, we can adjust our approach and move them out after processing so we won't get duplicated data. We use rsync to copy the delta from the prior {x} minutes so that the user can have a near-realtime view.

Thanks!
Alan

Would be nice to not have password stored in clear text. ;)

Hello,

First thank you for your very usefull tool !

Im trying to compile nmon2influxdb on Windows but I have a problem with

 go build
# github.com/adejoux/nmon2influxdb
.\nmon_importer.go:71: influxdbLog.ReadLastPoint undefined (type *influxdbclient.InfluxDB has no field or method ReadLastPoint)
.\nmon_importer.go:82: influxdbLog.ReadLastPoint undefined (type *influxdbclient.InfluxDB has no field or method ReadLastPoint)

I can't find ReadLastPoint on influxdbclient.
Could you help me.

Another minor thing :

go get -u github.com/adejoux/nmon2influxdb
# cd C:\CPPPERFS\gowork\src\github.com\adejoux\nmon2influxdb; git submodule update --init --recursive
fatal: No url found for submodule path 'docs/themes/darkdoc' in .gitmodules
package github.com/adejoux/nmon2influxdb: exit status 128

I managed to go further by removing the '-u'

Thank you again
n3biot

Would allow almost continuous load in influxdb.
The tool will cache at what timestamp it stopped on thie file and will resume from it.

Doc says

You can skip disks metrics with nd option

but the help text says

--nodisks, --nd add disk metrics

So does it skip them, or add them? :)

Hi,
Trier to use nmon2influxdb with influxdb 1.1.1
When importing a new server with nmon2influxdb import nmon-file I got the error that the default retention policy is not found. Ich found out that influxdb changed the parameter.
Can you please add support for influxdb 1.1.1 and up.

# ./nmon2influxdb import DBA1_170130_0000.nmon
2017/01/31 10:29:12 {"error":"retention policy not found: default"}

Thanks

Is is feasible to move the configuration file to the /etc directory? Perhaps in an /etc/nmon2influxdb directory?

This is a cool feature!
But it expected the data source in grafana to be defined as nmon2influxdb - it would be great if this could be configurable. Or at least, add a note in the CLI output so that the user knows to setup grafana accordingly.

Would be nice to have it on the network charts.

Not sure if this should work, or I need some extra steps? I just copied & pasted the three lines from the README.md, as I saw that there's an updated codebase but the release version has not changed, so I thought I'd build it locally myself.

    ec2-user@controller:nmon () $ git clone https://github.com/adejoux/nmon2influxdb
    Cloning into 'nmon2influxdb'...
    remote: Counting objects: 196, done.
    remote: Total 196 (delta 0), reused 0 (delta 0), pack-reused 196
    Receiving objects: 100% (196/196), 2.02 MiB | 1.26 MiB/s, done.
    Resolving deltas: 100% (122/122), done.
    Checking connectivity... done.
    ec2-user@controller:nmon () $ cd nmon2influxdb
    ec2-user@controller:nmon2influxdb (master) $ go build
    nmon_dashboard.go:12:2: cannot find package "github.com/adejoux/grafanaclient" in any of:
            /usr/lib/golang/src/github.com/adejoux/grafanaclient (from $GOROOT)
            ($GOPATH not set)
    nmon_importer.go:10:2: cannot find package "github.com/adejoux/influxdbclient" in any of:
            /usr/lib/golang/src/github.com/adejoux/influxdbclient (from $GOROOT)
            ($GOPATH not set)
    helpers.go:8:2: cannot find package "github.com/codegangsta/cli" in any of:
            /usr/lib/golang/src/github.com/codegangsta/cli (from $GOROOT)
            ($GOPATH not set)
    ec2-user@controller:nmon2influxdb (master) $

THX for your work but I have a problem to use your code.

I received the following output:

# /opt/nmon2influxdb/nmon2influxdb-linux-amd64 --debug hmc import                                                │[root@epinflux01-bs log]# tail -f /var/log/messages
[{[{databases map[] [name] [[_internal] [server_power]] <nil>}] }]                                                                   │Nov 23 11:00:31 epinflux01-bs influxd: [httpd] 172.17.68.32 - hmc [23/Nov/2016:11:00:31 +0100] "GET /query?db=_internal&q=show+databases%
ALTER RETENTION POLICY "autogen" ON "nmon2influxdbHMC" DURATION 60d REPLICATION 1 DEFAULT                                            │3Bdrop+database+nmon2influxdbHMC%3B HTTP/1.1" 200 233 "http://10.76.64.9:8083/?" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36
2016/11/23 11:00:06 Error status code: 401

The 'nmon2influxdbHMC' was created inside the influxdb so the config should be correct. I can also use the influxdb Webgui to

ALTER RETENTION POLICY "autogen" ON "nmon2influxdbHMC" DURATION 60d REPLICATION 1 DEFAULT

but your code again stop at this point.

Do you have some hints for me?

regards,
Sascha

data was sent multiple times to influxdb. Fixed in influxdbclient.

Import will be a lot faster in release 2.1.3.

Hi Alain,

"Disk %Busy" seems to display "Disk IO Reads per second". I compared an imported nmon file between the Grafana and NMON Visualiser.

Thanks!

Try to support remote file import with this syntax:

nmon2influxdb import remote_ip:/dir/file.nmon

Would be nice to not have to use "dashboard file" when creating a dashboard from a NMON file and "dashboard template" when creating it from a TOML template.
Would be simpler to have :

nmon2influxdb  dashboard testsrv.nmon

And

nmon2influxdb  dashboard template.toml

The tool should detect automatically the file type.