Code Monkey home page Code Monkey logo

snap-plugin-collector-scaleio's Introduction

DISCONTINUATION OF PROJECT.

This project will no longer be maintained by Intel.

This project has been identified as having known security escapes.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project. Build Status

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

snap collector plugin - ScaleIO

This plugin supports collecting metrics from a ScaleIO cluster

It's used in the Snap framework.

  1. Getting Started
  2. Documentation
  3. Roadmap
  4. Community Support
  5. Contributing
  6. License
  7. Acknowledgements

Getting Started

System Requirements

Installation

Download File plugin binary:

You can get the pre-built binaries for your OS and architecture at the plugin's GitHub Releases page.

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-collector-scaleio
Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-collector-scaleio.git

Build the plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build/

Configuration and Usage

First, be sure that you've familiarized yourself with the Snap framework by reading the Getting Started documentation.

This plugin requires several config options to run properly:

Required

  • gateway: This is the URL of the gateway server.
  • username and password: A username and password with the appropriate access to the REST API

Optional

  • verifySSL: If set to false this disables SSL validation. This should not be used in production.

A full config example is below:

"/intel/scaleio": {
    "username": "admin",
    "password": "password",
    "gateway": "https://my-cluster",
    "verifySSL": false
}

Documentation

Collected Metrics

List of metrics collected by this plugin can be found in METRICS.md file.

All metrics are exposed with a dynamic namespace that encompasses each StoragePool. You can collect metrics from all of them or specify a storage pool that you are interested by putting its name instead wildcard - see how to specify the instance of dynamic metric in Snap framework documentation.

Examples

There is an example config found in the examples directory.

Example watch

NAMESPACE 							 DATA 		 TIMESTAMP
/intel/scaleio/96eb24f700000000/bckRebuildWriteBwc/numOccured 	 0 		 2016-07-08 23:16:02.304238351 -0700 PDT
/intel/scaleio/96eb24f700000000/pendingMovingOutBckRebuildJobs 	 0 		 2016-07-08 23:16:02.304238351 -0700 PDT
/intel/scaleio/96eb24f700000000/snapCapacityInUseInKb 		 3.145728e+06 	 2016-07-08 23:16:02.304238351 -0700 PDT

Roadmap

This is currently in Alpha. Please let us know of any bugs you see.

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework or join the Snap Slack channel.

Contributing

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

And thank you! Your contribution, through code and participation, is incredibly important to us.

snap-plugin-collector-scaleio's People

Contributors

izabellaraulin avatar katarzyna-z avatar kdembler avatar king-jam avatar lmroz avatar nanliu avatar nguyenddn avatar rdower avatar taotod avatar thomastaylor312 avatar zhengqingyuan avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

snap-plugin-collector-scaleio's Issues

Plugin crashes from time to time

Similar issue as described here:
#3

After crash, I see in logs:

time="2017-12-12T09:25:59+02:00" level=error msg="collector run error" _module=scheduler-job block=run error="CollectMetrics call error : Error while parsing data from /api/types/StoragePool/instances: json: cannot unmarshal object into Go value of type []map[string]interface {}" job-type=collector 
time="2017-12-12T09:25:59+02:00" level=warning msg="Task failed" _block=spin _module=scheduler-task consecutive failure limit=-1 consecutive failures=1653 error="CollectMetrics call error : Error while parsing data from /api/types/StoragePool/instances: json: cannot unmarshal object into Go value of type []map[string]interface {}" task-id=06405472-dbc7-46a3-ae03-78b2b5f03e3e task-name=Task-06405472-dbc7-46a3-ae03-78b2b5f03e3e 

Task file:

{
    "version": 1,
    "schedule": {
        "type": "simple",
        "interval": "10s"
    },
    "max-failures": -1,
    "workflow": {
        "collect": {
            "metrics": {
                "/intel/scaleio/*/BackgroundScanCompareCount": {},
                "/intel/scaleio/*/BackgroundScannedInMB": {},
                "/intel/scaleio/*/activeBckRebuildCapacityInKb": {},
                "/intel/scaleio/*/activeFwdRebuildCapacityInKb": {},
                "/intel/scaleio/*/activeMovingCapacityInKb": {},
                "/intel/scaleio/*/activeMovingInBckRebuildJobs": {},
                "/intel/scaleio/*/activeMovingInFwdRebuildJobs": {},
                "/intel/scaleio/*/activeMovingInNormRebuildJobs": {},
                "/intel/scaleio/*/activeMovingInRebalanceJobs": {},
                "/intel/scaleio/*/activeMovingOutBckRebuildJobs": {},
                "/intel/scaleio/*/activeMovingOutFwdRebuildJobs": {},
                "/intel/scaleio/*/activeMovingOutNormRebuildJobs": {},
                "/intel/scaleio/*/activeMovingRebalanceJobs": {},
                "/intel/scaleio/*/activeNormRebuildCapacityInKb": {},
                "/intel/scaleio/*/activeRebalanceCapacityInKb": {},
                "/intel/scaleio/*/atRestCapacityInKb": {},
                "/intel/scaleio/*/bckRebuildCapacityInKb": {},
                "/intel/scaleio/*/bckRebuildReadBwc/numOccured": {},
                "/intel/scaleio/*/bckRebuildReadBwc/numSeconds": {},
                "/intel/scaleio/*/bckRebuildReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/bckRebuildWriteBwc/numOccured": {},
                "/intel/scaleio/*/bckRebuildWriteBwc/numSeconds": {},
                "/intel/scaleio/*/bckRebuildWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/capacityAvailableForVolumeAllocationInKb": {},
                "/intel/scaleio/*/capacityInUseInKb": {},
                "/intel/scaleio/*/capacityLimitInKb": {},
                "/intel/scaleio/*/degradedFailedCapacityInKb": {},
                "/intel/scaleio/*/degradedFailedVacInKb": {},
                "/intel/scaleio/*/degradedHealthyCapacityInKb": {},
                "/intel/scaleio/*/degradedHealthyVacInKb": {},
                "/intel/scaleio/*/failedCapacityInKb": {},
                "/intel/scaleio/*/failedVacInKb": {},
                "/intel/scaleio/*/fixedReadErrorCount": {},
                "/intel/scaleio/*/fwdRebuildCapacityInKb": {},
                "/intel/scaleio/*/fwdRebuildReadBwc/numOccured": {},
                "/intel/scaleio/*/fwdRebuildReadBwc/numSeconds": {},
                "/intel/scaleio/*/fwdRebuildReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/fwdRebuildWriteBwc/numOccured": {},
                "/intel/scaleio/*/fwdRebuildWriteBwc/numSeconds": {},
                "/intel/scaleio/*/fwdRebuildWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/inMaintenanceCapacityInKb": {},
                "/intel/scaleio/*/inMaintenanceVacInKb": {},
                "/intel/scaleio/*/inUseVacInKb": {},
                "/intel/scaleio/*/maxCapacityInKb": {},
                "/intel/scaleio/*/movingCapacityInKb": {},
                "/intel/scaleio/*/normRebuildCapacityInKb": {},
                "/intel/scaleio/*/normRebuildReadBwc/numOccured": {},
                "/intel/scaleio/*/normRebuildReadBwc/numSeconds": {},
                "/intel/scaleio/*/normRebuildReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/normRebuildWriteBwc/numOccured": {},
                "/intel/scaleio/*/normRebuildWriteBwc/numSeconds": {},
                "/intel/scaleio/*/normRebuildWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/numOfDevices": {},
                "/intel/scaleio/*/numOfMappedToAllVolumes": {},
                "/intel/scaleio/*/numOfSnapshots": {},
                "/intel/scaleio/*/numOfThickBaseVolumes": {},
                "/intel/scaleio/*/numOfThinBaseVolumes": {},
                "/intel/scaleio/*/numOfUnmappedVolumes": {},
                "/intel/scaleio/*/numOfVolumes": {},
                "/intel/scaleio/*/numOfVolumesInDeletion": {},
                "/intel/scaleio/*/numOfVtrees": {},
                "/intel/scaleio/*/pendingBckRebuildCapacityInKb": {},
                "/intel/scaleio/*/pendingFwdRebuildCapacityInKb": {},
                "/intel/scaleio/*/pendingMovingCapacityInKb": {},
                "/intel/scaleio/*/pendingMovingInBckRebuildJobs": {},
                "/intel/scaleio/*/pendingMovingInFwdRebuildJobs": {},
                "/intel/scaleio/*/pendingMovingInNormRebuildJobs": {},
                "/intel/scaleio/*/pendingMovingInRebalanceJobs": {},
                "/intel/scaleio/*/pendingMovingOutBckRebuildJobs": {},
                "/intel/scaleio/*/pendingMovingOutFwdRebuildJobs": {},
                "/intel/scaleio/*/pendingMovingOutNormrebuildJobs": {},
                "/intel/scaleio/*/pendingMovingRebalanceJobs": {},
                "/intel/scaleio/*/pendingNormRebuildCapacityInKb": {},
                "/intel/scaleio/*/pendingRebalanceCapacityInKb": {},
                "/intel/scaleio/*/primaryReadBwc/numOccured": {},
                "/intel/scaleio/*/primaryReadBwc/numSeconds": {},
                "/intel/scaleio/*/primaryReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/primaryReadFromDevBwc/numOccured": {},
                "/intel/scaleio/*/primaryReadFromDevBwc/numSeconds": {},
                "/intel/scaleio/*/primaryReadFromDevBwc/totalWeightInKb": {},
                "/intel/scaleio/*/primaryReadFromRmcacheBwc/numOccured": {},
                "/intel/scaleio/*/primaryReadFromRmcacheBwc/numSeconds": {},
                "/intel/scaleio/*/primaryReadFromRmcacheBwc/totalWeightInKb": {},
                "/intel/scaleio/*/primaryVacInKb": {},
                "/intel/scaleio/*/primaryWriteBwc/numOccured": {},
                "/intel/scaleio/*/primaryWriteBwc/numSeconds": {},
                "/intel/scaleio/*/primaryWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/protectedCapacityInKb": {},
                "/intel/scaleio/*/protectedVacInKb": {},
                "/intel/scaleio/*/rebalanceCapacityInKb": {},
                "/intel/scaleio/*/rebalanceReadBwc/numOccured": {},
                "/intel/scaleio/*/rebalanceReadBwc/numSeconds": {},
                "/intel/scaleio/*/rebalanceReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/rebalanceWriteBwc/numOccured": {},
                "/intel/scaleio/*/rebalanceWriteBwc/numSeconds": {},
                "/intel/scaleio/*/rebalanceWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/rfacheReadHit": {},
                "/intel/scaleio/*/rfcacheAvgReadTime": {},
                "/intel/scaleio/*/rfcacheAvgWriteTime": {},
                "/intel/scaleio/*/rfcacheIoErrors": {},
                "/intel/scaleio/*/rfcacheIosOutstanding": {},
                "/intel/scaleio/*/rfcacheIosSkipped": {},
                "/intel/scaleio/*/rfcacheReadMiss": {},
                "/intel/scaleio/*/rfcacheReadsFromCache": {},
                "/intel/scaleio/*/rfcacheReadsPending": {},
                "/intel/scaleio/*/rfcacheReadsReceived": {},
                "/intel/scaleio/*/rfcacheReadsSkipped": {},
                "/intel/scaleio/*/rfcacheReadsSkippedAlignedSizeTooLarge": {},
                "/intel/scaleio/*/rfcacheReadsSkippedHeavyLoad": {},
                "/intel/scaleio/*/rfcacheReadsSkippedInternalError": {},
                "/intel/scaleio/*/rfcacheReadsSkippedLockIos": {},
                "/intel/scaleio/*/rfcacheReadsSkippedLowResources": {},
                "/intel/scaleio/*/rfcacheReadsSkippedMaxIoSize": {},
                "/intel/scaleio/*/rfcacheReadsSkippedStuckIo": {},
                "/intel/scaleio/*/rfcacheSkippedUnlinedWrite": {},
                "/intel/scaleio/*/rfcacheSourceDeviceReads": {},
                "/intel/scaleio/*/rfcacheSourceDeviceWrites": {},
                "/intel/scaleio/*/rfcacheWriteMiss": {},
                "/intel/scaleio/*/rfcacheWritePending": {},
                "/intel/scaleio/*/rfcacheWritesReceived": {},
                "/intel/scaleio/*/rfcacheWritesSkippedCacheMiss": {},
                "/intel/scaleio/*/rfcacheWritesSkippedHeavyLoad": {},
                "/intel/scaleio/*/rfcacheWritesSkippedInternalError": {},
                "/intel/scaleio/*/rfcacheWritesSkippedLowResources": {},
                "/intel/scaleio/*/rfcacheWritesSkippedMaxIoSize": {},
                "/intel/scaleio/*/rfcacheWritesSkippedStuckIo": {},
                "/intel/scaleio/*/rmPendingAllocatedInKb": {},
                "/intel/scaleio/*/secondaryReadBwc/numOccured": {},
                "/intel/scaleio/*/secondaryReadBwc/numSeconds": {},
                "/intel/scaleio/*/secondaryReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/secondaryReadFromDevBwc/numOccured": {},
                "/intel/scaleio/*/secondaryReadFromDevBwc/numSeconds": {},
                "/intel/scaleio/*/secondaryReadFromDevBwc/totalWeightInKb": {},
                "/intel/scaleio/*/secondaryReadFromRmcacheBwc/numOccured": {},
                "/intel/scaleio/*/secondaryReadFromRmcacheBwc/numSeconds": {},
                "/intel/scaleio/*/secondaryReadFromRmcacheBwc/totalWeightInKb": {},
                "/intel/scaleio/*/secondaryVacInKb": {},
                "/intel/scaleio/*/secondaryWriteBwc/numOccured": {},
                "/intel/scaleio/*/secondaryWriteBwc/numSeconds": {},
                "/intel/scaleio/*/secondaryWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/semiProtectedCapacityInKb": {},
                "/intel/scaleio/*/semiProtectedVacInKb": {},
                "/intel/scaleio/*/snapCapacityInUseInKb": {},
                "/intel/scaleio/*/snapCapacityInUseOccupiedInKb": {},
                "/intel/scaleio/*/spareCapacityInKb": {},
                "/intel/scaleio/*/thickCapacityInUseInKb": {},
                "/intel/scaleio/*/thinCapacityAllocatedInKm": {},
                "/intel/scaleio/*/thinCapacityInUseInKb": {},
                "/intel/scaleio/*/totalReadBwc/numOccured": {},
                "/intel/scaleio/*/totalReadBwc/numSeconds": {},
                "/intel/scaleio/*/totalReadBwc/totalWeightInKb": {},
                "/intel/scaleio/*/totalWriteBwc/numOccured": {},
                "/intel/scaleio/*/totalWriteBwc/numSeconds": {},
                "/intel/scaleio/*/totalWriteBwc/totalWeightInKb": {},
                "/intel/scaleio/*/unreachableUnusedCapacityInKb": {},
                "/intel/scaleio/*/unusedCapacityInKb": {}
            },
            "config": {
                    "/intel/scaleio": {
                    "username": "<api_user>",
                    "password": "<and_pwd>",
                    "gateway": "https://<gw_ip>",
                    "verifySSL": false
                }
            },
            "publish": [
                        {
                            "plugin_name": "influxdb",
                            "config": {
                                "host": "127.0.0.1",
                                "port": 8086,
                                "database": "<some_db>",
                                "user": "admin",
                                "password": "<super_secret>",
                                "scheme": "http"
                    }
                }
            ]
        }
    }
}

I tried to "avoid" this crash with
"max-failures": -1,

No luck.
Basically I'd like to collect cluster statistics with snap, send it to influxdb and visualize with grafana. Works well until plugin crash.

snapteld -v
snapteld version 2.0.0

cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 

kernel: 3.10.0-693.5.2.el7.x86_64

Scale IO telemetry task create fails with required key missing

Snap daemon version (use snapteld -v):

root@b20b582634a0:/src/snapd# ./snapteld -v
snapteld version 1.0.0
root@b20b582634a0:/src/sna

Environment:

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Ubuntu 15.10
  • Kernel (e.g. uname -a):
    Linux b20b582634a0 4.8.0-34-generic #36-Ubuntu SMP Wed Dec 21 17:24:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Relevant tools (e.g. plugins used with Snap):
root@ced81655974c:/src/snapd# ./snaptel plugin list
NAME 		 VERSION 	 TYPE 		 SIGNED 	 STATUS 	 LOADED TIME
elasticsearch 	 4 		 collector 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:48 UTC
mongodb 	 2 		 collector 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:48 UTC
psutil 		 9 		 collector 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:48 UTC
scaleio 	 1 		 collector 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:49 UTC
file 		 2 		 publisher 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:49 UTC
cassandra 	 3 		 collector 	 false 		 loaded 	 Mon, 16 Jan 2017 01:49:47 UTC
root@ced81655974c:/src/snapd#
  • Others (e.g. deploying with Ansible):

What happened:

root@37714a74d115:/src/snapd# ./snaptel task create -t ./m2.json 
Using task manifest to create task
Error creating task:required key missing (gateway)
required key missing (username)
required key missing (password)
required key missing (verifySSL)

What you expected to happen:
Task should have been created.

Steps to reproduce it (as minimally and precisely as possible):

  1. run snapteld version 1.0.0 with latest scale io collector plugin.
  2. Using the following JSON to create the task.
{
    "version": 1,
    "schedule": {
        "type": "simple",
        "interval": "10s"
    },
    "max-failures": 10,
    "workflow": {
        "collect": {
            "metrics": {
                "/intel/scaleio/*/pendingMovingOutBckRebuildJobs": {},
                "/intel/scaleio/*/snapCapacityInUseInKb": {},
                "/intel/scaleio/*/bckRebuildWriteBwc/numOccured": {}
            },
            "config": {
		"/intel/scaleio": { 
                    "username": "monitor_only",
                    "password": "Fakepassword",
                    "gateway": "https://my-cluster",
                    "verifySSL": "false"
		}
            },
	   "publish": [
                        {
                            "plugin_name": "file",                            
                            "config": {
                                "file": "/tmp/snap_published_scaleio_file.log"
                            }
                        }
           ]
        }
    }
}

Anything else do we need to know (e.g. issue happens only occasionally):

Create integration tests

Write integration tests for the data collection from a ScaleIO gateway host. Right now this is difficult because there is no Docker image for it. Any suggestions from ScaleIO users would be helpful.

METRICS.md file

It would be nice to have separated METRICS.md file with table with columns "namespace", "data type" and "description" like here.

CollectMetrics call error : Error while parsing

Hi,

I've some issue with this plugin, it failed and disabled task as soon as API returns doesn't suit expectation.

My task worked fine during one day then failed ... Last error is : CollectMetrics call error : Error while parsing data from /api/types/StoragePool/instances: json: cannot unmarshal object into Go value of type []map[string]interface {}

Metrics collected : "metrics": { "/intel/scaleio/*/capacityInUseInKb": {}, "/intel/scaleio/*/capacityLimitInKb": {}, "/intel/scaleio/*/unusedCapacityInKb": {}, "/intel/scaleio/*/spareCapacityInKb": {}, "/intel/scaleio/*/failedCapacityInKb": {}, "/intel/scaleio/*/movingCapacityInKb": {}, "/intel/scaleio/*/maxCapacityInKb": {}, "/intel/scaleio/*/numOfDevices": {}, "/intel/scaleio/*/numOfVolumes": {}, "/intel/scaleio/*/totalReadBwc/*": {}, "/intel/scaleio/*/totalWriteBwc/*": {} },

Environment :

  • Snap in docker container
  • ScaleIO 1.32.5 clusters
  • CentOS 7

I'll continue to analyze it.

Regards,
Red

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.