graphite-project / carbonate Goto Github PK

View Code? Open in Web Editor NEW

515.0 515.0 80.0 235 KB

Utilities for managing graphite clusters

License: MIT License

Ruby 0.20% Python 97.67% Shell 2.13%

carbonate graphite graphite-clusters python

carbonate's People

Contributors

Stargazers

Watchers

Forkers

slillibri prfalken selfsk slyness bridgetkromhout ecanzonieri garrettux tail dimrozakis earthgecko butangero jaimegago eredding-rmn bitprophet seanpquig pkittenis criteo-forks kamaradclimber asana bruce-lyft skbkontur jjneely markrechler pombredanne kjoconnor piotr1212 shadow4125 xinity dmreiland mrmanc autotraderuk klynch ross gwaldo intfrr syaroslavtsev halcyonardency melza ainestal tgavriltg djoffrey tmack0 dozoisch chaturvedia deniszh linebreaker iliapolo guoyu07 princessd8251 yunstanford justincmoy iksaif teotikalki qiyishi jehiah cjhp1314 deejay1 zoidyzoidzoid ebuzzing fengyehong alemsh yevhenlukomskyi luke-heberling kbiernat smartree phymucs laashub-soa ryangsteele isabella232 aastha12 drawks fgsilva mdaffern reyjrar seanpm2001 kiraum

carbonate's Issues

option to use whisper.LOCK during sync/fill

splitting this issue from #19 since they are really two different things

carbonate should use whisper.LOCK if it can during sync/fill like carbon does to prevent corrupting whisper files if multiple writers are present (e.g. carbon-cache and carbonate)

When is aggregated-consistent-hashing supported?

Fix lint errors

Like https://travis-ci.org/graphite-project/carbonate/jobs/471662778

Ability to set different STORAGE_DIR for local vs remote

I'm trying to run carbon-sync where the remote STORAGE_DIR is different from the local STORAGE_DIR. I don't see any way to handle this currently. Make sense?

whisper-fill dst file doesn't exists

Hi,

I tried whisper-fill and it works mostly as a charm. But I have one issue with it.

It fails with a traceback if the dst file doesn’t exists.

From my point of view it's not that uncommon that at the new location not all whisper files are present after rearranging the destination.

Would it make sense for whisper-fill to simple put the src whisper file as it is at the destination if there is no file?

carbon-hosts fails

carbonate verison = 1.1.2
carbon version = 1.1.2

Installed from pip, only modified the logging location and the LOCAL_STORAGE_DIR.
OS: CentOS 7.4

I get the following error when using "carbon-hosts -C xxxxx".

Traceback (most recent call last):
File "/usr/bin/carbon-hosts", line 10, in
load_entry_point('carbonate==1.1.2', 'console_scripts', 'carbon-hosts')()
File "/usr/lib/python2.7/site-packages/carbonate/cli.py", line 35, in carbon_hosts
cluster = Cluster(config, args.cluster)
File "/usr/lib/python2.7/site-packages/carbonate/cluster.py", line 31, in init
r = ConsistentHashingRouter(Settings())
File "/opt/graphite/lib/carbon/routers.py", line 80, in init
hash_type = settings.ROUTER_HASH_TYPE or 'carbon_ch'
AttributeError: 'Settings' object has no attribute 'ROUTER_HASH_TYPE'

Current config is:

[xxxxx]
DESTINATIONS = 10.xxx.xxx.xxx:2004:server-a, 10.xxx.xxx.xxx:2004:server-b
REPLICATION_FACTOR = 2
SSH_USER = carbon

I am not sure if I should add all servers in the cluster to this config or exclude the local server, but the error occours with both configurations anyway.

Thanks for your help!

Traceback on carbon-sync

Seeing this traceback on a specific whisper file. I can make the file available if it helps with diagnosis.

Traceback (most recent call last):
  File "/usr/local/bin/carbon-sync", line 9, in <module>
    load_entry_point('carbonate==0.2.0', 'console_scripts', 'carbon-sync')()
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 190, in main
    run_batch(metrics_to_sync, remote, args.storage_dir)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 119, in run_batch
    merge_elapsed = sync_batch(metrics_to_heal)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 60, in sync_batch
    heal_metric(staging, local)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 75, in heal_metric
    fill_archives(source, dest, time())
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 85, in fill_archives
    fill(src, dst, gapstart - step, start)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 50, in fill
    (timeInfo, values) = fetch(src, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 711, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 786, in file_fetch
    unpackedSeries = struct.unpack(seriesFormat, seriesString)
struct.error: unpack requires a string argument of length 1524

Carbonate doesn't find Carbon already installed

Hi everyone,

I have a problem with the carbonate dependencies. I had already installed carbon and whisper but carbonate doesn't find carbon neither in the installation neither in the execution. I'm going to detail the steps that I followed:

Download carbonate: git clone https://github.com/jssjr/carbonate
Install carbonate: python setup.py install

Here I saw the first error, it seems that carbonate is successfully installed but the process doesn't find carbon:

Installed /usr/local/lib/python2.7/dist-packages/carbonate-0.2.1-py2.7.egg
Processing dependencies for carbonate==0.2.1
Searching for carbon
Reading http://pypi.python.org/simple/carbon/
Download error on http://pypi.python.org/simple/carbon/: timed out -- Some packages may not be found!
Reading http://pypi.python.org/simple/carbon/
Download error on http://pypi.python.org/simple/carbon/: timed out -- Some packages may not be found!
Couldn't find index page for 'carbon' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Download error on http://pypi.python.org/simple/: timed out -- Some packages may not be found!
No local packages or download links found for carbon
error: Could not find suitable distribution for Requirement.parse('carbon')

Search if the program is installed: pip freeze. Is installed the 0.2.1 version of carbonate.
Execute some commands to see if it works: carbon-hosts --help

Here is the second error:

Traceback (most recent call last):
File "/usr/local/bin/carbon-hosts", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in
working_set.require(requires)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: carbon

Can someone help me with this?

Thank you very much!

Description is not properly rendered on pypi.org

See description on https://pypi.org/project/carbonate/

carbon-lookup returns nodes on bad metric

not sure if this is intended but for some reason when i lookup a metric i know is not in graphite carbon-lookup still returns nodes.

bad metric:

# carbon-lookup foo.bar
10.1.1.211:2004:graphite-node-0
10.1.1.209:2004:graphite-node-2

good metric:

# carbon-lookup local.random.diceroll
10.1.1.210:2004:graphite-node-1
10.1.1.211:2004:graphite-node-0

Different "consistent hashing" results from Carbon and Carbonate

Carbon-sieve reports that some metrics belong to another node but these metrics are actually used and updated by carbon.

DESTINATIONS are defined identical in carbon.conf and carbonate.conf. All daemons are restarted and using this configuration.

Roughly 1/8 of all metrics are reported to belong to another host, the rest is fine.

Cluster has 6 nodes. 1 has haproxy which is load-balancing requests to 8 carbon-relays on the same machine. These carbon-relays use consistent hashing and DESTINATIONS = 172.22.5.14:2014:relay, 172.22.5.106:2014:relay, 172.22.5.107:2014:relay, 172.22.5.234:2014:relay, 172.22.5.235:2014:relay. Each of the entries in DESTINATIONS is a carbon-relay on the other 5 servers and has DESTINATIONS = 127.0.0.1:2004:cache, 127.0.0.1:2104:b, two carbon-cache instances using the same whisper storage path.

When I was using carbin-sieve to clean-up some stuff, I noticed that its results are wrong.

Example: rg.community.pagespeed.publicationDetail.loggedOut.connect.median

File exists on host with IP 172.22.5.14:

$ stat /data/graphite/whisper/rg/community/pagespeed/publicationDetail/loggedOut/connect/median.wsp
  File: ‘/data/graphite/whisper/rg/community/pagespeed/publicationDetail/loggedOut/connect/median.wsp’
  Size: 325816      Blocks: 640        IO Block: 4096   regular file
[...]
Access: 2016-03-06 08:50:09.246530787 +0000
Modify: 2016-05-01 12:58:52.698880921 +0000
Change: 2016-05-01 12:58:52.698880921 +0000
 Birth: -

But carbon-sieve wants it on 172.22.5.106:

$ echo "rg.community.pagespeed.publicationDetail.loggedOut.connect.median" | carbon-sieve -C main -n 172.22.5.106
rg.community.pagespeed.publicationDetail.loggedOut.connect.median

I am troubleshooting this since hours but cannot find a reason. I also noticed that carbon-sieve is using the hashing method from carbon library. Approximately 493000 of 4.11 million metrics are wrong, between 9.5% and 13.5% of the metrics per node. This is close to 12,5% (1/8) like the 8 carbon-relays in haproxy.

Any hints are highly appreciated. If you need more information, please do not hesitate to ask.

carbon-sync -s SOURCE_SERVER modify also source server metrics !

Hi @jssjr

I have 2 graphites completely mirrored between them in 2 datacenters. ( same metrics replicated with carbon-relay and REPLICATION FACTOR=2)

[GRAPHITE_SERVER_1] <----(replicated data)----> [ GRAPHITE_SERVER_2]

Yesterday datacenter 2 crash for 4 hours, after all servers recovered I sow that "GRAPHITE_SERVER_2" leaks for 4 hous of data. ( GRAPHITE_SERVER_1 has all data

I've tried to recover with this command executed in GRAPHITE_SERVER_2

#ssh GRAPHITE_SERVER_1 -- carbon-list -d /metrics | carbon-sync  -s GRAPHITE_SERVER_1 --source-storage-dir /metrics -d /metrics

After all synced I've noted that metrics in GRAPHITE_SERVER_1 has changed !!

What is the correct command to copy metrics from GRAPHITE_SERVER_1 to GRAPHITE_SERVER_2 ( without modify GRAPHITE_SERVER_1 metrics ?) ( only a one direction copy)

Allow carbon-sync to delete remote data after successful sync

After a successful carbon-sync , we eventually want to remove the original metric.

We cannot use last modification time (such as find /opt/graphite/storage/whisper -type f -mtime +2) since it will remove metric not updated (because there was no new point).
Any combination with carbon-sieve won't help us distinguish them either because carbon-sync ignore (logs actually) some errors.

It would be nice for carbon-sync to output metric that were successfully updated on its output to be able launch a delete.

We are currently doing this with a checksum of ``whisper-fetch.py --until xxx output (fetching all points up to now-1hour), but this is far from perfect.

read defaults from environment too?

hi,
I think some defaults for carbonate could be read from the environment too, e.g. the storage paths or the config file location. This would make it easier to e.g. switch to another config for all carbonate tools from the command line or more complex wrappers. If there's interest I'll be happy to submit a PR, what do you think?

thanks,
filippo

Intermittent rsync errors

Hi,
I'm getting intermittent rsync errors while running a script based on the 'Rebalance a cluster' example. The errors look like below:

WARNING:root:Failed to copy /var/lib/graphite/whisper/myprdbox1/disk/disk/mapper/rootvg-rootlv/avail.wsp! [Errno 2] No such file or directory: '/tmp/tmpgzUPfI/myprdbox1/disk/disk/mapper/rootvg-rootlv/avail.wsp'

Rerunning the script rarely yields the same errors. Sometimes it finishes cleanly, throwing no errors at all.
Googling and troubleshooting have yet to yield any satisfying results. Do you have any insight into what might cause this behavior? Would this kind of thing happen if attempting to rsync a database that's currently being written to?

If there's specific data I can provide to help narrow the scope, let me know.

Thanks!

Passing a file with metric names to carbon-lookup

Hi,

I have a need to pass a file with metric names to carbon-lookup. Currently it seems to accept only a single metric name. Is there a way a clever way to use STDIN or a file with metric names currently?

I've seen the implementation of carbon-lookup and it's rather straightforward to take that as the base and write my own (tweaked) version that would take a file, but I would like to do that only as a last resort.

Thanks!

Carbonate can't find carbon installed in virtualenv (or non virtualenv on Mac OS X)

Hey there,

When I install carbonate in a virtualenv on my local laptop (Mac OS X 10.11) or on a Debian 7 server, carbonate fails to find graphite and errors when I try to run any script:

$ carbon-hosts
No bueno. Can't import carbon! (No module named carbon)

But I can see that carbon is installed:

$ pip freeze | grep carbon
carbon==0.9.15
carbonate==0.2.2

The same thing happens on both the Mac OS X and Debian 7 machines. But if I exit the virtualenv on the Debian machine, then apt-get install graphite-carbon and install carbonate into global site-packages with pip install carbonate, it works:

root@test-node01:~# pip install carbonate
[...]
root@test-node01:~# carbon-hosts
Cluster 'main' not defined in /opt/graphite/conf/carbonate.conf  # it works!

Similarly, if I install carbon into the global site-packages using pip (after apt-get purge graphite-carbon to remove the Debian-packaged version) it also works fine on Debian:

root@edge-node01:~# pip install carbon
Downloading/unpacking carbon
[...]
Successfully installed carbon
Cleaning up...
root@edge-node01:~# carbon-hosts
Cluster 'main' not defined in /opt/graphite/conf/carbonate.conf

BUT not on Mac OS X:

% pip install carbon carbonate  
[...]
% carbon-hosts
No bueno. Can't import carbon! (No module named carbon)

Cheers,
@scottcunningham

Add LICENSE file to pypi package

Hi! I am building a port for freebsd. The pypi package does not include a LICENSE file to the package. Is it possible to add this to future release of the pypi package?

To able to run on Debian: pkg_resources.DistributionNotFound: carbon

Hello! I want to try out the project it looks very useful, but so I have not been able to so far. I have installed whisper and carbon via pip, and I have tried to install carbonate via pip as well.

OS:

Distributor ID: Debian
Description: Debian GNU/Linux 7.6 (wheezy)
Release: 7.6
Codename: wheezy

PIP is installed via 'python-pip' debian package, so it is a pretty standard install:

/home/vagrant# pip --version
pip 1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

/home/vagrant# carbon-list
Traceback (most recent call last):
  File "/usr/local/bin/carbon-list", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: carbon

Am I missing something?

1.1.8 sync broken on python3

Hello,
thank you for carbonate! Super useful. I'm trying 1.1.8 on python3 and sync_from_remote is broken for me:

Traceback (most recent call last):
  File "/usr/bin/carbon-sync", line 34, in <module>
    sys.exit(load_entry_point('carbonate==1.1.8', 'console_scripts', 'carbon-sync')())
  File "/usr/lib/python3/dist-packages/carbonate/cli.py", line 272, in carbon_sync
    run_batch(metrics_to_sync, remote,
  File "/usr/lib/python3/dist-packages/carbonate/sync.py", line 151, in run_batch
    sync_success = sync_from_remote(sync_file, remote, staging_dir,
  File "/usr/lib/python3/dist-packages/carbonate/sync.py", line 34, in sync_from_remote
    sys.stdout.write(line.decode("utf-8"))
AttributeError: 'str' object has no attribute 'decode'

how to speed up carbon-sync ?

After a crash in one server with 40000 metrics I've recovered by doing a carbon-sync with another server with all replicated metrics.

The recovery process take as long as 10 hours.

After review the logs for each 1000 recovered metrics.

    Rsync time: 143.773407936s
    Merge time: 765.371428013s
    Total time: 909.144835949s

Merge time seems to take the 85% of the time.

there is any way to speed up this process?
Can I specify to sync only from/to , by examaple from="1 day ago" to decrease the total taken time?

Update docs explaining the CARBONATE_* envvars

#49 has been merged. The docs need to be updated to reflect this behavior.

/cc @filippog

piping carbon-list to carbon-sieve produces nothing

Hi,

When I pipe carbon-list to carbon-sieve on the same carbon cache host, I'm expecting to see all the metrics that are piped to sieve, but I see nothing. What could be the reason? The config contains all the members of the cluster correctly.

Thanks,
Sergei

edit:

Oh, I see:

Finally, you can provide HASHING_TYPE of your cluster. Default is carbon_ch, also fnv1a_ch is supported. Please note that for using fnv1a_ch hashing you need carbon 1.0.2 or newer installed (or you need to use carbon-c-relay relay instead).

edit2:

my carbonate is hopelessly old (Build Date : Wed 08 Jul 2015 02:16:26 PM UTC).

I have 0.9.16 and I'm using fnv1a_ch. Shouldn't I get an error to the effect that my hashing algorithm is not recognized?

carbon-lookup returns None for cluster

Not sure if this is just a weird misconfig on my part but when I do I carbon-lookup I get the following:

# carbon-lookup carbon.agents.graphite-node-2-a.activeConnections
10.1.1.146:2004:None
10.1.1.147:2004:None

Not sure where the "None" is coming from.

inspect.getargspec() is deprecated in python 3.0

So, we'll need to migrate to https://pypi.org/project/funcsigs if we want to have 2.7 compatibility in the future.

Carbonate does not work on older version of Whisper?

Hi,

I'm trying to use carbonate to sync an old cluster (0.9.9) to a new one. When I try to run carbon-list, though:

Traceback (most recent call last):
  File "/usr/local/bin/carbon-list", line 9, in <module>
    load_entry_point('carbonate==0.2.0', 'console_scripts', 'carbon-list')()
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.40-py2.7.egg/pkg_resources.py", line 343, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.40-py2.7.egg/pkg_resources.py", line 2325, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.40-py2.7.egg/pkg_resources.py", line 2031, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/cli.py", line 11, in <module>
    from .fill import fill_archives
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 17, in <module>
    from whisper import info, operator, fetch, update_many
ImportError: cannot import name operator

Issue where carbon-list comes back empty

Hi,

I have added the conf file as suggested with a user that has keys to the servers on the cluster, however when running the commands (IE carbon-list) it just comes back blank.

Any ideas why this might be. The only command I can see to run is if I run it on the local graphite server and tell it to look at the directory specifically.

Any ideas why when trying to run it again the cluster it just seems to bring back nothing not even an error.

carbon-sync does not fill in missing data

I'm moving from a single server to an HA setup with consistent hashing setup with a RF of 2, and higher resolution in storage-schemas.conf. I've rsynced all the metrics, run whisper-resize on them to have them store metrics for longer, and started using relay on the old single server to dual write metrics to both clusters.

Old metrics that were rsynced look fine, new metrics that came through the relay to be dual written look fine, but there's a gap in between that I'm trying to fix with carbon-sync, but it doesn't seem to work. It also takes a crazy amount of time to merge - 13204 seconds for 5 metrics.

Would trying to merge together metrics created with different storage-schemas cause this issue?

Error when running pip install carbonate

This is the output when running pip install carbonate

Collecting carbonate
  Downloading https://files.pythonhosted.org/packages/86/ec/778a8e8c35bd7eec6922d2fe68d174d43c884270f7962a25528844a2a8e0/carbonate-1.1.3.tar.gz
Collecting carbon (from carbonate)
  Downloading https://files.pythonhosted.org/packages/fb/c1/94b2a4dec84ddb8b9ecedb0211f96d27a8e3149644874e186e1d38b38814/carbon-1.1.3.tar.gz (68kB)
Collecting whisper (from carbonate)
  Downloading https://files.pythonhosted.org/packages/bd/5c/fcf525c20f10d9bcfc83464bcd44338d86d6748600876ca79e5d95781eb3/whisper-1.1.3.tar.gz
Requirement already satisfied: Twisted in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from carbon->carbonate) (17.9.0)
Collecting txAMQP (from carbon->carbonate)
  Downloading https://files.pythonhosted.org/packages/c6/a0/7c18f4a3805af57e2be19a6502c1946f5dac51c8b1ed19368968a46a2f07/txAMQP-0.8.2.tar.gz
Requirement already satisfied: cachetools in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from carbon->carbonate) (2.1.0)
Requirement already satisfied: urllib3 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from carbon->carbonate) (1.23)
Requirement already satisfied: six in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from whisper->carbonate) (1.11.0)
Requirement already satisfied: incremental>=16.10.1 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from Twisted->carbon->carbonate) (17.5.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from Twisted->carbon->carbonate) (17.3.1)
Requirement already satisfied: zope.interface>=4.0.2 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from Twisted->carbon->carbonate) (4.4.3)
Requirement already satisfied: Automat>=0.3.0 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from Twisted->carbon->carbonate) (0.6.0)
Requirement already satisfied: constantly>=15.1 in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from Twisted->carbon->carbonate) (15.1.0)
Requirement already satisfied: setuptools in c:\users\jmorales\appdata\roaming\python\python35\site-packages (from zope.interface>=4.0.2->Twisted->carbon->carbonate) (39.1.0)
Requirement already satisfied: attrs in c:\users\jmorales\appdata\roaming\python\python35\site-packages (from Automat>=0.3.0->Twisted->carbon->carbonate) (18.1.0)
Building wheels for collected packages: carbonate, carbon, whisper, txAMQP
  Running setup.py bdist_wheel for carbonate: started
  Running setup.py bdist_wheel for carbonate: finished with status 'error'
  Complete output from command c:\users\jmorales\appdata\local\programs\python\python35\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\jmorales\\AppData\\Local\\Temp\\pip-install-_s11eqf_\\carbonate\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\jmorales\AppData\Local\Temp\pip-wheel-uza8jgt2 --python-tag cp35:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\carbonate
  copying carbonate\aggregation.py -> build\lib\carbonate
  copying carbonate\cli.py -> build\lib\carbonate
  copying carbonate\cluster.py -> build\lib\carbonate
  copying carbonate\config.py -> build\lib\carbonate
  copying carbonate\fill.py -> build\lib\carbonate
  copying carbonate\list.py -> build\lib\carbonate
  copying carbonate\lookup.py -> build\lib\carbonate
  copying carbonate\sieve.py -> build\lib\carbonate
  copying carbonate\stale.py -> build\lib\carbonate
  copying carbonate\sync.py -> build\lib\carbonate
  copying carbonate\util.py -> build\lib\carbonate
  copying carbonate\__init__.py -> build\lib\carbonate
  running egg_info
  writing requirements to carbonate.egg-info\requires.txt
  writing entry points to carbonate.egg-info\entry_points.txt
  writing top-level names to carbonate.egg-info\top_level.txt
  writing carbonate.egg-info\PKG-INFO
  writing dependency_links to carbonate.egg-info\dependency_links.txt
  reading manifest file 'carbonate.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'carbonate.egg-info\SOURCES.txt'
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  creating build\bdist.win-amd64
  creating build\bdist.win-amd64\wheel
  creating build\bdist.win-amd64\wheel\carbonate
  copying build\lib\carbonate\aggregation.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\cli.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\cluster.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\config.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\fill.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\list.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\lookup.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\sieve.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\stale.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\sync.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\util.py -> build\bdist.win-amd64\wheel\.\carbonate
  copying build\lib\carbonate\__init__.py -> build\bdist.win-amd64\wheel\.\carbonate
  running install_egg_info
  Copying carbonate.egg-info to build\bdist.win-amd64\wheel\.\carbonate-1.1.3-py3.5.egg-info
  running install_scripts
  error: [Errno 2] No such file or directory: 'LICENSE'

  ----------------------------------------
  Failed building wheel for carbonate
  Running setup.py clean for carbonate
  Running setup.py bdist_wheel for carbon: started
  Running setup.py bdist_wheel for carbon: finished with status 'error'
  Complete output from command c:\users\jmorales\appdata\local\programs\python\python35\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\jmorales\\AppData\\Local\\Temp\\pip-install-_s11eqf_\\carbon\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\jmorales\AppData\Local\Temp\pip-wheel-rnig1yjd --python-tag cp35:
  ################################################################################

  Carbon's default installation prefix is "/opt/graphite".

  To install Carbon in the Python's default location run:
  $ GRAPHITE_NO_PREFIX=True python setup.py install

  ################################################################################
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\carbon
  copying lib\carbon\amqp_listener.py -> build\lib\carbon
  copying lib\carbon\amqp_publisher.py -> build\lib\carbon
  copying lib\carbon\cache.py -> build\lib\carbon
  copying lib\carbon\carbon_pb2.py -> build\lib\carbon
  copying lib\carbon\client.py -> build\lib\carbon
  copying lib\carbon\conf.py -> build\lib\carbon
  copying lib\carbon\database.py -> build\lib\carbon
  copying lib\carbon\events.py -> build\lib\carbon
  copying lib\carbon\exceptions.py -> build\lib\carbon
  copying lib\carbon\hashing.py -> build\lib\carbon
  copying lib\carbon\http.py -> build\lib\carbon
  copying lib\carbon\instrumentation.py -> build\lib\carbon
  copying lib\carbon\log.py -> build\lib\carbon
  copying lib\carbon\management.py -> build\lib\carbon
  copying lib\carbon\manhole.py -> build\lib\carbon
  copying lib\carbon\pipeline.py -> build\lib\carbon
  copying lib\carbon\protobuf.py -> build\lib\carbon
  copying lib\carbon\protocols.py -> build\lib\carbon
  copying lib\carbon\regexlist.py -> build\lib\carbon
  copying lib\carbon\relayrules.py -> build\lib\carbon
  copying lib\carbon\rewrite.py -> build\lib\carbon
  copying lib\carbon\routers.py -> build\lib\carbon
  copying lib\carbon\service.py -> build\lib\carbon
  copying lib\carbon\state.py -> build\lib\carbon
  copying lib\carbon\storage.py -> build\lib\carbon
  copying lib\carbon\util.py -> build\lib\carbon
  copying lib\carbon\writer.py -> build\lib\carbon
  copying lib\carbon\__init__.py -> build\lib\carbon
  creating build\lib\carbon\aggregator
  copying lib\carbon\aggregator\buffers.py -> build\lib\carbon\aggregator
  copying lib\carbon\aggregator\processor.py -> build\lib\carbon\aggregator
  copying lib\carbon\aggregator\rules.py -> build\lib\carbon\aggregator
  copying lib\carbon\aggregator\__init__.py -> build\lib\carbon\aggregator
  package init file 'lib\twisted\plugins\__init__.py' not found (or not a regular file)
  creating build\lib\twisted
  creating build\lib\twisted\plugins
  copying lib\twisted\plugins\carbon_aggregator_cache_plugin.py -> build\lib\twisted\plugins
  copying lib\twisted\plugins\carbon_aggregator_plugin.py -> build\lib\twisted\plugins
  copying lib\twisted\plugins\carbon_cache_plugin.py -> build\lib\twisted\plugins
  copying lib\twisted\plugins\carbon_relay_plugin.py -> build\lib\twisted\plugins
  copying lib\carbon\amqp0-8.xml -> build\lib\carbon
  running build_scripts
  creating build\scripts-3.5
  copying and adjusting bin\carbon-aggregator-cache.py -> build\scripts-3.5
  copying and adjusting bin\carbon-aggregator.py -> build\scripts-3.5
  copying and adjusting bin\carbon-cache.py -> build\scripts-3.5
  copying and adjusting bin\carbon-client.py -> build\scripts-3.5
  copying and adjusting bin\carbon-relay.py -> build\scripts-3.5
  copying and adjusting bin\validate-storage-schemas.py -> build\scripts-3.5
  installing to build\bdist.win-amd64\wheel
  running install
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\jmorales\AppData\Local\Temp\pip-install-_s11eqf_\carbon\setup.py", line 104, in <module>
      **setup_kwargs
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "C:\Users\jmorales\AppData\Roaming\Python\Python35\site-packages\wheel\bdist_wheel.py", line 238, in run
      self.run_command('install')
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\dist.py", line 973, in run_command
      cmd_obj.ensure_finalized()
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\cmd.py", line 107, in ensure_finalized
      self.finalize_options()
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages\setuptools\command\install.py", line 38, in finalize_options
      orig.install.finalize_options(self)
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\command\install.py", line 345, in finalize_options
      'userbase', 'usersite')
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\command\install.py", line 485, in convert_paths
      setattr(self, attr, convert_path(getattr(self, attr)))
    File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\util.py", line 125, in convert_path
      raise ValueError("path '%s' cannot be absolute" % pathname)
  ValueError: path '/opt/graphite/lib' cannot be absolute

  ----------------------------------------
  Failed building wheel for carbon
  Running setup.py clean for carbon
  Running setup.py bdist_wheel for whisper: started
  Running setup.py bdist_wheel for whisper: finished with status 'done'
  Stored in directory: C:\Users\jmorales\AppData\Local\pip\Cache\wheels\e7\1b\50\d605fd548e9215eec6f706835be2b6cf02c20235786f39536b
  Running setup.py bdist_wheel for txAMQP: started
  Running setup.py bdist_wheel for txAMQP: finished with status 'done'
  Stored in directory: C:\Users\jmorales\AppData\Local\pip\Cache\wheels\52\44\f0\7d2102660281d25008c0c3eb29efd41d6701d33d2f5bd86971
Successfully built whisper txAMQP
Failed to build carbonate carbon
Installing collected packages: txAMQP, carbon, whisper, carbonate
  Running setup.py install for carbon: started
    Running setup.py install for carbon: finished with status 'error'
    Complete output from command c:\users\jmorales\appdata\local\programs\python\python35\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\jmorales\\AppData\\Local\\Temp\\pip-install-_s11eqf_\\carbon\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\jmorales\AppData\Local\Temp\pip-record-5gsegao2\install-record.txt --single-version-externally-managed --compile:
    ################################################################################

    Carbon's default installation prefix is "/opt/graphite".

    To install Carbon in the Python's default location run:
    $ GRAPHITE_NO_PREFIX=True python setup.py install

    ################################################################################
    running install
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\jmorales\AppData\Local\Temp\pip-install-_s11eqf_\carbon\setup.py", line 104, in <module>
        **setup_kwargs
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\dist.py", line 973, in run_command
        cmd_obj.ensure_finalized()
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages\setuptools\command\install.py", line 38, in finalize_options
        orig.install.finalize_options(self)
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\command\install.py", line 345, in finalize_options
        'userbase', 'usersite')
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\command\install.py", line 485, in convert_paths
        setattr(self, attr, convert_path(getattr(self, attr)))
      File "c:\users\jmorales\appdata\local\programs\python\python35\lib\distutils\util.py", line 125, in convert_path
        raise ValueError("path '%s' cannot be absolute" % pathname)
    ValueError: path '/opt/graphite/lib' cannot be absolute

    ----------------------------------------
Command "c:\users\jmorales\appdata\local\programs\python\python35\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\jmorales\\AppData\\Local\\Temp\\pip-install-_s11eqf_\\carbon\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\jmorales\AppData\Local\Temp\pip-record-5gsegao2\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\jmorales\AppData\Local\Temp\pip-install-_s11eqf_\carbon\

This is my setup:

Python version: 3.5
OS: Windows 10 Home
System Type: 64-bit Operating System, x64-based processor
Processor: AMD A10-8700P Radeon R6

New release

Hi!
The latest carbonate release is from 2014, are there plans to release the last commits ? I'm particularly interested in --lock.

Thanks!

struct.error: unpack requires a string argument of length 3006984

I've got an graphite host which has a lot of metrics (according to carbon-list 45762) this is standard graphite install single carbon-cache. I want to introduce a new server with multiple carbon-caches, etc. Before trying carbon-sync I had manually rsync-ed the whisper directory to the new server.

Then I ran (from graphite-vm-new)

ssh graphite-vm-old -- carbon-list|carbon-sync -s graphite-vm-old

Then I got a couple of rsync: link_stat errors but the process seemed to work so I left it running until I got the following

* Running batch 14001-15000
  - Rsyncing metrics
rsync: link_stat "/opt/graphite/storage/whisper/stats_counts/mobile_redirect/kept/detail/undefined/desktop/wsp.wsp" failed: No such file or directory (2)
rsync: link_stat "/opt/graphite/storage/whisper/stats_counts/mobile_redirect/kept/result/undefined/desktop/wsp.wsp" failed: No such file or directory (2)
rsync: link_stat "/opt/graphite/storage/whisper/stats_counts/mobile_redirect/kept/start/undefined/desktop/wsp.wsp" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [generator=3.0.9]
  - Syncing 1 of 1000 metrics. Avg: 0.100000s  Time Left: n/as (0%)

It seemed to work right up until

  - Syncing 30 of 1000 metrics. Avg: 0.026024s  Time Left: 0:00:25.268973s (3%)
Traceback (most recent call last):
  File "/usr/local/bin/carbon-sync", line 9, in <module>
    load_entry_point('carbonate==0.2.1', 'console_scripts', 'carbon-sync')()
  File "/usr/local/lib/python2.7/dist-packages/carbonate/cli.py", line 180, in carbon_sync
    run_batch(metrics_to_sync, remote, args.storage_dir)
  File "/usr/local/lib/python2.7/dist-packages/carbonate/sync.py", line 116, in run_batch
    merge_elapsed = sync_batch(metrics_to_heal)
  File "/usr/local/lib/python2.7/dist-packages/carbonate/sync.py", line 57, in sync_batch
    heal_metric(staging, local)
  File "/usr/local/lib/python2.7/dist-packages/carbonate/sync.py", line 72, in heal_metric
    fill_archives(source, dest, time())
  File "/usr/local/lib/python2.7/dist-packages/carbonate/fill.py", line 110, in fill_archives
    fill(src, dst, gapstart - step, start)
  File "/usr/local/lib/python2.7/dist-packages/carbonate/fill.py", line 72, in fill 
    (timeInfo, values) = fetch(src, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 711, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 786, in file_fetch 
    unpackedSeries = struct.unpack(seriesFormat, seriesString)
struct.error: unpack requires a string argument of length 3006984

I have no idea what could be the problem. Any suggestions? I will try to troubleshoot as keep updated if I find something.

call to ConsistentHashRing with wrong args

carbonate 1.0.2, carbon - master

# carbon-lookup abc
Traceback (most recent call last):
  File "/usr/local/bin/carbon-lookup", line 10, in <module>
    load_entry_point('carbonate==1.0.2', 'console_scripts', 'carbon-lookup')()
  File "/usr/local/lib/python2.7/dist-packages/carbonate/cli.py", line 85, in carbon_lookup
    cluster = Cluster(config, args.cluster)
  File "/usr/local/lib/python2.7/dist-packages/carbonate/cluster.py", line 36, in __init__
    r.ring = ConsistentHashRing(hash_type=config.hashing_type(cluster))
TypeError: __init__() takes at least 2 arguments (2 given)

The problem is in lines (carbonate/cluster.py)

34         args = inspect.getargspec(ConsistentHashRing.__init__).args
35         if 'hash_type' in args:
36             r.ring = ConsistentHashRing(hash_type=config.hashing_type(cluster))

while in master carbon (/opt/graphite/lib/carbon/hashing.py):

26 class ConsistentHashRing:
27   def __init__(self, nodes, replica_count=100, hash_type='carbon_ch'):
28     self.ring = []

As workaround I commented out lines 35,36 in cluster.py
It it problem in my setup or in carbonite?

Thanks!

Carbon-sieve errors out when results from carbon-list are piped to it.

While trying to measure time to complete a rebalance in preparation of scaling out our cluster using the example provided. I'm getting an error from carbon-sieve:

carbon-list -c /opt/graphite/conf/carbonate.conf -C main -d /opt/graphite/whisper/ | carbon-sieve -c /opt/graphite/conf/carbonate.conf -C fnv -n 172.26.2.28 | carbon-sync -s 172.26.2.28

Traceback (most recent call last):
File "/usr/local/bin/carbon-sieve", line 10, in
load_entry_point('carbonate==1.0.2', 'console_scripts', 'carbon-sieve')()
File "/usr/local/lib/python2.7/dist-packages/carbonate/cli.py", line 119, in carbon_sieve
cluster = Cluster(config, args.cluster)
File "/usr/local/lib/python2.7/dist-packages/carbonate/cluster.py", line 36, in init
r.ring = ConsistentHashRing(hash_type=config.hashing_type(cluster))
TypeError: init() takes at least 2 arguments (2 given)

~ # pip show carbonate

Name: carbonate
Version: 1.0.2
Location: /usr/local/lib/python2.7/dist-packages
Requires: carbon, whisper

If I'm doing something wrong, please let me know.

Tests for Python 2.7 producing error

See https://travis-ci.org/github/graphite-project/carbonate/jobs/663101276

Looks like because pypa/setuptools#1136

how does carbonate interact with multiple caches ?

Hi,

I'm trying to figure out how carbonate works with multiple carbon caches running on the same host. It clearly has an impact when using e.g. carbon-lookup.

Am I correct one actually has to use the same DESTINATION in carbonate.conf as in carbon.conf?

E.g. with this in the config:

DESTINATIONS = graphite1.mynode.com:2104:1, graphite1.mynode.com:2204:2, graphite1.mynode.com:2304:3, graphite2.mynode.com:2104:1, graphite2.mynode.com:2204:2, graphite2.mynode.com:2304:3, graphite3.mynode.com:2104:1, graphite3.mynode.com:2204:2, graphite3.mynode.com:2304:3, graphite4.mynode.com:2104:1, graphite4.mynode.com:2204:2, graphite4.mynode.com:2304:3

On the other hand this doesn't seem to be right:

root@graphite1:~/scripts# carbon-hosts -c /etc/carbon/carbonate.conf 
graphite1.mynode.com
graphite1.mynode.com
graphite1.mynode.com
graphite2.mynode.com
graphite2.mynode.com
graphite2.mynode.com
graphite3.mynode.com
graphite3.mynode.com
graphite3.mynode.com
graphite4.mynode.com
graphite4.mynode.com
graphite4.mynode.com

Is carbonate still considered beta quality?

While working on packaging carbonate for Gentoo, one of the developers expressed concerns about the following remark in License and Warnings section of the README:

These tools should be considered beta quality right now

So to decide how to move forward with packaging, I'd like to ask if carbonate is still considered as beta quality software?

push out new release?

Hi, could you push out a new release to https://pypi.python.org/pypi/carbonate/ ?
So I can easily update so I can use python-path. In the current version it is not available...

Thanks

struct.error: unpack requires a string argument of length 12

I am trying to sync whisper file from my another node from cluster.
I am getting below error. I have checked for corrupted whisper file with whisper-info but result was Zero
Below is the Traceback :

Traceback (most recent call last):
File "/usr/bin/carbon-sync", line 10, in
load_entry_point('carbonate==0.2.2', 'console_scripts', 'carbon-sync')()
File "/usr/lib/python2.7/dist-packages/carbonate/cli.py", line 197, in carbon_sync
args.storage_dir, args.rsync_options)
File "/usr/lib/python2.7/dist-packages/carbonate/sync.py", line 116, in run_batch
merge_elapsed = sync_batch(metrics_to_heal)
File "/usr/lib/python2.7/dist-packages/carbonate/sync.py", line 57, in sync_batch
heal_metric(staging, local)
File "/usr/lib/python2.7/dist-packages/carbonate/sync.py", line 72, in heal_metric
fill_archives(source, dest, time())
File "/usr/lib/python2.7/dist-packages/carbonate/fill.py", line 110, in fill_archives
fill(src, dst, gapstart - step, start)
File "/usr/lib/python2.7/dist-packages/carbonate/fill.py", line 72, in fill
(timeInfo, values) = fetch(src, fromTime, untilTime)
File "/usr/lib/python2.7/dist-packages/whisper.py", line 711, in fetch
return file_fetch(fh, fromTime, untilTime)
File "/usr/lib/python2.7/dist-packages/whisper.py", line 751, in file_fetch
(baseInterval,baseValue) = struct.unpack(pointFormat,packedPoint)
struct.error: unpack requires a string argument of length 12

Someone please help me

carbon-sync doesn't preserve file sparseness during copy

It seems shutil.copyfile doesn't support sparse files. Sort of annoying.

flock

Great effort, whisper can be a pain.

I have always been led to believe that "Graphite will lock the Whisper file using flock() during updates so that no other Graphite process reads it mid-update and finds inconsistent data.

Unfortunately, flock() is only an advisory lock and rsync ignores it. This means that if Graphite updates a particular Whisper file while rsync is in the middle of backing it up, the backup copy might be inconsistent." from the boys SwiftStack - @smerritt

Having looked through the code I do not see any reference to flock in anyway. Is this just trivial because of the context of healing to local whisper file? If so, adding an ssh key option pull request is in order for me I think.

preserving timestamps with sync

I want transferred whisper files to preserve the timestamp of the donor. By default, the files on the recipient node have their mtimes updated to the current time. Options that I would I expect to alter this behavior appear to do nothing.

My command looks like this:

ssh $OTHER_HOST -- 'carbon-list' | carbon-sync -C $CLUSTER "--rsync-options="-c -s $OTHER_HOST

I have also tried that command with the -a and -t flags for rsync-options instead of -c.

support other consistent hashing algorithms ?

hi,
as pointed out in the README ports and "labels" are not taken into consideration when running the consistent hashing algo (as #38 found out too)
I think it'd make sense to support other hashing implementations, e.g. fnv1a would make carbonate compatible with https://github.com/grobian/carbon-c-relay which in turn would allow hashing ip+port, what do you think?

Connections issue(?) - how to get some logging?

Hi there,

Carbonate 0.2.2 is installed with Python 2.6.6 on CentOS 6.6 with running Graphite carbon and Whisper.
Graphite is in cluster.
The config is:

[root@graphitecarbon-172 ~]# cat /opt/graphite/conf/carbonate.conf
# !!! generated by puppet !!!

[main]
DESTINATIONS = 172.31.41.121:2004:carbon01, 172.31.41.221:2004:carbon02, 172.31.41.31:2004:carbon03, 172.31.41.81:2004:carbon04, 172.31.41.161:2004:carbon05, 172.31.41.211:2004:carbon06
REPLICATION_FACTOR = 1
SSH_USER = ec2-user

ec2-user credentials added.
Local whisper-fill works just perfect.

But when I run:
carbon-sync -s 172.31.44.222 -d /data/statsite/counts/numStats.wsp --source-storage-dir /data/statsite/counts/numStats.wsp the command is just waiting without any output...

How to check what is going on? How to turn some debug log?

whisper.InvalidTimeInterval from carbon-sync

Hi, we're trying to use carbon-sync, and are running into whisper.InvalidTimeInterval every time. It doesn't seem to be an issue with specific whisper files, because it happens at a different point in the sync every time, and seems to happen regardless of batch size. Here's the traceback:

Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-sync", line 9, in <module>
    load_entry_point('carbonate==0.2.2', 'console_scripts', 'carbon-sync')()
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/cli.py", line 197, in carbon_sync
    args.storage_dir, args.rsync_options)
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/sync.py", line 116, in run_batch
    merge_elapsed = sync_batch(metrics_to_heal)
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/sync.py", line 57, in sync_batch
    heal_metric(staging, local)
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/sync.py", line 72, in heal_metric
    fill_archives(source, dest, time())
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/fill.py", line 110, in fill_archives
    fill(src, dst, gapstart - step, start)
  File "/opt/graphite/lib/carbonate-0.2.2-py2.6.egg/carbonate/fill.py", line 72, in fill
    (timeInfo, values) = fetch(src, fromTime, untilTime)
  File "/opt/graphite/lib/whisper.py", line 667, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/opt/graphite/lib/whisper.py", line 683, in file_fetch
    raise InvalidTimeInterval("Invalid time interval")
whisper.InvalidTimeInterval: Invalid time interval

Update docs to explain --dirty option for carbon-sync

/cc #39 @kamaradclimber

Missing newline in batch files causes rsync errors

    * Running batch 2401-2700
      - Rsyncing metrics: rsync -azpS --copy-dest="/u2/carbon" --files-from /tmp/tmp1heicu_k [email protected]:/u2/carbon/ /tmp/1
    0.20.36.96dz00hbdq/
    ERROR: rejecting unrequested file-list name: whisper/diamond/xxxxxxx/xxxxxx/xxxxxx/memory/SwapFree.wsp
    rsync error: protocol incompatibility (code 2) at flist.c(911) [Receiver=3.1.2]
    WARNING:root:Failed to sync from [email protected]:/u2/carbon/! rsync rc=2

If I echo >> /tmp/tmp1heicu_k .failed and retry that rsync command, it's successful.

This is fixed in #130

graphite-project / carbonate Goto Github PK

carbonate's People

Contributors

Stargazers

Watchers

Forkers

carbonate's Issues

Current config is:

Recommend Projects

Recommend Topics

Recommend Org