Code Monkey home page Code Monkey logo

btrdb-server's Introduction

BTrDB golang bindings

GoDoc

These are the go BTrDB bindings. This branch is compatible with version 5, a very recent rewrite of BTrDB. Note that it is likely you should be using the version 4 bindings of BTrDB as v5 is not widely released.

You can read the API documentation and code examples by clicking the godoc button above. To import this package in your code, add

import "gopkg.in/BTrDB/btrdb.v5"

For more information about installing BTrDB, look at https://docs.smartgrid.store

btrdb-server's People

Contributors

emptyflash avatar filipjanitor avatar gtfierro avatar immesys avatar samkumar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

btrdb-server's Issues

Installation

I have a kubernetes cluster with ceph as well. Are there any docs to getting btrdb setup?

Also what is the state of the project? Is it alpha / beta / or pretty stable?

Thanks!

block cache breaks GOGC

True fact: a block cache of 32GB means that go will NEVER GC until the heap is bigger than 64GB. Tweak GOGC to fix.

flush open trees under starvation

When the open_trees pool is starved, a scavenger should pick the best tree and flush it to open up a slot. At the moment the client must wait until a tree times out.

GPL vs Apache vs BSD-*

I'm a bit surprised to see this is under the GPL. I expected it would be under a BSD or Apache license, especially given the funding source. I wouldn't say it's a deal-breaker but it does give me pause when even considering this database because future use will be architecturally limited (e.g., can't go fully embedded or rework it as an in-memory storage library) and it will complicate distribution of other (more permissively licensed) software. That's not a great database feature. Especially given the integration with Apache Spark and DoE funding, I would have expected Apache license more appropriate.

server crash on restart

I had to restart a server pod, which did not work.

The process did not shut down. So i had to kill it.

When trying to start it again, the server crashes during journal checking after emitting "panic: (404: stream does not exist)"

I have put the ceph pools aside so i could provide them for further analysis.

logs-from-btrdb-in-btrdb-0 .txt

Does BtrDB support non-scalar data?

Love the idea behind btrdb. I get thousands of ticks per second of financial data, but I often only want to get the aggregates out.

Problem is I also want to use this for computed data. So for example, as time series come in, I create yield curves. These yield curve structures require two numeric vectors to describe. Is it possible to store data that is more than scalar, in BtrDB, and in a related question, can I pass my own aggregation functions in (since in this vector structure case, mean, min, max etc will require custom functions)?

Error: Strange datablock type

I am using BTrDb with Mr.Plotter. Everything was working fine when, one day I accessed Mr.Plotter and suddenly btrdb stopped working. I am unable to send data to the database, as whenever it receives it gives following error and breaks down:

panic: Strange datablock type

goroutine 15 [running]:
github.com/op/go-logging.(*Logger).Panic(0xc20803a900, 0xc2080b69b0, 0x1, 0x1)
        /home/ubuntu/raqmn/go/src/github.com/op/go-logging/logger.go:188 +0xc0
github.com/SoftwareDefinedBuildings/btrdb/internal/bstore.(*BlockStore).ReadDatablock(0xc2080841b0, 0xc20c93e048, 0x10, 0x5b8, 0x10000000000367, 0x3, 0xc2080e9938, 0xf000000000000000, 0x0, 0x0)
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/internal/bstore/blockstore.go:331 +0x6a6
github.com/SoftwareDefinedBuildings/btrdb/qtree.(*QTree).LoadNode(0xc2080e9950, 0x10000000000367, 0x3, 0x538, 0xf000000000000000, 0xc2080e9920, 0x0, 0x0)
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/qtree/qtree_utils.go:120 +0xc1
github.com/SoftwareDefinedBuildings/btrdb/qtree.NewReadQTree(0xc2080841b0, 0xc20c93e048, 0x10, 0x5b8, 0xffffffffffffffff, 0x20, 0x0, 0x0)
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/qtree/qtree_utils.go:187 +0x185
github.com/SoftwareDefinedBuildings/btrdb.(*Quasar).QueryValuesStream(0xc2080bc000, 0xc20c93e048, 0x10, 0x5b8, 0x1470cd147b97c60d, 0x1470d05aac51fa9e, 0xffffffffffffffff, 0x0, 0x20, 0x0)
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/quasar.go:194 +0x5d
github.com/SoftwareDefinedBuildings/btrdb/cpinterface.func·010()
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/cpinterface/cpinterface.go:94 +0x402
created by github.com/SoftwareDefinedBuildings/btrdb/cpinterface.(*CPInterface).dispatchCommands
        /home/ubuntu/raqmn/go/src/github.com/SoftwareDefinedBuildings/btrdb/cpinterface/cpinterface.go:441 +0x58e

There was no change in anything and I am unable to debug it. I can read from BTrDb and Plotter works fine until I send any data.

Thank You.

Aligned Window Query Start/End Inclusivity is Inconsistent

When BTrDB receives an aligned windows query, it rounds the start/end timestamps down to the nearest point boundary.

If pw < 32, it returns all points from start (inclusive) to end (exclusive), after rounding down.
If pw >= 32, it returns all points from start (inclusive) to end (inclusive), after rounding down.

I believe this is a bug. This causes rendering artifacts in the WaveViewer plotter.

The following Python3 script reproduces this issue (I've emailed this to Michael). All timestamps used are point-aligned, so the rounding down behavior isn't tested.

#!/usr/bin/env python3
import btrdb4
import time
import uuid

uu = uuid.UUID("80741975-8247-4d12-9b73-becf9eb8ff31")

conn = btrdb4.Connection("compound-3.cs.berkeley.edu:4410")
b = conn.newContext()

print("Connected to compound-3.cs.berkeley.edu:4410")

s = b.streamFromUUID(uu)
assert(s.exists())



print("All of the endpoints of these queries are pointwidth-aligned.")



print()
print("Making a query at pw=31 from 1458040387951132672 to 1458040396541067264")
for statpoint, version in s.alignedWindows(1458040387951132672, 1458040396541067264, 31):
    print(statpoint)

print()
print("Making a query at pw=31 from 1458040387951132672 to 1458040398688550912")
for statpoint, version in s.alignedWindows(1458040387951132672, 1458040398688550912, 31):
    print(statpoint)

print()
print("The above two queries show that BTrDB is treating the endpoint as exclusive.")
print("The point at 1458040396541067264 is not in the result of the first query, but such a point does exist with count != 0.")



print()
print("Making a query at pw=31 from 1458040387951132672 to 1458040396541067264")
for statpoint, version in s.alignedWindows(1458040387951132672, 1458040396541067264, 31):
    print(statpoint)

print()
print("Making a query at pw=32 from 1458040387951132672 to 1458040396541067264")
for statpoint, version in s.alignedWindows(1458040387951132672, 1458040396541067264, 32):
    print(statpoint)

print()
print("The last query shows that BTrDB is treating the endpoint as inclusive.")
print("The point at 1458040396541067264 is in the result of the query.")



print()
print("In conclusion, it seems that at pw=32, BTrDB treats the endpoint as inclusive, but at pw=31, BTrDB treats the endpoint as exclusive.")

Check times earlier

Check times/values on insert and give error early rather than panicing later

Could not read allocator for cold pool! Has the DB been created properly?

I try to run btrdb in Docker with image btrdb/db:4.6.0 but got these error:

db1_1  | [CRITICAL]cephprovider.go:489 > Could not read allocator for cold pool! Has the DB been created properly?
db1_1  | panic: Could not read allocator for cold pool! Has the DB been created properly?
db1_1  |
db1_1  | goroutine 1 [running]:
db1_1  | github.com/op/go-logging.(*Logger).Panic(0xc420149260, 0xc420011920, 0x1, 0x1)
db1_1  | 	/home/immesys/w/go/src/github.com/op/go-logging/logger.go:188 +0xc7
db1_1  | github.com/SoftwareDefinedBuildings/btrdb/internal/cephprovider.(*CephStorageProvider).Initialize(0xc420083a20, 0xf610e0, 0xc420196270)
db1_1  | 	/home/immesys/w/go/src/github.com/SoftwareDefinedBuildings/btrdb/internal/cephprovider/cephprovider.go:489 +0x8f4
db1_1  | github.com/SoftwareDefinedBuildings/btrdb/internal/bstore.NewBlockStore(0xf610e0, 0xc420196270, 0x80, 0xd, 0x0)
db1_1  | 	/home/immesys/w/go/src/github.com/SoftwareDefinedBuildings/btrdb/internal/bstore/blockstore.go:133 +0x303
db1_1  | github.com/SoftwareDefinedBuildings/btrdb.NewQuasar(0xf610e0, 0xc420196270, 0x1, 0xd, 0x0)
db1_1  | 	/home/immesys/w/go/src/github.com/SoftwareDefinedBuildings/btrdb/quasar.go:116 +0x4d
db1_1  | main.main()
db1_1  | 	/home/immesys/w/go/src/github.com/SoftwareDefinedBuildings/btrdb/btrdbd/main.go:113 +0x312

My Ceph pool list:

ceph osd dump | grep pool
pool 0 'rbd' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 2 flags hashpspool stripe_width 0
pool 1 'cephfs_data' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 10 flags hashpspool crash_replay_interval 45 stripe_width 0
pool 2 'cephfs_metadata' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 9 flags hashpspool stripe_width 0
pool 3 '.rgw.root' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 11 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 4 'default.rgw.control' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 13 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 5 'default.rgw.data.root' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 6 'default.rgw.gc' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 16 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 7 'default.rgw.lc' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 8 'default.rgw.log' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 18 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 11 'btrdb' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 27 flags hashpspool stripe_width 0

I use btrdb pool.

And the status of Ceph cluster:

ceph -s
    cluster 136dff26-fe9d-4513-b5f3-7ba1b290a609
     health HEALTH_OK
     monmap e2: 1 mons at {ccb1.circeboard.com=10.174.7.90:6789/0}
            election epoch 7, quorum 0 ccb1.circeboard.com
      fsmap e5: 1/1/1 up {0=mds-ccb1.circeboard.com=up:active}
        mgr no daemons active
     osdmap e28: 1 osds: 1 up, 1 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v148: 256 pgs, 10 pools, 3829 bytes data, 224 objects
            40092 kB used, 102209 MB / 102249 MB avail
                 256 active+clean

Any one know what should i do?

Changes in V5

Hi, I noticed that development of version 5 is in progress now. Could you please shed some light on what to expect from the new version? Are there any major feature additions, architecture or API changes planned?
Thanks!

psl.pqube.DEVICEID

The current implementation expects that the device will always be defined as psl.pqube.DEVICEID which is not optimal considering device may not be a pqube etc.

In manifest:
add psl.pqube.p300001 path=myDevice

I like the idea of the dot notation for the name for logical groupings or as a label.

Possible bug (or API suggestion) for nearest

I was noticing some weird results with the bracket queries in Mr. Plotter, so I investigated. This led me to make the following queries against BTrDB in a certain production cluster (details communicated to @immesys offline):

root@btrdb-0:/sam# ./btrdb-cmd-static localhost:4410
localhost:4410> windows bdff2d37-9f9d-553a-8f2d-e8240e48c622
-1152921504606846976,1.23079e-05,192.655,264.361,1635436680
Version: 0
localhost:4410> nearest bdff2d37-9f9d-553a-8f2d-e8240e48c622
1507507200000000000,158.534
Version: 129146
localhost:4410> windows 3c122dcf-c0ff-5da0-bf4b-31b4309ef446
Version: 10
localhost:4410> raw
Usage: raw UUID [start] [end] [version]
localhost:4410> raw 3c122dcf-c0ff-5da0-bf4b-31b4309ef446 -1152921504606846975 3458764513820540927 0
Version: 10
localhost:4410> nearest 3c122dcf-c0ff-5da0-bf4b-31b4309ef446
0,0
Version: 10
localhost:4410> nearest
Usage: nearest UUID [backward] [timestamp] [version]
localhost:4410> nearest 3c122dcf-c0ff-5da0-bf4b-31b4309ef446 false -100 0  
0,0
Version: 10
localhost:4410> nearest 3c122dcf-c0ff-5da0-bf4b-31b4309ef446 false 100 0
0,0
Version: 10

The stream bdff2d37-9f9d-553a-8f2d-e8240e48c622 is a stream with data, and things work as expected.

The stream 3c122dcf-c0ff-5da0-bf4b-31b4309ef446 exists but has no data. For this stream, the nearest queries always return the point (time = 0, value = 0) even if it's an invalid answer for the query at hand.

The same behavior actually happens if there is no valid "nearest" point (e.g., no points after that timestamp)

localhost:4410> nearest bdff2d37-9f9d-553a-8f2d-e8240e48c622 false 3458764513820540926
0,0
Version: 129148

This is bad because if I make a nearest query at a negative timestamp, I can't tell from the answer (0, 0) whether there was actually no data after that time, or whether the point is at (0, 0). I suggest that you revise the API by returning an error, rather than returning (0, 0). Alternatively, you could choose to return a point outside of the valid range to signal an error, or add a "no such point" flag to the returned protobuf.

BTrDB use case for irregular interval time series

We are considering migrating time series data from mongoDB to BTrDB and trying to understand whether the peculiarities of our data will be supported.

The time series data comes on irregular intervals between 30 seconds up to a few minutes. Additionally, each key is mapped to a vector value within each time series.

Spontaneous down+out

After a runtime of ~43 days, BTrDB spontaneously goes out

State changed: 
CLUSTER STATE e376
  LEADER: -- @0
  MASH: 0 (0 members, 100.00% unmapped)
  STATE: Degraded
  NODENAME  STATUS       MASH   WEIGHT RPREF
  btrdb-0   down+out     0      100    1.00 
ACTIVE MASH

PROPOSED MASH
[e38c8109]      btrdb-0 : 0 - 4294967296 (100)

$$ btrdb-0 allcurrent is false
$$ btrdb-0 proposedMash is 2, our target mash is 2
$$ btrdb-0 we do not want to advance our mash

How to install btrdb?

I am sorry to ask a so naive question, I am a beginner of golang, and after I install the go 1.12 in my ubuntu , I just use command “make” to make this btrdb project, but i receive a error message:

hadoop@ubuntu:~/Desktop/btrdb-server-master$ sudo make
mkdir -p /home/hadoop/Desktop/btrdb-server-master/build/gopath/bin
go get -u github.com/golang/dep/cmd/dep
cd /home/hadoop/Desktop/btrdb-server-master/build/gopath/src/github.com/BTrDB/btrdb-server &&  dep ensure
/bin/sh: 1: dep: not found
Makefile:23: recipe for target '/home/hadoop/Desktop/btrdb-server-master/build/gopath/src/github.com/BTrDB/btrdb-server/vendor' failed
make: *** [/home/hadoop/Desktop/btrdb-server-master/build/gopath/src/github.com/BTrDB/btrdb-server/vendor] Error 127

and i do not know why this happen, if I want to use btrdb and get intuitive grasp,how should I do? please help me ,thank you

BTrDB API service crash

This isn't a fully baked issue yet since I'm having trouble reproducing the behavior, however, I thought I'd log it in case I can refine the information later.

I seem to have been able to take down our BTrDB API service while sending curl requests to our deployed copy of Mr Plotter. While I haven't been able to completely crash it again I did notice the following.

Specifically, I was sending posts to the /csv endpoint while experimenting with different JSON values in the payload to see the effects on the data returned. I noticed that when sending a request for an alignment queryType of "windows" that I could get a non deterministic response from the DB. Sometimes I would get data, sometimes I would get an error message, and sometimes I'd get an error trying to connect.

Here's the JSON payload with the auth token removed:

json={"StartTime":1494405229456,"EndTime":1494405345800,"UUIDS":["2e59878a-62f8-57ff-8834-16de04a45d62"],"Labels":["NavyYard/ pmu3003151/ L1MAG"],"QueryType":"windows","WindowText":"3","WindowUnit":"seconds","UnitofTime":"ms","PointWidth":44,"_token":""}

The primary thing that seemed to cause it to fail was changing the PointWidth value, but I'm not sure if I'm simply seeing patterns in randomness.

Here is the curl command I was sending (with auth token stripped again):

curl --request POST \
  --url https://viz.predictivegrid.com/csv \
  --header 'accept: text/csv' \
  --header 'accept-encoding: gzip, deflate, br' \
  --header 'accept-language: en-US,en;q=0.8' \
  --header 'cache-control: no-cache' \
  --header 'content-type: text/plain' \
  --header 'dnt: 1' \
  --header 'origin: https://viz.predictivegrid.com' \
  --header 'postman-token: 32b4c31a-fe7c-156c-2d40-cea40fa1b1e3' \
  --header 'referer: https://viz.predictivegrid.com/' \
  --header 'upgrade-insecure-requests: 1' \
  --header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' \
  --data 'json={"StartTime":1494405229456,"EndTime":1494405345800,"UUIDS":["2e59878a-62f8-57ff-8834-16de04a45d62"],"Labels":["NavyYard/ pmu3003151/ L1MAG"],"QueryType":"windows","WindowText":"3","WindowUnit":"seconds","UnitofTime":"ms","PointWidth":44,"_token":""}'

That either resulted in data being returned, or this error message:

Could not complete CSV query: rpc error: code = Internal desc = transport is closing

Or it flat out refused to connect.

As I said I haven't been able to reproduce a full crash, but when I did all the API requests stopped responding (e.g. I noticed I could no longer query for streams) although Jerry (@pingthings) relayed that he didn't see anything unusual in the logs.

I'll update this if I figure out more.

Which version supports local filesystem as store + build support

Hello, I am working on single node installation and I would like to build from source and use local filesystem as store. I managed to build and run the database but apparently the file storage engine is no longer supported.

So my question is - which version should I use in this case?

I also tried building what is in branches v4dev and v3 as it seems like there is still the local filesystem support, but I was unable to build. There are some package dependencies I don't have, and I am not sure, how to systematically obtain them (namely for example go-bindata-assetfs - is there a way how to systematically obtain them instead of manual resolution?)

Thanks a lot for help

How to replace the kubernetes controller manager container using minikube

I'm installing the BTrDB on my laptop, using ceph/daemon and minikube as suggested, following the instruction in
https://docs.smartgrid.store/prerequisites.html#connecting-kubernetes-to-ceph.

For kubeadm

edit the file /etc/kubernetes/manifests/kube-controller-manager.json. In that file, change the specified image to btrdb/kubernetes-controller-manager-rbd:1.6.2

However, I cannot find the equivalent configure file for minikube. Any suggestion for this?

Thanks

Nearest N values

Right now, Quasar only supports "nearest value before or after time T", but many sMAP-related applications also want to get the nearest N values.

Issue in Statistical Queries

As documented in interface.capnp, the End Time on statistical queries is exclusive. However, the database in reality may include a record whose start time is equal to the end time of a query.

For an example of this, run the quasarloadgenerator in insert mode (commit fedc74a1d741deeea0b9348584358fd60783b593) with the following config file:


TOTAL_RECORDS=16777216
TCP_CONNECTIONS=3
POINTS_PER_MESSAGE=4096
NANOS_BETWEEN_POINTS=1048576
MAX_TIME_RANDOM_OFFSET=1048575
FIRST_TIME=1420582217226125312
NUM_SERVERS=1
DB_ADDR1=localhost:4410
NUM_STREAMS=1
UUID1=9f67541c-95ee-11e4-a7ac-0026b6df9cf2
MAX_CONCURRENT_MESSAGES=4
RAND_SEED=15
PERM_SEED=0
DETERMINISTIC_KV=false
GET_MESSAGE_TIMES=false
STATISTICAL_PW=27

To see the bug, run it in verify mode. Or, decrease TOTAL_RECORDS to 4096, and the program will receive 4224 points because it received an extra record that contains 128 points.

Interestingly, this problem only shows up when the Cap'n Proto interface is used. The HTTP interface gives proper results.

Help with comparison table details

Hi, I have a table listing open source time series database that I've been collaborating on with some other people who are interested in this subject.

With BtrDB being so new there isn't a lot of information about it. Any chance someone could click the cells and leave some comments for key items like average bytes used per data point stored on disk after compression?

Also, other benchmarks for databases have been done without batching. For example the DalmatinerDB benchmark was setup across three Joyent servers. Two of the servers used to generate the metrics and the 3rd running a copy of the database

https://gist.github.com/sacreman/b77eb561270e19ca973dd5055270fb28

This simulates a more real world scenario where metrics are sent over the network with no batching at all. Batching in memory would inflate the Dalmatiner numbers significantly.

If you could let me know ballpark figures for metrics per second on similar hardware under similar test criteria I'll update the fields.

It would also be good to understand how the clustering works. MongoDB for the metadata and Ceph for the storage are easy to figure out, but how does the BtrDB layer cluster? Can you simply add nodes with everything running to scale out? Also is it configurable with a certain number of replicas of data kept in memory for redundancy.

Lots of questions. Apologies for asking them all as I know you're not released yet, but I got a bit excited :)

Support for streaming data on an edge device

We are evaluating BTrDB (and others) with an eye towards using it as short-term storage (~1 day) on an edge device. Such a device would be a linux server with limited memory and CPU (2-4 cores, 4-8 GB) and would typically be responsible for ingesting data at machine resolution. The simple nature of what was described on the readme for the 3.x distro therefore appealed; and having read the Berkeley paper, this paradigm seems the most promising to us for time series acquisition. A stripped down container with perhaps clustering ability and (ideally) some method for expiring data, particularly at fine-grained levels, would be ideal.

Support grafana

Could btrdb support the ever popular viewing front end grafana, I know this is probably a grafana issue but I would mind knowing how to pull values via http for grafana or if it is even possible

Unexpected (very) short read

I get this panic whenever I want to read/write to btrdb.
Reads/writes happen through giles2.
Any ideas on this?
I already updated both giles and btrdb to the most recent version.
It started when I moved my database files to a new drive, but this problem persists when I use my original drive.

Cheers!
Jonathan

`2016/09/27 11:23:28 cpinterface.go:59 ▶^[[0m cpnp connection
2016/09/27 11:23:32 ^[[35mfileprovider.go:225 ▶^[[0m Unexpected (very) short read
panic: Unexpected (very) short read

goroutine 35 [running]:
panic(0x833100, 0xc8200f4a10)
/usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/op/go-logging.(_Logger).Panic(0xc8200589c0, 0xc8200f4980, 0x1, 0x1)
/home/jofu/go/src/github.com/op/go-logging/logger.go:188 +0xc8
github.com/SoftwareDefinedBuildings/btrdb/internal/fileprovider.(_FileStorageProvider).Read(0xc82004ccb0, 0xc8203a6048, 0x10, 0x5b8, 0x36400004255f6b3, 0xc8203e2000, 0x5007, 0x5007, 0x0, 0x0, ...)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/internal/fileprovider/fileprovider.go:225 +0x3ba
github.com/SoftwareDefinedBuildings/btrdb/internal/bstore.(_BlockStore).ReadDatablock(0xc82007c120, 0xc8203a6048, 0x10, 0x5b8, 0x36400004255f6b3, 0x11dd4, 0x38, 0xf000000000000000, 0x0, 0x0)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/internal/bstore/blockstore.go:308 +0x19c
github.com/SoftwareDefinedBuildings/btrdb/qtree.(_QTree).LoadNode(0xc8200f8630, 0x36400004255f6b3, 0x11dd4, 0x538, 0xf000000000000000, 0xc8200f8600, 0x0, 0x0)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/qtree/qtree_utils.go:120 +0xbb
github.com/SoftwareDefinedBuildings/btrdb/qtree.NewReadQTree(0xc82007c120, 0xc8203a6048, 0x10, 0x5b8, 0xffffffffffffffff, 0x0, 0x0, 0x0)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/qtree/qtree_utils.go:187 +0x165
github.com/SoftwareDefinedBuildings/btrdb.(_Quasar).QueryValuesStream(0xc8200129b0, 0xc8203a6048, 0x10, 0x5b8, 0x147822aee5ce5355, 0x147823c64b68efbc, 0xffffffffffffffff, 0xc8200f8240, 0x20, 0x0)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/quasar.go:194 +0x5a
github.com/SoftwareDefinedBuildings/btrdb/cpinterface.(_CPInterface).dispatchCommands.func1(0xc8200f8240, 0xc820119f28, 0x7f845e403bb0, 0xc820021108, 0xc8200129b0, 0x0, 0x0)
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/cpinterface/cpinterface.go:94 +0xa12
created by github.com/SoftwareDefinedBuildings/btrdb/cpinterface.(*CPInterface).dispatchCommands
/home/jofu/go/src/github.com/SoftwareDefinedBuildings/btrdb/cpinterface/cpinterface.go:441 +0x475
`

Installation error

I was hoping to evaluate btrdb but ran into difficulty during installation while following the instructions in the README in the master branch. The instructions say to execute:

go get github.com/SoftwareDefinedBuildings/btrdb/server

However, this results in the error:

package github.com/SoftwareDefinedBuildings/btrdb/server: cannot find package "github.com/SoftwareDefinedBuildings/btrdb/server" in any of:
/usr/lib/go/src/github.com/SoftwareDefinedBuildings/btrdb/server (from $GOROOT)

While I'm not a Go developer it seems as if the installation is expecting a server folder under the root which is no longer available and I'm unsure how to continue. Using the alternate installation method also appears to error as shown:

$ go get -d ./... && go install ./btrdbd
package ./btrdb
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bstore: use of internal package not allowed
package ./btrdb/btrdbd
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bstore: use of internal package not allowed
package ./btrdb/internal/bstore
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bprovider: use of internal package not allowed
package ./btrdb/internal/bstore
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/cephprovider: use of internal package not allowed
package ./btrdb/internal/bstore
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/fileprovider: use of internal package not allowed
package ./btrdb/internal/cephprovider
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bprovider: use of internal package not allowed
package ./btrdb/internal/fileprovider
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bprovider: use of internal package not allowed
package ./btrdb/qtree
    imports github.com/SoftwareDefinedBuildings/btrdb/internal/bstore: use of internal package not allowed
gocode/src/github.com/stretchr/graceful/tests/main.go:7:2: cannot find package "github.com/codegangsta/negroni" in any of:
    /usr/lib/go/src/github.com/codegangsta/negroni (from $GOROOT)
    /home/allen/gocode/src/github.com/codegangsta/negroni (from $GOPATH)
gocode/src/github.com/stretchr/graceful/tests/main.go:8:2: cannot find package "github.com/tylerb/graceful" in any of:
    /usr/lib/go/src/github.com/tylerb/graceful (from $GOROOT)
    /home/allen/gocode/src/github.com/tylerb/graceful (from $GOPATH)
gocode/src/golang.org/x/net/html/charset/charset.go:20:2: cannot find package "golang.org/x/text/encoding" in any of:
    /usr/lib/go/src/golang.org/x/text/encoding (from $GOROOT)
    /home/allen/gocode/src/golang.org/x/text/encoding (from $GOPATH)
gocode/src/golang.org/x/net/html/charset/charset.go:21:2: cannot find package "golang.org/x/text/encoding/charmap" in any of:
    /usr/lib/go/src/golang.org/x/text/encoding/charmap (from $GOROOT)
    /home/allen/gocode/src/golang.org/x/text/encoding/charmap (from $GOPATH)
gocode/src/golang.org/x/net/html/charset/charset.go:22:2: cannot find package "golang.org/x/text/encoding/htmlindex" in any of:
    /usr/lib/go/src/golang.org/x/text/encoding/htmlindex (from $GOROOT)
    /home/allen/gocode/src/golang.org/x/text/encoding/htmlindex (from $GOPATH)
gocode/src/golang.org/x/net/html/charset/charset.go:23:2: cannot find package "golang.org/x/text/transform" in any of:
    /usr/lib/go/src/golang.org/x/text/transform (from $GOROOT)
    /home/allen/gocode/src/golang.org/x/text/transform (from $GOPATH)
gocode/src/golang.org/x/net/http2/h2i/h2i.go:38:2: cannot find package "golang.org/x/crypto/ssh/terminal" in any of:
    /usr/lib/go/src/golang.org/x/crypto/ssh/terminal (from $GOROOT)
    /home/allen/gocode/src/golang.org/x/crypto/ssh/terminal (from $GOPATH)
gocode/src/gopkg.in/mgo.v2/dbtest/dbserver.go:13:2: cannot find package "gopkg.in/tomb.v2" in any of:
    /usr/lib/go/src/gopkg.in/tomb.v2 (from $GOROOT)
    /home/allen/gocode/src/gopkg.in/tomb.v2 (from $GOPATH)

This was using a clean install of Ubuntu 14.04, fully updated, with librados-dev and MongoDB already installed. I found the FAST paper extremely interesting and was hoping someone could point me in the right direction. Thanks

BTrDB Single Node Installation

We are working on a single-node installation of BTrDB. We have gotten the basic Ceph installation from https://docs.smartgrid.store/ working (from the Prerequisites section), but we are having trouble finding a good guide for installing kubernetes on a single node, as well as knowing which features we need to look at installing (e.g. do we need 'flannel' for networking?). Can you recommend one?

Path trailing / omit

When using the admin console we need to trim any potential / that would end the tag at every level of the path. The value should never end in / it's messing with the tree in Mr. Plotter.
eg.
path=myPMU/mydevice/ WRONG
path=myPMU/mydevice CORRECT

make fails; dep not found

Hello developer team,
I wanted to install the btrdb-server on Ubuntu 16.04, but the make process fails with in the step:
cd $(GOPATH)/src/github.com/BTrDB/btrdb-server && dep ensure
and fails with:
/bin/sh: 1: dep: not found
Please can you let me know what could be the issue and which other OS would be preferred?
Thank you,
Christoph

License :(

GPLv3 is just about the most commercial-unfriendly license that could have been chosen. Was the choice intentional?

Use BTrDB Instantly

Is there any way that I can try BTrDB instantly without installing Ceph or Kubernetes? like InfluxDB.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.