featurebasedb / featurebase Goto Github PK

A common use case is to set/clear the same bit on many bitmaps. In order to reduce the http overhead we have batched this process into a a single request with the endpoints /set_bits and /clear_bits. The current JSON format looks like..

[{ "db": "3", "frame":"b.n","profile_id": 122,"filter":0, "bitmap_id":123},
    { "db": "3", "frame":"b.n","profile_id": 122,"filter":2, "bitmap_id":124}]

we should modify to handle the new more flexible attributes(filter)
cc @benbjohnson

So I've manage to kill the pilosa process attempting to import one of our larger properties on a single memory constrained, 8GB, box. The process simply died and didn't appear to go through the shutdown routine. I will try and get you better example but basically i'm just importing many (millions) of small bitmaps with a single attribute with around 700 possible values into around 60 slices. In the dream world, this wouldn't be a problem because only the large bitmaps would be in memory window, the smaller ones would be paged out. I'm going to continue testing, hopefully so i will be able to provide a concrete example and we could develop a strategy for handling this use case a bit more elegantly. I'm working under the assumption this is a problem with the storage layer, but i can't confirm at this time, it just behaves like a memory leak.

Lemme know if you have any thoughts @benbjohnson or requests for specific data.

In db.go we use remoteMaxSlice to hold the DB's max slice according to this node.
The value represents the highest slice for DB reported by any node via server.monitorMaxSlices() which runs every 60 seconds.

On every query execution, we distribute the query to all slices based on the maximum slice as it is understood by this node. MaxSlice is determined by taking the highest value of:

• remoteMaxSlice
• maximum slice for DB on this node
  This logic is encapsulated by db.MaxSlice().

We might consider changing remoteMaxSlice to be globalMaxSlice and update it via both server.monitorMaxSlices() and locally as slices are added to DB. This would allow us to simply reference db.globalMaxSlice instead of looking through all the frames as is currently done in db.MaxSlice(). We could keep db.MaxSlice() to do a deep lookup based on the schema, but use db.GlobalMaxSlice() as a cached lookup.

Currently the toml package does not validate for non-existent fields. It skips them silently. This needs to be changed to report an error.

pilosactl bspawn $GOPATH/src/github.com/pilosa/pilosa/cmd/pilosactl/multidbspawn.json

gives results like: min:173.408µs max:581.302146ms avg:295.155µs

173µs to 581ms is a massive range - gonna try to figure out why and see if it can be fixed.

Hola! @raskle has created a ZenHub account for the pilosa organization. ZenHub is the only project management tool integrated natively in GitHub – created specifically for fast-moving, software-driven teams.

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Multi-Repository burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• Time-saving shortcuts – like a quick repo switcher, a “Move issue” button, and much more.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @raskle.

PR: #139
broke the pilosactl command

cmd/pilosactl/main.go:482: not enough arguments in call to client.SliceN
cmd/pilosactl/main.go:490: not enough arguments in call to client.ExportCSV
cmd/pilosactl/main.go:683: not enough arguments in call to client.BackupTo
cmd/pilosactl/main.go:776: not enough arguments in call to client.RestoreFrom
cmd/pilosactl/main.go:1137: not enough arguments in call to client.ExecuteQuery```

If I import data to a bitmap that already has bits set, then any new profiles in the import are not represented in the Bitmap unless I restart Pilosa. This implies that the import data are making it to disk storage, but not the cache in memory.

To recreate this:

pilosactl import -host 127.0.0.1:15000 -d 1 -f bench import1.csv
where import1.csv contains

1,1
1,2
1,3
1,4
1,5

verify with this
curl -X POST "http://127.0.0.1:15000/query?db=1" -d 'Bitmap(id=1, frame="bench")'
{"results":[{"attrs":{},"bits":[1,2,3,4,5]}]}

Then run this
pilosactl import -host 127.0.0.1:15000 -d 1 -f bench import2.csv
where import2.csv contains

1,1
1,2
1,3
1,4
1,5
1,6

verify with this and you'll still only see 5 bits:
curl -X POST "http://127.0.0.1:15000/query?db=1" -d 'Bitmap(id=1, frame="bench")'
{"results":[{"attrs":{},"bits":[1,2,3,4,5]}]}

Next, import into a different bitmap:
pilosactl import -host 127.0.0.1:15000 -d 1 -f bench import3.csv
where import3.csv contains

3,1
3,2
3,3

verify with this and you'll see no bits set:
curl -X POST "http://127.0.0.1:15000/query?db=1" -d 'Bitmap(id=3, frame="bench")'
{"results":[{"attrs":{},"bits":[]}]}

Restart Pilosa, run the same query and you'll get:
{"results":[{"attrs":{},"bits":[1,2,3]}]}

I also tested by manually setting a bit to see if it mattered whether or not the existing bits had been set by the import function or now and there was no difference.

set operations do not appear to be functional at this point. Here was my test
`
$curl -X POST http://localhost:15000/query?db=2 -d'set(id=123,frame="b.n",profile_id=3)'
{}

2015121109:03:38(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'set(id=123,frame="b.n",profile_id=2)'
{}

2015121109:03:43(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'set(id=123,frame="b.n",profile_id=1)'
{}

2015121109:03:46(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'get(id=123,frame="b.n")'
{"result":{"chunks":[{"Key":0,"Value":[14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}]}}

$curl -X POST http://localhost:15000/query?db=2 -d'count(get(id=123,frame="b.n"))'
{"result":3}

2015121109:05:43(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'count(union(get(id=123,frame="b.n")))'
{"result":3}

2015121109:06:08(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'count(intersect(get(id=123,frame="b.n")))'
{"result":3}

2015121109:06:35(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'count(intersect(get(id=123,frame="b.n"),get(id=123,frame="b.n")))'
{"result":0}

2015121109:07:10(Todds-Machine)(~/review/src/github.com/umbel/pilosa)
$curl -X POST http://localhost:15000/query?db=2 -d'count(union(get(id=123,frame="b.n"),get(id=123,frame="b.n")))'
{"result":0}
`

The expected result would be 3 for all the actions except the set

Overview

Currently pilosa kicks off a goroutine for every fragment in order to flush the cache periodically. This means that a server with tens of thousands of fragments will have a lot of goroutines which cause IO spikes if all fragments attempt to flush simultaneously. It also makes debugging stack traces impossible.

The flush monitor should be extracted out so that it iterates the fragments serially and flushes one at a time.

Notes

I'm not sure if this is related but I'm seeing a SIGABRT with the following error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable

Add the ability to load bitmaps in bulk instead of by individual "set bit" commands.

Command to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets to test pilosa

I'm having difficulty getting you a simple case but it might work if you load this file

It does seem that it tries to access a large number of file handles in the process

https://s3.amazonaws.com/tgruben.umbel.com/problem.csv.gz
curl -XPOST "http://pilosa2-test1:15000/query?db=13" -d'ClearBit(123456,"b.n",3)'

You might just look at clearbit @benbjohnson and see if anything is glaring. I can't reproduce on an empty dataset, but a fully loaded one seems to sometime hang indefinately

Process to create create and map a distributes auto incremented int ID for incoming profile UUID's

The pilosactl import should support multiple hosts. The pilosa.Client should retrieve the cluster configuration and use that to direct each slice to its host(s).

Add the ability to set key/values for a given profile ID.

Use BoltDB to store bitmap snapshots and changelog. Segment by db.

This should improve start up time (which is currently 30m).

Use a consistent hashing library to route incoming profile UUIDs to the node that is responsible for looking up the Pilosa ID.

Specifically, we should distribute a map of virtual hash ring partitions per database across the cluster. The hostname of the node should not be identifying owner of the id space.

Success criteria:
Any allocator node can hash an incoming key and route the request to the correct node that will handle the request.

This epic is a placeholder for doing general testing of pilosa cluster in the event of network disturbances. Potentially using the Jepsen framework.

Here's the stackdump.
2016/08/27 16:05:27 GET /fragment/blocks?db=554&frame=tag.n&slice=1 0.001s
2016/08/27 16:05:42 Go Import: 554 frame1.n 0
2016/08/27 16:05:42 Import into Fragment: 554 frame1.n 0 1279535633
2016/08/27 16:07:40 fragment: snapshotting 554/frame1.n/0
2016/08/27 16:07:41 http: panic serving 10.0.3.184:40951: runtime error: slice bounds out of range
goroutine 108 [running]:
net/http.(_conn).serve.func1(0xc42015c280)
/usr/local/Cellar/go/1.7/libexec/src/net/http/server.go:1491 +0x12a
panic(0x7b82c0, 0xc42000c100)
/usr/local/Cellar/go/1.7/libexec/src/runtime/panic.go:458 +0x243
github.com/umbel/pilosa.(_Fragment).openStorage(0xc42011a0b0, 0x2e, 0xc5729ec5d0)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:194 +0x5a6
github.com/umbel/pilosa.(_Fragment).snapshot(0xc42011a0b0, 0x0, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:958 +0x726
github.com/umbel/pilosa.(_Fragment).Import(0xc42011a0b0, 0xcc72bde000, 0x4c442a11, 0x559eb800, 0xc572bde000, 0x4c442a11, 0x559eb800, 0x0, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:899 +0x17f
github.com/umbel/pilosa.(_Handler).handlePostImport(0xc4200187d0, 0x9e8dc0, 0xc4201925b0, 0xc4200f61e0)
/Users/tgruben/review/src/github.com/umbel/pilosa/handler.go:414 +0xc68
github.com/umbel/pilosa.(_Handler).ServeHTTP(0xc4200187d0, 0x9e8dc0, 0xc4201925b0, 0xc4200f61e0)
/Users/tgruben/review/src/github.com/umbel/pilosa/handler.go:90 +0x708
net/http.serverHandler.ServeHTTP(0xc42015c000, 0x9e8dc0, 0xc4201925b0, 0xc4200f61e0)
/usr/local/Cellar/go/1.7/libexec/src/net/http/server.go:2202 +0x7d
net/http.(_conn).serve(0xc42015c280, 0x9e98c0, 0xc42d632200)
/usr/local/Cellar/go/1.7/libexec/src/net/http/server.go:1579 +0x4b7
created by net/http.(_Server).Serve
/usr/local/Cellar/go/1.7/libexec/src/net/http/server.go:2293 +0x44d

It did work successfully on the smaller frame types Here is the importer log

~/pilosactl import -host 10.0.3.249:15000 -d 554 -f d 554-d.bits
2016/08/27 15:06:48 parsing: 554-d.bits
2016/08/27 15:09:01 grouping 180630636 bits
2016/08/27 15:09:43 grouped into 2 slices
2016/08/27 15:09:43 importing slice: 0, n=94859160
2016/08/27 15:10:25 importing slice: 1, n=85771476
~/pilosactl import -host 10.0.3.249:15000 -d 554 -f l.n 554-l.n.bits
2016/08/27 15:11:01 parsing: 554-l.n.bits
2016/08/27 15:11:16 grouping 20352874 bits
2016/08/27 15:11:21 grouped into 2 slices
2016/08/27 15:11:21 importing slice: 0, n=10193192
2016/08/27 15:11:30 importing slice: 1, n=10159682
~/pilosactl import -host 10.0.3.249:15000 -d 554 -f tag.n 554-tag.n.bits
2016/08/27 15:11:40 parsing: 554-tag.n.bits
2016/08/27 15:15:49 grouping 329451476 bits
2016/08/27 15:17:10 grouped into 2 slices
2016/08/27 15:17:10 importing slice: 0, n=178228868
2016/08/27 15:18:31 importing slice: 1, n=151222608
~/pilosactl import -host 10.0.3.249:15000 -d 554 -f frame1.n 554-frame1.n.bits
2016/08/27 15:19:36 parsing: 554-frame1.n.bits
2016/08/27 15:51:11 grouping 2434818638 bits
2016/08/27 16:02:02 grouped into 2 slices
2016/08/27 16:02:02 importing slice: 0, n=1279535633
import node: host=10.0.3.249:15000, err=Post http://10.0.3.249:15000/import: EOF

Its failing on this line

    data := (*[0x7FFFFFFF]byte)(unsafe.Pointer(&f.storageData[0]))[:fi.Size()]

@benbjohnson I'll do some more digging and see what i can figure out.

• Consistent hashing (64 partitions)
• Remove transport/hold/dispatch
• HTTP endpoint
• Remove BitmapHandle
• gogoprotobuf serialization

Add the ability to replicate data across multiple nodes

Wire up the top-n() functionality again.

Steps to recreate:

Against a completely empty Pilosa instance, run the following:

curl -X POST "http://127.0.0.1:15000/query?db=wideDB" -d 'SetBit(id=10, frame="foo", profileID=1)'
curl -X POST "http://127.0.0.1:15000/query?db=wideDB" -d 'SetBit(id=10, frame="foo", profileID=10000000)'
curl -X POST "http://127.0.0.1:15000/query?db=narrowDB" -d 'SetBit(id=20, frame="foo", profileID=1)'
curl -X POST "http://127.0.0.1:15000/query?db=narrowDB" -d 'SetBit(id=20, frame="foo", profileID=1300000)'

This creates two databases.
wideDB contains data in slices 0 and 9
narrowDB contains data in slices 0 and 1

The data file tree looks like this:

├── narrowDB
│   ├── data
│   └── foo
│       ├── 0
│       ├── 0.cache
│       ├── 1
│       ├── 1.cache
│       └── data
└── wideDB
    ├── data
    └── foo
        ├── 0
        ├── 0.cache
        ├── 9
        ├── 9.cache
        └── data

After index sync runs, the data file tree looks like this:

├── narrowDB
│   ├── data
│   └── foo
│       ├── 0
│       ├── 0.cache
│       ├── 1
│       ├── 1.cache
│       ├── 2
│       ├── 3
│       ├── 4
│       ├── 5
│       ├── 6
│       ├── 7
│       ├── 8
│       ├── 9
│       └── data
└── wideDB
    ├── data
    └── foo
        ├── 0
        ├── 0.cache
        ├── 1
        ├── 2
        ├── 3
        ├── 4
        ├── 5
        ├── 6
        ├── 7
        ├── 8
        ├── 9
        ├── 9.cache
        └── data

Notice that narrowDB now contains 10 slices, even though only wideDB should contain 10 slices.

Need some process to move a fragment to a new node. Worse case an offline process, ideally completely online without noticeable service interruption

So TopN will work after import

Seems we broke something. The size of the intersect is larger than the smallest bitmap. This is from this dataset. You mind checking that out @benbjohnson

https://s3.amazonaws.com/pilosa.umbel.com/loader2/295/296-d.bits

$time curl -XPOST "http://localhost:15005/query?db=296" -d'Count(Bitmap(id=25803919,frame=d))' | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 53 100 19 100 34 3010 5386 --:--:-- --:--:-- --:--:-- 5666
{
"results": [
1982
]
}

real 0m0.034s
user 0m0.023s
sys 0m0.018s
[s]2016092214:29:35(Todds-Machine)(~/review/src/github.com/umbel/pilosa/cmd/pilosactl):master
$time curl -XPOST "http://localhost:15005/query?db=296" -d'Count(Bitmap(id=1345,frame=d))' | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 50 100 20 100 30 3216 4825 --:--:-- --:--:-- --:--:-- 6000
{
"results": [
24658
]
}

real 0m0.036s
user 0m0.024s
sys 0m0.019s
[s]2016092214:29:55(Todds-Machine)(~/review/src/github.com/umbel/pilosa/cmd/pilosactl):master
$time curl -XPOST "http://localhost:15005/query?db=296" -d'Count(Intersect(Bitmap(id=1345,frame=d),Bitmap(id=25803919,frame=d)))' | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 88 100 19 100 69 2976 10809 --:--:-- --:--:-- --:--:-- 11500
{
"results": [
3314
]
}

real 0m0.033s
user 0m0.023s
sys 0m0.018s

Add the ability to back up and restore a partition.

Query is returning different values depending on which node hit.

$curl -X POST "http://localhost:15000/query?db=3" -d 'count(get(id=1237,frame=d))'
{"result":64687} <- correct
2016011215:19:22(Todds-Machine)(~)
$curl -X POST "http://localhost:15001/query?db=3" -d 'count(get(id=1237,frame=d))'
{"result":49718} <-incorrect

Possibly only calculate changed blocks.

Overview

Currently the PQL language uses get() to retrieve bitmaps and set() to set individual bits. The set() also allows the filter bitmap attribute to be set.

As more bitmap attributes are added and profile attributes are added, I think that adjusting the query language would be helpful. It would also provide performance benefits as bitmap attributes don't need to be set on every bit set.

Proposal

I propose the following PQL changes:

1. Rename get() to bitmap(). This makes it more clear what is being retrieved and if we add profile() to get profile attributes then it prevents ambiguity. The bitmap() call will also return a map of bitmap attributes.
2. Rename set() to set-bit(). Again, this makes it more clear and prevents ambiguity with other set operations.
3. Add profile() to return profile attributes.
4. Add set-bitmap-attr() to set one or more bitmap attributes.
5. Add set-profile-attr() to set one or more profile attributes.

/cc @travisturner @tgruben

Wire up range() functionality again.

Seems to be an off by one kinda problem that shows up in the iterator. I've managed to narrow it to somewhere in the container allocation i think. I've got two import files that only differ by one line and one works and the other is broken. I happens when the highbits it the magic number of 1024, specifically the j value is between 65536 and 65599 inclusive.

broke.txt
works.txt

#97 still needs to be addressed, although it's partly a question of whether sync is fast enough.
Parallelizing it will spike the CPU more

Currently the cache stores bitmap counts and bitmap data. However, with the new roaring mmap'd storage this is redundant and causes huge heap allocations and long startup times.

The bitmap data should be removed from the cache and the roaring bitmaps should be used directly. The bit counts will still remain in the cache for fast count and top execution.

It would be nice to have an endpoint that would provide all database/frames contained in cluster

Add the ability to set attributes on a bitmap that can be filtered and sorted on. (e.g. category & sort index).

Overview

Previously, local storage was going to be backed by BoltDB, however, it will be more space efficient and time efficient to back the data with a format that can be optimized for bitmaps.

This proposal suggests using a combination of roaring bitmaps for periodically snapshotting slices and then appending a write-ahead log for each set or clear operation.

Data Structures

Roaring bitmaps are a two-level structure. The top level groups integers together into "containers" of integers. It does this by using the high bits of a uint32. The containers contain only the low bits.

There are two types of containers:

• array containers - these contain 4096 or fewer integers
• bitmap containers - these contain more than 4096 integers

These containers are automatically converted between each other once the cardinality threshold is passed.

Our resulting data structures are:

type Bitmap struct {
    keys       []uint16
    containers []*container
}

type container struct {
    n      int      // number of integers in container
    array  []uint16 // used for array containers
    bitmap []uint64 // used for bitmap containers
}

Memory-mapping

Roaring bitmaps file format

The roaring file format is as follows:

<cookie> <key-count> (<key>,<cardinality>)+ (<offset>)+ (<array> | <bitmap>)+

These values are all encoded with little endianness so we can use the pointers to array and bitmap directly from the mmap and use them as []uint16 and []uint64 via the unsafe package:

b := ([0xFFFFFFFF]byte)(unsafe.Pointer(&buf[offset]))[:]

This gives us two benefits:

1. We can handle datasets larger than RAM.
2. Memory maps are stored in the OS page cache so they persist across restarts. This gives us a low start up cost.

We don't need to change the bitmap file format at all.

Write ahead log

The roaring bitmap file format works well for snapshots of bitmaps but it's costly to rewrite the entire snapshot on each change. To prevent this, we can simply append operations to the end of the file in TLV fomat and using protobufs to encode the value:

<checksum(uint32)><type(uint8)><length(uint32)><value([]byte)>

When pilosa needs to read in a slice for the first time, it can open the file, materialize the bitmap from the snapshot, and then perform the operations within the WAL.

When operations are made to the bitmap, they are first written to the WAL and then performed on the bitmap data in-memory. When the in-memory operation is done, the container will need to copy its data from the underlying mmap into the heap and mark itself dirty. Future operations on a dirty container won't require a copy.

The container will need a dirty bool flag added to the structure mentioned above.

Snapshotting

Periodically, the bitmaps will need to write a new snapshot to disk. This is for two reasons:

1. The write ahead log will continue to get larger and larger.
2. Dirty containers take up memory in RAM and will cause an OOM if they take up too much space.

To prevent these issues, we'll need two checks:

1. Check the WAL size after each Set() and Clear() operation and snapshot after a threshold.
2. Maintain the total heap size used by all bitmaps and force large or least recently used bitmaps to be flushed after a configurable threshold.

Conclusion

Todd's existing roaring implementation could be revamped to support references to mmaps, however, I think it'll require a decent amount of change -- especially since it would require adding non-standard delta operations.

Since pilosa only requires a few simple bitmap operations I think it's easier to simply extract out a subset of the package into a roaring package within pilosa. I already did part of this work while I was researching this task.

/cc @travisturner @tgruben

Our roaring implementation uses uint32 values (which matches the standard). This needs to be upgraded to uint64 so we can use larger values.

Godep is really confusing. We should try something different.

Currently the clustering only uses the slice when determining the location of a fragment in the cluster. However, if there are not many slices then this causes much of the data to be placed on a single node (and its secondary replicas).

We should combine the db, frame, and slice using an fnv1a hash to determine the location of the primary node for a fragment.

set up link to docker compose that allow containers to discover each other and securely transfer information about one container to another container

There is something wrong with the AntiEntropy. The setBit works fine from the pql path(SetBit via the query endpoint), but sometimes fails via the syncIndex route.

Sometimes....once every hour or so, the container array doesn't have any elements and this line will throw an out of bounds panic.

func (c *container) arrayWriteTo(w io.Writer) (n int64, err error) {
        nn, err := w.Write((*[0xFFFFFFF]byte)(unsafe.Pointer(&c.array[0]))[:4*c.n])
}

I've put in some bounds checking and it prevents the panic but the problem still remains.  Here is a stack trace of the bad path, i suspect something in syncBlock or MergeBlock

github.com/umbel/pilosa/roaring.(*container).arrayWriteTo(0xc441afbac0, 0x9f07c0, 0xc5d60254c0, 0x8, 0x0, 0x0)
    /Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:1031 +0xc0
github.com/umbel/pilosa/roaring.(*container).WriteTo(0xc441afbac0, 0x9f07c0, 0xc5d60254c0, 0x8, 0x0, 0x0)
    /Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:1006 +0x91
github.com/umbel/pilosa/roaring.(*Bitmap).WriteTo(0xc440c52d70, 0x9f07c0, 0xc5d60254c0, 0xc886cdb000, 0x1000, 0x1000)
    /Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:461 +0x2c0
github.com/umbel/pilosa.(*Fragment).snapshot(0xc440c986e0, 0x0, 0x0)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:965 +0x3d8
github.com/umbel/pilosa.(*Fragment).incrementOpN(0xc440c986e0, 0xc442720b40, 0xc444cf7608)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:938 +0x4a
github.com/umbel/pilosa.(*Fragment).setBit(0xc440c986e0, 0x53f, 0x2c0669, 0x1, 0x21, 0xc5d1f5b200)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:363 +0x13f
github.com/umbel/pilosa.(*Fragment).MergeBlock(0xc440c986e0, 0xd, 0xc6dccd2330, 0x1, 0x1, 0xc9008c11a0, 0x2, 0x2, 0xc9008c1200, 0x2, ...)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:841 +0xf1e
github.com/umbel/pilosa.(*FragmentSyncer).syncBlock(0xc444cf7c40, 0xd, 0x2, 0xc6dca2f400)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:1371 +0x688
github.com/umbel/pilosa.(*FragmentSyncer).SyncFragment(0xc444cf7c40, 0xc42015ed33, 0x3)
    /Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:1322 +0x714
github.com/umbel/pilosa.(*IndexSyncer).syncFragment(0xc444cf7eb8, 0xc42015ed33, 0x3, 0xc440c4d637, 0x1, 0x2, 0x1, 0x0)
    /Users/tgruben/review/src/github.com/umbel/pilosa/index.go:469 +0xfa
github.com/umbel/pilosa.(*IndexSyncer).SyncIndex(0xc444cf7eb8, 0x83782a, 0x14)
    /Users/tgruben/review/src/github.com/umbel/pilosa/index.go:354 +0x32a
github.com/umbel/pilosa.(*Server).monitorAntiEntropy(0xc42015a070)
    /Users/tgruben/review/src/github.com/umbel/pilosa/server.go:173 +0x410
github.com/umbel/pilosa.(*Server).Open.func2(0xc42015a070)
    /Users/tgruben/review/src/github.com/umbel/pilosa/server.go:117 +0x57

You have any ideas @benbjohnson ?

Something not right on the roaring bitmap. I've got it narrowed down to the WriteTo method of the array.

The error reported is

2016/01/08 10:09:30 import error: db=3, frame=b.n, slice=59, bits=5434

I'm thinking there might be an overflow somewhere, but i can't find it.
1237.txt

The topn needs a bit of adjustment. The process should be:

scatter out topn to each slice
gather results
-->fetch missing counts from non reporting slices<-- the missing part
sort and return top n

I've added a test, TestExecutor_Execute_TopN_fill, to express the problem

you think you can fix that @benbjohnson ?

Found a little snag @benbjohnson Let me know what you think

https://s3.amazonaws.com/tgruben.umbel.com/problem.csv.gz

Looks like an issue with scatter part of TopN.

$ curl -XPOST "http://pilosa2-test1:15000/query?db=13" -d'TopN(Intersect(Bitmap(id=0,frame=s.n),Bitmap(id=1,frame=s.n)),frame=s.n,n=2)' | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   176  100   100  100    76    326    247 --:--:-- --:--:-- --:--:--   325
{
    "results": [
        [
            {
                "count": 82024,
                "key": 1
            },
            {
                "count": 1,
                "key": 2
            }
        ]
    ]
}

$ curl -XPOST "http://pilosa2-test1:15000/query?db=13" -d'TopN(Bitmap(id=0,frame=s.n),frame=s.n,n=2)' | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   104    0    62    0    42      0      0 --:--:--  0:01:39 --:--:--     5
{
    "results": [
        [
            {
                "count": 1999998,
                "key": 1
            },
            {
                "count": 1,
                "key": 2
            }
        ]
    ]
}

FYI, this is correct both ^ and v. Just fyi bitmap 0 is all prime numbers less than 32 million ish

$ curl -XPOST "http://pilosa2-test1:15000/query?db=13" -d'Count(Intersect(Bitmap(id=0,frame=s.n),Bitmap(id=0,frame=s.n)))' | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    85  100    22  100    63    142    408 --:--:-- --:--:-- --:--:--   409
{
    "results": [
        2000000
    ]
}