tidwall / tile38 Goto Github PK

version: 1.2.0
OS: Windows 10 x86_64

tile38-cli.exe outputs:

D:\programs\tile38>tile38-cli.exe
GetFileInformationByHandle /dev/stdout: Incorrect function.

Hi -

I'm having trouble getting something simple to work: I want to test to see if a GeoJson Point is inside a GeoJson Polygon:

Here is the polygon definition:

SET cities tempe OBJECT {"type":"Polygon","coordinates":[[[-111.9787,33.4411],[-111.8902,33.4377],[-111.8950,33.2892],[-111.9739,33.2932],[-111.9787,33.4411]]]}
{"ok":true,"elapsed":"8.463753ms"}

Here is the query for a point inside the Polygon (Tempe lat/lon from Google) that I expect to work but does not:

WITHIN cities OBJECT { "type": "Point", "coordinates": [ -111.9400, 33.4255] }
{"ok":true,"objects":[],"count":0,"cursor":0,"elapsed":"65.084µs"}

I also tried INTERSECTS:

INTERSECTS cities OBJECT { "type": "Point", "coordinates": [ -111.9400, 33.4255] }
{"ok":true,"objects":[],"count":0,"cursor":0,"elapsed":"12.298599ms"}

If I query with the entire Polygon, it does work:

INTERSECTS cities object {"type":"Polygon","coordinates":[[[-111.9787,33.4411],[-111.8902,33.4377],[-111.8950,33.2892],[-111.9739,33.2932],[-111.9787,33.4411]]]}
{"ok":true,"objects":[{"id":"tempe","object":{"type":"Polygon","coordinates":[[[-111.9787,33.4411],[-111.8902,33.4377],[-111.895,33.2892],[-111.9739,33.2932],[-111.9787,33.4411]]]}}],"count":1,"cursor":0,"elapsed":"11.765324ms"}

if you do, do you have https://devcenter.heroku.com/articles/deploying-go or procfile?

First let me say this is a very cool project, specifically the geo fencing.

I have not had a ton of time to play around with this let so I figured I may as well just ask, is there a limit to the size of the geojson you support?

I ask this because one of my files is about 200mb give or take.

Thanks

Hello:
First off - I love the idea of Tile38. Thank you.

I'm trying to use HTTP POSTs to talk to the tile38-server but, regardless of trying from Ruby or Java, I keep getting invalid argument responses like this: {"ok":false,"err":"invalid argument '500\r\n'","elapsed":"66.045µs"}

It seems that the return-newline pair in the request body is being run on to the last parameter in the command string which, of course, results in a parse error.

To compound matters, I know nothing about Go! Any advice?

Thanks,

David.

Hello !

Would you be interested that we (Redsmin) promote tile38 as a compatible server for our online real-time administration and monitoring service?

As for as I can see Tile38 already handles scan command, if only it could implement the type and info command, you would get a GUI and monitoring/alerting for free :)

When quickly looking at the code I don't see any option or setting to define max memory tile38 can use up - are there plans to add this?
Just would rather have errors returned from tile38 that max memory has been reached than an OOM killer killing the whole process.

In our project, we need to check if two users are near to each other. How should we go about using webhooks to achieve this result? If we needed to check if a user was close to a static location, but since these are relative and moving positions...

if I store about ten thousand fence , how do I export a copy of the data?

thanks.

hi，i met a problem about query.
When i query using geojson object, such as
'within city object OBJECT {"type":"Polygon","coordinates":[[[-111.9787,33.4411],[-111.8902,33.4377],[-111.8950,33.2892],[-111.9739,33.2932],[-111.9787,33.4411]]]}
it always returns error 'invalid argument object'.
can you help me solve the problem,thank you!

If Tile38 have a mechanism that can set a range of monitoring would be the best. For example, the client sent the "fset" command to set the values of a field can also bind a hook, specifically, if a truck‘s speed reaches a certain value,tile38 will trigger a values fencing event.

Sorry if this is a dumb question...just a little lost on how to interact w/ the tile38-server via redis-rb. I've got as far as starting the the tile38-server and then connecting to the server & port using redis-rb, but then I'm lost. What syntax should I use to do something like setting a point?

Redis-rb docs say this for setting a key: redis.set("mykey", "hello world")
So I tried: redis.set("fleet", "truck", "point", "33.5123 -112.2693") (didn't worK)
..and, stuff like redis.set("fleet", "truck", point: "33.5123 -112.2693").

Hi,
you can set multiple fields, but internally it's always stored as a float64..which take up 8 bytes,
I have a lot of boolean values/ small integer values < 255 that could fit in one byte,
would be nice if there was an option to define the size during set..
maybe some prefix b: f.e. SET FIELD b:somefield 5

Regards
Jayme

Hi, tidwall:
In order to quickly locate the data within a key, I need to know the minimum bounding rectangle of all the objects in this key. What shall I do?
R tree indexs should store this information, but I did not find such an interface.

scan FenceXY cursor 0 match 00010/13/00010002*
{"ok":true,"fields":["routeid","routesubid","stationid","ordernumber"],"objects":[{"id":"00010/13/00010002","object":{"type":"Point","coordinates":[113.223213,23.078522]},"fields":[427,840811,10003314,2]}],"count":1,"cursor":0,"elapsed":"70.87µs"}

scan FenceXY cursor 0 match 00010/13*
{"ok":true,"fields":["routeid","routesubid","stationid","ordernumber"],"objects":[],"count":0,"cursor":0,"elapsed":"75.706µs"}

why?

I've seen that you are just supporting the configuration of a listening port, but not of the host/address. If one wants to let tile38-server to just listen e.g. on 127.0.0.1, one needs another option. I recommend to apply a patch like the following to support this. Thanks.

Index: cmd/tile38-server/main.go
--- cmd/tile38-server/main.go.orig  2016-03-05 13:08:02.000000000 +0100
+++ cmd/tile38-server/main.go   2016-03-05 13:18:59.712581631 +0100
@@ -17,6 +17,7 @@

 var (
    dir         string
+   host        string
    port        int
    verbose     bool
    veryVerbose bool
@@ -25,6 +26,7 @@
 )

 func main() {
+   flag.StringVar(&host, "h", "127.0.0.1", "The listening host for communication.")
    flag.IntVar(&port, "p", 9851, "The listening port for communication.")
    flag.StringVar(&dir, "d", "data", "The data directory.")
    flag.BoolVar(&verbose, "v", false, "Enable verbose logging.")
@@ -59,7 +61,7 @@
   |_______|_______|
 `+"\n", core.Version, core.GitSHA, strconv.IntSize, runtime.GOARCH, runtime.GOOS, port, os.Getpid())

-   if err := controller.ListenAndServe(port, dir); err != nil {
+   if err := controller.ListenAndServe(host, port, dir); err != nil {
        log.Fatal(err)
    }
 }
Index: controller/controller.go
--- controller/controller.go.orig   2016-03-05 13:08:02.000000000 +0100
+++ controller/controller.go    2016-03-05 13:18:09.792727850 +0100
@@ -52,6 +52,7 @@
 // Controller is a tile38 controller
 type Controller struct {
    mu        sync.RWMutex
+   host      string
    port      int
    f         *os.File
    cols      *btree.BTree                      // use both tree and map. provide ordering.
@@ -80,9 +81,10 @@
 }

 // ListenAndServe starts a new tile38 server
-func ListenAndServe(port int, dir string) error {
+func ListenAndServe(host string, port int, dir string) error {
    log.Infof("Server started, Tile38 version %s, git %s", core.Version, core.GitSHA)
    c := &Controller{
+       host:    host,
        port:    port,
        dir:     dir,
        cols:    btree.New(16),
@@ -127,7 +129,7 @@
        }
        return nil
    }
-   return server.ListenAndServe(port, handler)
+   return server.ListenAndServe(host, port, handler)
 }

 func (c *Controller) setCol(key string, col *collection.Collection) {
Index: server/server.go
--- server/server.go.orig   2016-03-05 13:08:02.000000000 +0100
+++ server/server.go    2016-03-05 13:18:41.413563459 +0100
@@ -20,10 +20,11 @@

 // ListenAndServe starts a tile38 server at the specified address.
 func ListenAndServe(
+    host string,
    port int,
    handler func(command []byte, conn net.Conn, rd *bufio.Reader, w io.Writer, websocket bool) error,
 ) error {
-   ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
+   ln, err := net.Listen("tcp", fmt.Sprintf("%s:%d", host, port))
    if err != nil {
        return err
    }

I suggest a clear description of the tile38's default spatial data mathematical reference's parameters, so that the client will convert other ellipsoidal reference data to the default data reference system,and submit data. otherwise, it is prone to error when the lenth of arc is converted to meters.
Although this error is not great. :)

First of all, I type the following command："config+set+requirepass+1234",
Then enter the command "config+rewrite".Now the password is permanent.
Could I please without manually delete the config file, how to remove the password

When one lets tile38-server listening on not just 127.0.0.1 one needs authentication. For this tile38-server should perhaps support a config file with users/passwords pairs and tile38-cli should have options to supply username/password during connect. Optimally, the connection between tile38-cli and tile38-server should be SSL/TLS encrypted so that the password is not transmitted unprotected. Or at least a hash of the password should be transmitted only.

I use the FIELD clause in the set command.
SET name id Field routecode abc Field stationid 123456 POINT lat lon
which "routecode" value is string

NEARBY FenceXY where routecode 69030 POINT 23.135278 113.325000 100.000000
where conditions how to express equal?

I'm thinking of using tile38 as a library rather than separate process. Looks like I can just instantiate a controller/collection.Collection and go. Is that the best drop-in place? I'd basically like access to most of your features, but I don't need/want replication and I can't afford a separate daemon and the complexity that entails.

Also, with regard to my tweet: when wrapping around -180/180, at what point in the code does mapping to multiple geometries occur?

I'm using Tile38, but i have a problem.
I will try to explain it with an example application:
This is an app that let you communicate with the first person near you but this feature is a communication with the opposite sex, so for example i have a person (male) that is looking for someone (woman) and in the application i set a static geofence (i need it static) with a radius of 150m that detect if a woman is inside the area.
When a woman enters to the area the man is notified, but i have this problem:

• The man is looking for a woman in a radius of 150m, and there is no one.
• Another man, closer to the first man (10m) is also looking for a woman, but again, there is no one.
• A woman enters to the area.

In this case, both men are notified and i want that just the first (time based) man looking for a woman receives the notification and the other man needs to wait until another woman enters to the area.

How can i do it?

I was thinking severals methods to do it in the code, but in every case i have problems with two instances doing the same thing (hook receives two notification, one for every man, so the backend will do the same operation for every man) for example:

• The woman enters
• The resources get notified (two men)
• I update a field of the men, with the user id (woman) that is looking for
• I have to query all the users (man) that are looking for the same user (woman). To send the notification to the older one looking for the woman.

In this case, maybe one of the two resources that are executing the same operation in the same time, hasn't updated the field, this could be the first man, so the second man will detect there is no other man looking for that woman so will get notified, but when the resource for the first man detect the two men, the resource will also detect that this user is the older one and will get notified. So, the both men are notified.

I haven't been able to get pipelines to work (using Python Client). In #14 it appears that this should be supported?

My code looks something like

pipe = self.r.pipeline()
        for item in items:
            id = item['id']
            geojson = item['geometry']
            pipe.execute_command("set", "key", id, "object", geojson)
pipe.execute()

Tile38 reports [ERRO] Protocol error: unbalanced quotes in request

Thoughts?

Redis client is redis==2.10.5

Individual execute_commands directly against the client, "non-pipelined" work great, I just want to speed up bulk loads. Thanks

hi,tidwall
I was a little white,there is a problem need to consult you.
I want to send some instructions through the lettuce to tile38,
But I can't find the call in the lettuce source.For example,
redis.call(["set", "fleet", "truck", "point", "33.5123", "-112.2693"])
Looking forward to your reply，thank you

hi,i have two questions.
one is dose tile38 use redis to store geodata, and if does, how can i find related code in the project.
The other is how can i use redis client to insert data by batch as tile38 uses Redis Protocol.
thanks for your help!

How to use "AUTH" in the HTTP request,when I call the "CONFIG SET requirepass mypass" command,?
Can you give me an example? Thank you!

hi, tidwall

1. when using the HOOKS or KEYS command, the results can provide a number of HOOK or KEY display?
2. for what reason, HOOK collection can not be seen through the KEYS command?
3. when I set the KEY expiration date, when the KEY expires, Will affect the results of FENCE event?

thanks ：)

I have GEOJSON objects, in this format:

{"type":"MultiPolygon","coordinates":[[[[-69.47657387098774,-20.577241016355124],[-69.47743687205649,-20.730774540841214],[-69.47073936300552,-20.729771961542188],[-69.46597958706174,-20.72983009675763],[-69.45877723512393,-20.730084380669112],[-69.44686656121634,-20.730974121623348],[-69.43723371262506,-20.73045672747964],[-69.42880075385169,-20.731381525443332],[-69.41873262589603,-20.731487645905986],[-69.40542652633845,-20.731835767898335],[-69.39577493752837,-20.73263531750617],[-69.38372958341907,-20.730241017367664],[-69.36200202890991,-20.71836987920766],[-69.35253550207952,-20.714973987675318],[-69.33589939397065,-20.71372416190152],[-69.3306638946128,-20.713029501441095],[-69.3191057828514,-20.706789468081535],[-69.31293032540394,-20.70372860854936],[-69.30836410831125,-20.7031495131714],[-69.29004944359707,-20.700036843688718],[-69.27794024100051,-20.696557660221398],[-69.27767269653012,-20.696793024726926],[-69.27732856059063,-20.696990109602993],[-69.27692694816037,-20.697099186283605],[-69.27650612989353,-20.69720829285809],[-69.27612342146159,-20.697317337332464],[-69.2757982302911,-20.697372073560402],[-69.27547277143243,-20.697444881381966],[-69.27509039024032,-20.697569735853595], ...

I want to know if i can search within this MultiPolygon and what data can i store in (or with) a MultiPolygon. If it can't be done will be a nice feature.

Is there any thought going on or something I've maybe missed about writing custom outputs for hooks? As in, we can define webhooks right now, but in my application, I don't use HTTP at all, and would like to be able to call back to clients via some other protocol. To support this, there'd need to exist some generalized hook interface that could support multiple protocols. Is this something that is interesting enough to consider?

This looks like a great project and especially (dynamic) geofencing is super-interesting.

However, I didn't find any documentation on performance. To take one example of interest, if
one stored a set of 1 million moving points in tile38 representing people in a large country, and then issued a dynamic geofencing query on that set, what is the memory usage and what latency can we expect?

To answer such questions, it would be helpful to have performance (time+memory) comments per query type/operators, maybe big-O-notation hints related to the underlying algorithms, and hints on scalability, the various limits affecting it etc.

Even a subset of such documentation would be very helpful.

Hi, i'm trying to install Tile38 with no success.
I set the gopath (/opt/go) and i clone the repo in a src directory, and i run "make test" and show me errors like this:

/opt/go/src/_/opt/src/github.com/tidwall/tile38/cmd/tile38-cli (from $GOPATH)
can't load package: package _/opt/src/github.com/tidwall/tile38/cmd/tile38-server: cannot find package "_/opt/src/github.com/tidwall/tile38/cmd/tile38-server" in any of:
        /usr/local/go/src/_/opt/src/github.com/tidwall/tile38/cmd/tile38-server (from $GOROOT)
        /opt/go/src/_/opt/src/github.com/tidwall/tile38/cmd/tile38-server (from $GOPATH)
can't load package: package _/opt/src/github.com/tidwall/tile38/controller: cannot find package "_/opt/src/github.com/tidwall/tile38/controller" in any of:
        /usr/local/go/src/_/opt/src/github.com/tidwall/tile38/controller (from $GOROOT)
        /opt/go/src/_/opt/src/github.com/tidwall/tile38/controller (from $GOPATH)

Could you add an explanation step by step please? I'm new with go and I just want to test it.
Thanks u very much

Currently the pidfile must be managed outside the application. Adding an arg to the startup process will streamline this process. See #19.

Mostly due to #18 it would be great to implement both golang json interfaces (json.Unmarshaler and json.Marshaler) so a GeoJSON can be loaded directly from a file.

Also checking the code i noticed that most the work (the actual byte parsing) is already done.

I'm seeing some odd issues with long(ish) id lengths. I have some multipart IDs I'm placing inside the same key in order to do a NEARBY search. Basically its a concatenation of the type + as I want to search multiple classes of things within the same key. With longer IDs of 77 characters, I'm seeing that the records are never found with a NEARBY search.

When In invert the ID, (category at the end), which puts greater uniqueness up front, the record is found so I'm pretty sure that the issue is related to ID character length. Unrelated, I am trying to use "SEARCH" to debug but it always returns 0 count and IDs so I have to use SCAN with a high limit. This is version 1.4.1

Does tile38 use CPU's single core? The source code does not seem to set the use of multi-core, if set to multi-core, whether it can enhance the performance of it?

Hello,

this server is a piece of great work. I like it very much. Do you have any benchmarks or metric numbers we can expect? How many points / fences / notifications can be handled by one instance?

Thank you fro this great piece of software !

Maciej Bednarz, Hannover, Germany

First of all, this is a really cool project. I like that is very fast. I was doing some testing and computed distances for each of my matches returned by tile38 (point to linestring). I'm doing a nearby search with 1km and still getting some items over 74km away. Looking at the code it looked like was doing an intersection so I would of thought that it would work or is this bounding box based?

tested Tile38 1.1.4
Geojson in error -> http://bl.ocks.org/d/6d5b8a53c7a055d880f41dacf1d46297

curl --data "nearby Infrastructure/Pipelines point 33.5 36.3833 1000" 192.168.99.100:9851
{"ok":true,"objects":[{"id":"1FuHkvZ5KENLF-nV3SUxgR8FVSRJIXFcUUidKRyaM_1251_LineString","object":{"type":"LineString","coordinates":[[35.380068,33.569721],[35.841426,32.365757],[42.115382,33.606739]]}}],"count":1,"cursor":0,"elapsed":"100.212µs"}

https://github.com/avelino/awesome-go

HI,
Recently, I am using Lettuce channel model,
But tile38 seems doesn't work.I want to ask you, whether tile38 support channel model?
And then, there is a problem, Lettuce with dispitch tile38 operation, then the jedis is in what way?
I hope you can help me

Before loading the AOF file in tile38, is there a relatively conservative and simple method to evaluate how much memory tile38 will use ?

In addition, in order to reduce memory fragmentation, do you have any suggestion on the design of GeoJSON ?

hello:
I set up a nginx . I send a request to nginx reverse proxy tile38 after the visit .But This request
Does not return anything. This request is only "http://localhost:9851/server" . I have tried many ways, hope to get your help.
thanks

Hi,
I have a question about the attributes of the query.
‘GET’ is similar to the 'where' in the SQL statement?
I want to pass the ‘where’ condition to check the properties of the node containing the attribute value

How can i do reverse geocoding

This is in reference to #25

Hi tidwall
Buntdb can provide support for the transaction，it can also be achieved in the tile38 ?

There is a scene in the business, the use of real-time GPS data of the device to simulate multiple real-time access to the fence, the message can be triggered at each access（out/in）.

GPS data generated by the terminal about tens of thousands, each terminal every 10 seconds to produce a GPS message; and the fence is relatively static definition（Hundreds of thousands of）.

How to use TILE38 to achieve such a scene more concise it?

Thanks!

127.0.0.1:9851> STATS FenceXY
{"ok":true,"stats":[{"in_memory_size":6813930,"num_objects":104830,"num_points":104830}],"elapsed":"57.39µs"}

127.0.0.1:9851> STATS FenceXY # note: 2 spaces
{"ok":true,"stats":[null,{"in_memory_size":6813930,"num_objects":104830,"num_points":104830}],"elapsed":"135.822µs"}

Add the eviction commands EXPIRE, TTL, and the EX keyword to the SET command.

Examples

# Remove truck1 from fleet after 5.4 seconds.
EXPIRE fleet truck1 5.4  
# Check how many second before truck1 is removed from fleet. 
TTL fleet truck1
# Add truck1 to fleet and remove after 5.4 seconds.
SET fleet truck1 EX 5.4 POINT 33.5 -115.6

I have started a Leader and a Follower,then send command 'aof 0' to the Leader,it does not work……