Code Monkey home page Code Monkey logo

marcel's Introduction

Geophile

Geophile is a Java implementation of spatial join. Given two sets of spatial objects, R and S, the spatial join is the set of pairs, ( x, y), such that x belongs to R, y belongs to S, and x and y overlap.

This implementation is built on two abstractions:

  • Index: Any key/value data structure supporting both random and sequential access. Binary trees, skiplists, and b-trees are examples of indexes that are compatible with Geophile. Hash tables are not, because they do not provide for sequential access in key order.

  • Spatial Object: Any kind of spatial object, in any number of dimensions. The implementation must provide a method that determines the relationship of the object to a box: the box is outside the spatial object; the box is contained by the spatial object; or the box overlaps the spatial object.

These implementations can be mixed arbitrarily, as long as all the spatial objects involved are located in the same space. For example, you could compute the spatial join of a b-tree containing millions of polygons with an in-memory sorted array containing a small number of points. The output would contain the (polygon, point) pairs for which the point is inside the polygon.

Installation

Geophile can be built from source using maven:

    mvn install

This creates target/geophile-1.1.2.jar.

To create Javadoc files:

    mvn javadoc:javadoc

These files are located in target/site/apidocs.

Concepts

In addition to Index and Spatial Object, described above, Geophile relies on the following concepts.

Space

In order to use Geophile, you must describe the space in which spatial objects reside by giving the number of cells in each dimension, e.g.

    Space space = Space.newSpace(new double[]{0.0, 1_000_000.0},
                                 new double[]{0.0, 1_000_000.0}, 
                                 new int[]{10, 10});

A Space describes the space from the application's point of view. In this case, the space has two dimensions, whose coordinates go from 0 to 1,000,000 in each dimension. The last argument says that Geophile represents the space internally using a grid with 10 bits of resolution in each dimension.

The total resolution must never exceed 57. In this case, the total resolution is 20 (10 + 10).

Spatial Index

A spatial index is layered over a Space and an Index. This is the interface for adding and removing spatial objects.

Geophile works by mapping spatial objects into z-values, 64-bit integers which encode spatial information. This encoding of spatial objects as z-values, and the use of z-values as index keys is managed by the SpatialIndex implementation.

Spatial Join

Spatial join is computed by the SpatialJoin class, given two spatial indexes. Output from spatial join comprises a set of overlapping pairs of spatial objects, which can be accessed through an java.util.Iterator.

Examples

The source code for these examples can be found in src/test/java/com/geophile/z/examples. Scripts running the examples can be found in src/test/examples.

Both examples rely on the following Index and SpatialObject classes, included with the distribution:

  • com.geophile.z.index.treeindex.TreeIndex: An adapter between com.geophile.z.Index and java.util.TreeSet.

  • com.geophile.z.spatialobject.d2.Point: A 2d point.

  • com.geophile.z.spatialobject.d2.Box: A 2d box.

To run the examples, set the CLASSPATH to contain target/geophile-1.1.2.jar and target/test-classes.

Find all overalpping pairs of boxes

OverlappingPairs loads two spatial indexes, each with 100,000 tiny boxes, and then finds all the overlapping pairs.

The space is created as follows:

private static final int X = 1_000_000;
private static final int Y = 1_000_000;
private static final int X_BITS = 20;
private static final int Y_BITS = 20;
private static final int N_BOXES = 100_000;
...
private static final Space SPACE = Space.newSpace(new double[]{0, 0}, 
                                                  new double[]{X, Y}, 
                                                  new int[]{X_BITS, Y_BITS});

The spatial indexes are created and loaded as follows:

    ExampleRecord.Factory recordFactory = 
        new ExampleRecord.Factory();
    SpatialIndex<ExampleRecord> left = 
        SpatialIndex.newSpatialIndex(SPACE, new ExampleIndex());
    SpatialIndex<ExampleRecord> right = 
        SpatialIndex.newSpatialIndex(SPACE, new ExampleIndex());
    Box box;
    for (int i = 0; i < N_BOXES; i++) {
        Box leftBox = randomBox();
        Box rightBox = randomBox();
        left.add(leftBox, recordFactory.setup(leftBox, i));
        right.add(rightBox, recordFactory.setup(rightBox, i));
    }
    ...
    private static final int N_BOXES = 1_000_000;
  • ExampleRecord is a simple class encapsulating a SpatialObject and an integer identifier.

  • ExampleRecord.Factory creates ExampleRecord objects. ExampleRecord.Factory.setup(SpatialObject, int) modifies the Factory so that Factory.newRecord() calls will create an ExampleRecord with the desired SpatialObject and identifier. It also returns the Factory object itself.

  • ExampleIndex is a wrapper around TreeIndex (a TreeSet-based index) which specifies that it contains records of type ExampleRecord, and a Comparator for these records.

The spatial join output is created and scanned as follows:

    Iterator<Pair<ExampleRecord, ExampleRecord>> iterator =
        SpatialJoin.iterator(left, 
                             right, 
                             BOX_OVERLAP, 
                             SpatialJoin.Duplicates.EXCLUDE);
    while (iterator.hasNext()) {
        Pair<ExampleRecord, ExampleRecord> overlappingPair = iterator.next();
        ...
    }

To run the example (exact results will differ):

    $ src/test/examples/overlapping_pairs 
    Overlapping pairs
        (808041.0:808050.0, 309510.0:309519.0)	(808038.0:808047.0, 309505.0:309514.0)
        (912678.0:912687.0, 766340.0:766349.0)	(912673.0:912682.0, 766348.0:766357.0)

Find all the points in a given box

PointsInBox loads 100,000 points in a spatial index, and then does a spatial join to locate all the points within a query box. There is a simplified API for cases like this, in which one of the spatial join inputs is a single spatial object.

The spatial index with 1,000,000 points is created and loaded as follows:

    ExampleRecord.Factory recordFactory = 
        new ExampleRecord.Factory();
    SpatialIndex<ExampleRecord> points = 
        SpatialIndex.newSpatialIndex(SPACE, new ExampleIndex());
    for (int i = 0; i < N_POINTS; i++) {
        Point point = randomPoint();
        points.add(point, recordFactory.setup(point, i));
    }
    ...
    private static final int N_POINTS = 100_000;

The query box is created:

        Box box = randomBox();

The spatial join is done as follows:

        Iterator<ExampleRecord> iterator =
            SpatialJoin.iterator(box, 
                                 points, 
                                 BOX_CONTAINS_POINT, 
                                 SpatialJoin.Duplicates.EXCLUDE);
        // Print points contained in box
        System.out.println(String.format("Points inside %s", box));
        while (iterator.hasNext()) {
            ExampleRecord record = iterator.next();
            System.out.println(String.format("    %s", record.spatialObject()));
        }

The iterator returns spatial objects from points, the second argument to the spatial join. (There is no reason to return Pair objects, because the left part of every pair would be the same query object.)

The example runs 5 queries for the same data set. To run the example (exact results will differ):

    $ src/test/examples/points_in_box 
    Points inside (913802.0:923801.0, 454256.0:464255.0)
        (914856.0, 457038.0)
        (914733.0, 460286.0)
        (917114.0, 458769.0)
        (917084.0, 460012.0)
        (919723.0, 455706.0)
        (920704.0, 462415.0)
        (920952.0, 461800.0)
        (922320.0, 455902.0)
        (923100.0, 459578.0)
        (922532.0, 461437.0)
        (922380.0, 463585.0)
    Points inside (113254.0:123253.0, 832710.0:842709.0)
        (115443.0, 839890.0)
        (119119.0, 840991.0)
        (118426.0, 842531.0)
        (121833.0, 841634.0)
        (122308.0, 841260.0)
    Points inside (322956.0:332955.0, 898825.0:908824.0)
        (324159.0, 900132.0)
        (326677.0, 901713.0)
        (329265.0, 901457.0)
        (330675.0, 904247.0)
        (332428.0, 901001.0)
        (332606.0, 900930.0)
        (332891.0, 904158.0)
        (328246.0, 906392.0)
        (330033.0, 907402.0)
        (330479.0, 908126.0)
        (331725.0, 906492.0)
        (331660.0, 908755.0)
    Points inside (543685.0:553684.0, 619044.0:629043.0)
        (545674.0, 624489.0)
        (548309.0, 622728.0)
        (543778.0, 626348.0)
        (548836.0, 627232.0)
        (549439.0, 628060.0)
        (551537.0, 625792.0)
        (551633.0, 626626.0)
        (552856.0, 628017.0)
    Points inside (449049.0:459048.0, 238101.0:248100.0)
        (452099.0, 240735.0)
        (451890.0, 244515.0)
        (455139.0, 239030.0)
        (455306.0, 241785.0)
        (456368.0, 242123.0)
        (457867.0, 240053.0)
        (457873.0, 245254.0)

marcel's People

Contributors

da4089 avatar geophile avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

marcel's Issues

Pickle failure

So, trying to run anything in Marcel appears to cause the following error:

laptop> marcel
$ print('a')
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/main.py", line 209, in
MAIN.run()
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/main.py", line 116, in run
self.run_command(line)
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/main.py", line 137, in run_command
self.job_control.create_job(command)
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/job.py", line 235, in create_job
job = Job(self.env, command)
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/job.py", line 77, in init
self.start_process()
File "/Users/d/work/third-party/marcel-play/venv38/lib/python3.8/site-packages/marcel/job.py", line 168, in start_process
self.process.start()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Job.start_process..run_command'
laptop>

I'm not sure where to start debugging that. Any hints?

macOS 10.14.6 / Python 3.8.2

Example request: arguments

The pipeline example shows how to use data items passed by the previous part of the pipeline, but doesn't allow specifying arguments:

$ l = (|ls|)
$ l foo
Wrong number of arguments for pipeline l = ls

This makes creating custom commands limited.

Missing tags

Hello, I'm creating a package for Arch Linux AUR and I've noticed this repository is missing the Git tag for the latest version (0.18.3 as of now). These tags are handy for packagers because GitHub creates permanent URLs for them.

marcel doesn't run ipython

Screenshot from 2023-11-22 11-21-58

Tried out marcel, some of it's features I liked, some issues I think could use some work.

One issue is I could not run iPython in my shell once I invoked marcel.

Too many errors from stdin

Ubuntu: 18.04.4
Python3: 3.6.9
Marcel: 0.12.2

Starting nano is not possible (with or without file argument). -> Results in "Too many errors from stdin"
Runs in bash.

Bug: spaceful path handling

$ ls /mnt/space/games/Cryostasis\ 2.0.0.13\ [GOG]/
No qualifying paths, (possibly due to permission errors): ['/mnt/space/games/Cryostasis 2.0.0.13 [GOG]/']

Meanwhile, bash can list it just fine.

Getting the path through tab-completion does not escape the spaces, either.

Problem with executing bash commands ( bash cat $(which marcel) )

Hmmm ... why is it a problem to pass shell commands to bash executing bash -c "any command, also multi-line" and to mirror bash output to the Terminal ??? General flaw in the approach of parsing the input line ?

~ $ cat $(which marcel)
#!/bin/bash

function suppress_interruptions()
{
  return 0
}

trap suppress_interruptions SIGINT
trap suppress_interruptions SIGTSTP

python3 -m marcel.main $*
~ $ 
~ $ dash
$ bash -c "cat $(which marcel)"
#!/bin/bash

function suppress_interruptions()
{
  return 0
}

trap suppress_interruptions SIGINT
trap suppress_interruptions SIGTSTP

python3 -m marcel.main $*
$ 
/home/o $ bash cat $(which marcel)
Invalid function syntax: f'''${which marcel}'''
/home/o $ bash "cat $(which marcel)"
Error: ['/bin/bash: line 1: cat /home/o/.local/bin/marcel: No such file or directory']
/home/o $ bash -c "cat $(which marcel)"
Error: ['/bin/bash: - : invalid option']
Error: ['Usage:\t/bin/bash [GNU long option] [option] ...']
Error: ['\t/bin/bash [GNU long option] [option] script-file ...']
Error: ['GNU long options:']
Error: ['\t--debug']
Error: ['\t--debugger']
Error: ['\t--dump-po-strings']
Error: ['\t--dump-strings']
Error: ['\t--help']
Error: ['\t--init-file']
Error: ['\t--login']
Error: ['\t--noediting']
Error: ['\t--noprofile']
Error: ['\t--norc']
Error: ['\t--posix']
Error: ['\t--pretty-print']
Error: ['\t--rcfile']
Error: ['\t--restricted']
Error: ['\t--verbose']
Error: ['\t--version']
Error: ['Shell options:']
Error: ['\t-ilrsD or -c command or -O shopt_option\t\t(invocation only)']
Error: ['\t-abefhkmnptuvxBCHP or -o option']
/home/o $ 

Needs Nix(OS) package

AFAICT there's currently no package for this lovely little shell in Nix (and therefore, NixOS.)

Generally making .nix packages from Python packages is fairly trivial, so I'm going to try later this week.

If anyone wants to beat me to it, go ahead. :D

The bash command crashes the shell

/home/o $ bash
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/o/.local/lib/python3.10/site-packages/marcel/main.py", line 314, in <module>
    main()
  File "/home/o/.local/lib/python3.10/site-packages/marcel/main.py", line 297, in main
    MAIN.run(print_prompt)
  File "/home/o/.local/lib/python3.10/site-packages/marcel/main.py", line 131, in run
    self.run_command(self.input)
  File "/home/o/.local/lib/python3.10/site-packages/marcel/main.py", line 143, in run_command
    pipeline = parser.parse()
  File "/home/o/.local/lib/python3.10/site-packages/marcel/parser.py", line 948, in parse
    return self.command()
  File "/home/o/.local/lib/python3.10/site-packages/marcel/parser.py", line 954, in command
    command = self.pipeline()
  File "/home/o/.local/lib/python3.10/site-packages/marcel/parser.py", line 1034, in pipeline
    pipeline.append(self.create_op(*op_args))
  File "/home/o/.local/lib/python3.10/site-packages/marcel/parser.py", line 1149, in create_op
    op = self.create_op_builtin(op_token, arg_tokens)
  File "/home/o/.local/lib/python3.10/site-packages/marcel/parser.py", line 1182, in create_op_builtin
    op_module.args_parser().parse(args, op)
  File "/home/o/.local/lib/python3.10/site-packages/marcel/op/bash.py", line 62, in parse
    if args[0] in ('-i', '--interactive'):
IndexError: list index out of range
~ $ 

Now after the IndexError the shell is back to bash. How to exit the marcel shell gracefully? exit or quit do not work ...

New behavior for existing standard-shell commands?

I am exploring marcel and I see that existing commands have different behavior in marcel.
Most of us have ingrained muscle memory with regular shell commands (like I always do "ls -ltr").

Is this by design or are these bugs that need to be fixed?

for e.g.

(with bash)

(venv) bhakta@blr:~/dev/marcel$ ls -fr
.  ..  .git  .gitignore  .idea  COPYING  LICENSE  README.md  bin  experiments  marcel  notes  setup.py  test  venv
(venv) bhakta@blr:~/dev/marcel$ ./bin/marcel

(with marcel)


/home/bhakta/dev/marcel> ls -fr | less
Missing filename ("less --help" for help)

/home/bhakta/dev/marcel>
/home/bhakta/dev/marcel> ls -fr > x
/home/bhakta/dev/marcel> vim x

<hangs - I had to ctrl-c>
^C

Ctrl-c did not come out, I had to kill the window


/home/bhakta/dev/marcel> ls -ltr
Operator ls: Unknown flag -ltr
/home/bhakta/dev/marcel>

Missing GitHub releases

Hello, I've noticed the last few releases (tags) are missing their corresponding GitHub releases. These are useful because I can then create a custom Watch on the repository and get notified when a new release is available (I package marcel for Arch Linux AUR). No need to create them retroactively, just for the latest one and then moving forward. Thank you!

Problem with `bash ls -i` ( -l option is OK, -i option fails )

I am currently working towards a concept of user interaction with the system ( https://github.com/oOosys/oOo ) which seems to share some of main thoughts with the marcel approach to shell scripting.

Reading about marcel shell I had thought I have understood the approach the right way, but it seems I haven't:

~ $ echo $0
bash
~ $ ls -i *.txt
9311566 tmp.txt
/home/o $ bash ls -l *.txt
-rw-rw-r-- 2 o o 101 2023-09-16 18:49 tmp.txt
/home/o $ bash ls -i *.txt
Operator bash: Flags must all appear before the first anonymous arg
/home/o $ version
0.18.4
/home/o $ ls *.txt
-rw-rw-r--   o      o           101   2023 Sep 16 18:49:37 tmp.txt
/home/o $ ls -i *.txt
Operator ls: Unknown flag -i
/home/o $

I have thought that the bash operator will take the shell command as it is and pass it to the bash shell ...

Any hints towards better understanding why I have failed to obtain the inode value for the file? Or have I run into a bug?

Claudio ( https://github.com/oOosys/oOo )

Community ?

Hi,

Thanks for marcel, though I think it'd deserve some more acknoledgment
Maybe you should have an irc/matrix/discord/* chan somewhere ?
have a nice day

marcel startup fails

On Ubuntu 22.04 with Python 3.11.6 in a virtual environment:

python3 -mvenv venv
. ./venv/bin/activate
pip install marcel
$ marcel

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 317, in <module>
    main()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 293, in main
    MAIN = Main(None, same_process=False, old_namespace=old_namespace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 107, in __init__
    self.initialize_reader()  # Sets self.reader
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 177, in initialize_reader
    self.reader = Reader(self.env, self.env.locations.history_path())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 58, in __init__
    super().__init__(history_file=history_file)
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 46, in __init__
    self._fix_history()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 115, in _fix_history
    readline.read_history_file(self.history_file)
OSError: [Errno 22] Invalid argument

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.