To interface with mapget, clients should use the mapget::cache API (in remote mode or as a C++ library). As a first step, we will facilitate a basic API which supports only the Cache::tiles() function (or the /GET tiles REST API). From the following UML, the Application, Cache, DataSourceConnection and Configuration classes will be implemented.

Currently, starting mapget with custom data sources requires lengthy command line invocations. CLI11, the command line argument parser used by mapget, can be used to parse config files containing the arguments as documented here.

The goal is for mapget to support reading arguments from a configuration file in addition to the command line, and to document this option. Existing integration tests triggered by CLI arguments should be extended to include one example config file.

mapget::Service::request currently allows basic queries based on mapId, layerId, and tiles. This issue aims to extend the existing API to include basic feature filtering capabilities. Implementing this enhancement will allow users to test the function's accuracy and scalability more efficiently. Special attention will be given to meeting general scalability requirements.

Extended 'tiles' Endpoint Example

// Extract from mapget's updated README
// Fetch streamed features based on strict constraints.
// Support for Accept-Encoding text/jsonl or application/binary

+ POST /tiles(list<{
    mapId: string,
    layerId: string,
    tileIds: list<TileId>,
    maxKnownFieldIds* 
  }>, filter: optional<string>): 
  bytes<TileLayerStream>

Acceptance Criteria:

• Implement filtering in both local and Http mapget services
• Automated tests for both services, with:
  • Customizable request options
  • Metrics on time spent fetching and filtering data
• Performance regression detection using standardized tests on defined hardware/target

Open Questions:

• How to restrict the filter language in this step?
• Is it also intended to use this functionality for functions like 'viewport-based' filtering?

Background:

The existing in-memory cache in mapget serves its purpose for short-lived sessions but poses limitations for long-term or recurring data retrieval needs. This transient nature of the cache can lead to repetitive data fetching from source, negatively impacting performance and user experience. Implementing a persistent cache would alleviate this issue by storing data between sessions, saving time and computational resources.

Goal:

Introduce a persistent caching mechanism in mapget to improve long-term data retrieval efficiency and to complement the existing in-memory cache.

Acceptance Criteria:

1. Persistence: The cache should be capable of storing data between different sessions and system reboots.

2. Compatibility: The persistent cache should function alongside the existing in-memory cache, allowing users the flexibility to choose between the two depending on their needs.

3. Cache Flushing: Users should have the ability to manually clear the persistent cache when needed.

4. Future-Proofing: Like the in-memory cache, the implementation should be designed to be compatible with other caching mechanisms that may be introduced later.

5. Cache configuration: Users can specify the max number of tiles, path to cache-file and the type of used cache (memory, persistent)

6. Cache statistics: The persitent cache tracks number of hits and misses. It's sufficient to get this information via logs.

A basic mapget DataSource abstraction should be implemented, which allows implementing a feature data provider.

For now, only the /tile- and /metadata-endpoints shall be supported. On startup, a DataSource shall automatically select a port to bind to, if no port is explicitely provided. The port shall be written in a message to stdout, which will allow a management application to register a newly launched data source with a mapget server.

Currently, all workers for a RemoteDataSource share the same httplib Client. However, it is not possible to run parallel requests through one socket, so the parallelization is effectively disabled.

The system should provide improved diagnostic capabilities that allow for tracking and displaying various statistics about the data from each active map data source. The diagnostic data should include, but not be limited to, the number of map features obtained from each data source and the associated data sizes.

This feature requires extending the mapget component to collate and store these statistics as part of its data aggregation and caching processes. Furthermore, the map data sources should be enhanced to provide diagnostic data to the mapget component during the data retrieval and conversion processes.

The diagnostic data should be accessible through the mapget's query interface. This way, users can get insights into the data's origins, the volume of data sourced from each source, and the associated data size, thus improving transparency and troubleshooting capabilities.

Both to transfer data from sources to the mapget cache, and to receive tile data from the mapget cache, the TileLayerStream protocol is required. It will be implemented in two classes:

• The TileLayerStreamReader is constructed with a callback for onParsedLayer. It has pushBytes and numUnparsedBytes methods, which may be used to insert binary data to parse, and to check whether there is any unparsed data.
• The TileLayerStreamWriter is constructed with a callback for onWrite, which may be called when something is ready to be sent over the wire, and onGetFieldCacheOffset, which may be called when the stream must decide how much of the field cache must be re-sent. It has a pushFeatureLayer method, which is used to insert a (Tile-)FeatureLayer into the stream.

tiny-process-library (gitlab link) was selected for the current DataSourceProcess implementation. However, it is badly documented. We should find an alternative or a good reason to keep it.

When there are no datasources connected, the mapget service should still answer requests from cache.

Problem: mapget currently does not cache DataSourceInfo.

The objective of this issue is to implement the '/locate' endpoint. This endpoint is crucial for enabling erdblick to obtain a list of tile-layer combinations that specify the the TileFeatureLayer which contains the feature satisfying given ID field constraints.

Endpoint Functionality:

• The mapget HTTP service '/locate' endpoint should accept feature typeId and a map of string to scalar values (ID parts) as input parameters. It must return a list of [MapTileKey,FeatureId] pairs.
• The mapget base service must implement a locate-function, creating requests to all connected datasources without cache checking for now, and return a vector<pair<MapTileKey,string>> if a match is found.
• The DataSource HTTP server interface has been extended to cover the /locate endpoint analogously to mapget.
• Example data sources demonstrate how to implement this endpoint.
• The mapget service interface documentation covers the implementation.

The mapget base service evaluates the connected data sources and raises the errors:
- [ ] 'No data source provides specified feature type' if no data source can provide the feature type ID.
- [ ] 'Unknown ID (parts)' when provided ID parts are not applicable to any known feature type with the provided ID.
- [ ] 'Missing non-optional ID parts.'

(advanced error detection and reporting left for future refinement)

Hints/Resources:

• Have a look at the C++ codebase, particularly focusing on the IdPart and FeatureTypeInfo structures.

The most recent RocksDB version on conan.io is 6.29.5 (current on github is 8.6.7). Let's wait for conan.io to catch up (or open a PR) and pull RocksDB via conan.

Pull RocksDB from conan.io.
See https://conan.io/center/recipes/rocksdb?version=8.8.1

When a fetch request is received for a map+layer combination for which no connected datasource can provide the response, a 400 response should be given with a descriptive error message.

At the moment mapget uses a mixture of CMake's FetchContent* and git submodules (./cmake/cmake_modules) - using conan would (potentially) allow reducing build times. For example the introduction of rocksdb as a dependency increased build times by up to ~6x (for example 10min vs 58min ). If some aspects cannot be covered by conan (not even with a custom recipe), using CMake's FetchContent should be still fine.

Background:
While the current lazy-loading approach to cache filling is effective in many situations, it falls short when dealing with multiple or slow data sources. Users often know the specific spatial extent they wish to analyze and would benefit from a cache prefilling phase, even if it takes time to complete. This also allows developers to initially focus on optimizing the interaction between mapget and client applications like a map viewer, before turning their attention to optimizing the entire data pipeline, including data source interactions.

Goal:
Implement cache prefilling in mapget to enhance user experience by making data rapidly available post-prefill.

Acceptance Criteria:

1. Spatial Extend Support: mapget should support 'populate' requests that specify a spatial extent for prefilling.
2. Cancellation: Users should be able to cancel populate requests if needed.
3. Configurable In-Memory Cache: The existing in-memory cache should be easily configurable in terms of its capacity, allowing users to tailor it to the volume of data they intend to explore.
4. Progress Indication: A way to track the prefilling phase's progress is essential, ideally with an estimated time-to-completion indicator.
5. Future-Proofing: The implementation should be designed to be compatible with other caching mechanisms that may be introduced later.

To facilitate integration of mapget in various upstream and downstream projects, and initial subset of the mapget::model library must be implemented. This should include all classes in the following UML, except for the TileLayerStream class.

Currently, the client relies on the server to close the connection as a signal that a /tiles request was fully processed. However, the Connection: close header is prohibited in modern browsers. So cpp-httplib waits for the response to time out until it closes the request. By sending EndOfStream, the client can now close the connection once it is received.

For some use-cases, it is necessary that a datasource can annotate features of another datasource with additional attributes, relations, or geometry. For this purpose, the introduction of an add-on: true field in the datasource info is planned. Any such connected datasource will always be called to produce add-on feature data for each requested tile. Mapget will first request the tile from a non-add-on source, then request data from each add-on source, and then merge the retreived set.

Currently, Python bindings for the /locate-request callback and the corresponding LocateResponse/LocateRequest structs are missing for the DataSourceServer class.

The mapget model implementation currently creates features from a tile without validating if the feature ID fields are set. This will lead to problems with implementing e.g. jump-to-feature down the line, where a feature reference must contain all the information to be resolved to a feature.

To fix the issue, the mapget model should check that the ID fields are set, before adding a feature or feature reference to the feature layer.

The current implementation of our Feature Model API permits basic linking of map features by utilizing their IDs as attributes. This method is effective for simple navigational tasks, allowing users to transition between related features with ease. However, this approach is somewhat limited, as it only facilitates basic ID-based connections without considering the complex relationships that can exist between map features.

We need to enhance the Feature Model API to encompass a broader range of feature relations. The revised API should enable defining relations that go beyond mere ID linking, allowing for associations based on specific segments or characteristics of other features. For instance, it should support scenarios like "Feature A is related to Feature B (e.g. visible from), but this relation is valid only within a particular section of Feature B's geometry."

Some additional thoughts/ideas:

How would we model "Feature A, a public park, is related to Feature B, a residential area, because they share a characteristic of being in a 'low-pollution zone.'” - should it be covered by relations too? The key difference is to the previous example is that this about grouping.

Sometimes the LOCK file stays when I stop the source (with SIGTERM), either we avoid this kind of locking behavior or we ensure that it gets properly reset on-demand (at every startup as there should be only one user of the cache). At the moment I cannot share any way of how to reproduce this behavior as soon as I have the needed details I will share them in the scope of this issue.

When the mapget process is killed with the SIGKILL, the datasource processes spun up by it continue running.

There should be a POST /heartbeat request from mapget to datasources. Datasource subprocesses read the MAPGET_DATASOURCE_PROCESS_TIMEOUT environment variable, which indicates the timeout window in seconds. If this variable exists, the process shuts down when a heartbeat has not been received from mapget in the specified time window.

We should add a --datasource-executable-timeout command line argument to mapget.

The datasources should reply to the /heartbeat request with their unique datasource nodeID. In the future, mapget should use the keep-alive reply to:

1. Check if a datasource process is still alive. If not, mapget reports it and removes worker threads for the datasource.
2. Validate that the datasource at the given endpoint has the same ID as the originally registered datasource.

A similar approach is used in zswag, which can be taken as inspiration. Specifically, the header should be adapted. Then all throws and cout/cerr calls should be changed.

This issue reflects the following additional requirements towards attribute/relation validites:

• It shall be possible to specify more than one validity extent for each attribute.
• It shall be possible to specify a validity stretch indirectly using (fractional) lengths or positions as start/end bounds for the stretch. This will allow add-on datasources to specify validity stretches without needing access to the feature's base geometry.

Use pybind11 to create the mapget python package, which should initially include only bindings for the mapget-model and mapget-datasource libraries.