Comments (1)
Python's cProfile module was used to profile an ingestion run.
A faster JSON decoder may help.
>>> p.sort_stats('time').print_stats(25)
Wed Apr 20 22:52:58 2011 prof.1
606063949 function calls (538093040 primitive calls) in 2355.761 CPU seconds
Ordered by: internal time
List reduced from 569 to 25 due to restriction <25>
ncalls tottime percall cumtime percall filename:lineno(function)
2894113/2423987 293.772 0.000 1037.676 0.000 /usr/lib/python2.6/json/decoder.py:162(JSONObject)
62802266/179064 252.699 0.000 1249.725 0.007 /usr/lib/python2.6/json/scanner.py:38(iterscan)
1841908 236.037 0.000 236.037 0.000 {built-in method acquire}
89532 191.165 0.002 191.165 0.002 {method 'sendall' of '_socket.socket' objects}
186671 100.853 0.001 100.853 0.001 {method 'recv' of '_socket.socket' objects}
33943035 86.064 0.000 86.064 0.000 {_json.scanstring}
4965746/89532 84.138 0.000 1247.784 0.014 /usr/lib/python2.6/json/decoder.py:206(JSONArray)
12299462 83.452 0.000 131.961 0.000 /usr/lib/python2.6/json/decoder.py:60(JSONNumber)
2423987 81.286 0.000 81.286 0.000 ./apiserver/osmelement.py:138(from_mapping)
897109 73.387 0.000 143.122 0.000 ./dbmgr/dbm_input.py:39(_make_osm_iterator)
128843635 67.806 0.000 67.806 0.000 {built-in method end}
50502797 52.652 0.000 52.652 0.000 {built-in method scanner}
897497 50.951 0.000 94.286 0.000 ./dbmgr/dbm_stats.py:98(increment_stats)
153426 50.850 0.000 212.364 0.001 ./datastore/lrucache.py:258(insert_slab)
31401133 46.769 0.000 46.769 0.000 {built-in method match}
11241805 45.127 0.000 91.058 0.000 /usr/lib/python2.6/json/decoder.py:152(JSONString)
3330996 32.307 0.000 76.072 0.000 ./apiserver/osmelement.py:116(__init__)
153426 32.257 0.000 48.994 0.000 ./datastore/lrucache.py:233(_remove_slab_items)
387550 27.715 0.000 27.742 0.000 {map}
4443187 27.343 0.000 27.343 0.000 ./datastore/slabutil.py:35(_make_numeric_slabkey)
43465635 24.734 0.000 24.734 0.000 {getattr}
632818 24.690 0.000 91.443 0.000 /usr/lib/python2.6/threading.py:116(acquire)
43700595 24.624 0.000 24.624 0.000 {built-in method span}
34895320/34895075 17.521 0.000 17.521 0.000 {len}
897108 17.395 0.000 2285.756 0.003 ./dbmgr/dbm_ops.py:50(add_element)
from mapquest-osm-server.
Related Issues (20)
- Investigate better ways to access a Membase backend
- Create a tool for verifying the contents of the data store
- Extend the /capabilities API to indicate an upstream server
- Add support for ingesting osmChange documents
- Allow ingestion to be restricted to a given geographical region
- Investigate alternate geo-grouping and storage schemes for OSM data
- Move project documentation to the main source tree.
- Need a tool that can extract a geographical subset of a 'full' planet.
- Retrieve slab configuration information from the data store
- Create an installation guide
- Investigate support for queries using XAPI-style predicates
- Implement cache invalidation on the front-end
- Provide APIs to query meta-data about the contents of the data store
- Support "full" planet dumps HOT 1
- Create an installable package using distutils
- Support a riak based data store
- Redirect write requests to an upstream server
- Provide a way for restarting interrupted ingestion runs
- Store map data in the data store in a compact, encoded form HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mapquest-osm-server.