pyfa-org / phobos Goto Github PK
View Code? Open in Web Editor NEWScript for getting full data dump from EVE Online client
License: GNU General Public License v3.0
phobos's People
Contributors
Stargazers
Watchers
phobos's Issues
Missing dogma data?
Hi there,
I've been trying to use this tool for fetching all of the Dogma related data but there appears to be files missing when I run the command. I believe there should be dogmaexpressions and dogmaoperands files which are missing entirely, and probably related is the dogmaeffects which is missing the preExpression/postExpression fields.
Is this data available to be pulled from the client?
Appreciate any help, thanks.
Just for reference this is the command I've been running, nothing special:
python run.py --eve="C:\EVE" --server="tq" --json="D:\phobos_output"dungeons, mission data not processed
eve-21.05.2291839
processing dungeons
unable to fetch data - IOError: Unexpected cFSD schema version. Loader schema: (5508473695024844245, 6829693746456926295) Header schema field: (631582858141601371, 3392568259313855816)
processing missions
unable to fetch data - IOError: Unexpected cFSD schema version. Loader schema: (7061632699330772098, 11478092164589306168) Header schema field: (17443728786762165585, 12624431704568200047)
example says --output, needs to be --json
Just a minor issue, really. The example in the readme says to use --output with dumpToJson.py. Should say --json
Cannot dump any table containing localized data
After the Scylla update:
Without -t argument:
processing invtypes
failed to fetch data - ContainerNameError: container "res/localization/localization_main" is not available for miner StuffedPickleMiner
With -t en-us:
processing billtypes
failed to fetch data - LanguageNotAvailable: data for language "en-us" cannot be loaded
Implement FSDLite staticdata support
Sisi client types/groups/categories data has been moved to bin/staticdata/*.db files, we have to implement support for these.
There're actually some files already.
DBs contain two tables, cache and indexes, actual data is in cache. Cache table contains 3 columns, key (for evetypes it's typeID of item), value (actual data row in JSON) and time.
I think it's best to dump just values into raw list form - so it will be like old invtypes table, just with new name.
Can't dump Singularity database
This is happening for me, with Singularity build 595908 and using the singularity branch.
python2 dumpToJson.py -e ~/.eve/clients/singularity -c ~/.eve/settings/singularity/0/cache -s Singularity -j ~/Documents/EVE/json
failed to process schematics
failed to process ramcompletedstatuses
failed to process invtypematerials
failed to process ramaltypesdetailpergroup
failed to process constellations
failed to process schematicsByPin
failed to process invgroups
processing sounds
failed to process sounds
failed to process ramaltypesdetailpercategory
failed to process eveowners
failed to process shiptypes
failed to process ramaltypes
failed to process groupsByCategories
processing icons
failed to process icons
failed to process invcontrabandTypesByFaction
failed to process solarsystems
failed to process schematicstypemap
failed to process regions
failed to process invreactiontypes
failed to process evelocations
failed to process invmetatypesByTypeID
failed to process certificates
failed to process schematicsByType
failed to process typesByMarketGroups
failed to process factions
failed to process schematicspinmap
failed to process typesByGroups
failed to process invcategories
failed to process corptickernames
failed to process ramtyperequirements
failed to process dgmeffects
failed to process fsdTypeOverrides
failed to process invmetagroups
failed to process invmetatypes
failed to process dgmexpressions
failed to process staoperationtypes
failed to process npccorporations
failed to process dgmunits
failed to process nebulas
failed to process ramactivities
processing graphics
failed to process graphics
failed to process invtypes
failed to process fsdPlanetAttributes
failed to process dgmtypeattribs
failed to process billtypes
failed to process invbptypes
failed to process mapcelestialdescriptions
failed to process invcontrabandTypesByType
failed to process dgmattribs
failed to process locationwormholeclasses
failed to process stations
failed to process certificaterelationships
failed to process dgmtypeeffects
Traceback (most recent call last):
File "dumpToJson.py", line 109, in <module>
for service, call in discoverSvc(eve):
File "/home/romain/Projects/phobos/phobos/remoteSvc.py", line 33, in discover
info, data = blue.marshal.Load(_readfile(filename))
UnmarshalError: find_global failed to resolve: carbon.common.script.net.objectCaching.CachedMethodCallResult
Exception TypeError: 'expected string or Unicode object, NoneType found' in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Adapt phobos to changes in data layout on singularity
- Update itemdiff script to work with new evetypes, evegroups etc tables
2) Update how phbtraits is generated - there's new db called infoBubbles.db, probably contains data from fsdTypeOverrides - Update how itemdiff fetches/processes info about materials - it's been buggy for a while, and now data about that is centralized in blueprints.db
Logistic Frigates and Command Destroyer traits missing
The generated phbtraits.json doesn't contain traits data for the (relatively) new Logistic Frigates and Command Destroyers.
(Still, phbtraits has been more than helpful for me :) )
Support for FSDLiteStorage is missing
processing blueprints
failed to fetch data - UnknownContainerTypeError: unable to route <class 'reverence.config.FSDLiteStorage'>
According to reverence changelog, it's JSON in sqlite
RuntimeError: Specified protocol version (448) not found
The full error row is:
RuntimeError: Specified protocol version (448) not found in C:\Users\Willy\AppData\Local\CCP\EVE\c_program_files_(x86)_steam_steamapps_common_eve_online_sharedcache_tq_tranquility\cache\MachoNet\87.237.38.200
Any clue?
fsd unable to fetch data
eve build (sisi) 19.10.1955063
python run.py --eve="C:\Program Files (x86)\Steam\steamapps\common\Eve Online" --json=C:\Phobos\export\eve-19.10.1955063 --server=sisi
processing metadata
Miner FsdLiteMiner:
processing blueprints
processing clonegrades
processing dbuffcollections
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing evecategories
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing evegroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing evetypes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing fighterabilities
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing fighterabilitiesbytype
processing industry_activities
processing industry_activity_modifier_sources
processing industry_activity_target_filters
processing industry_assembly_lines
processing industry_installation_types
processing infobubbles
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing skinlicenses
processing skinmaterials
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing skins
Miner FsdBinaryMiner:
processing accountingentrytypes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing accountingkeys
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing achievementevents
processing activitynodes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing agencyhelpvideogroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing agencyhelpvideos
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing agentsinspace
processing ancestries
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing atoms
processing billboards
processing billtypes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing bloodlines
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing careerpaths
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing challengeconditions
processing challengeevents
processing challenges
processing character_colorlocations
processing character_colornames
processing character_modifierlocations
processing character_portraitresources
processing character_resources
processing character_sculptinglocations
processing checkpoints
processing commandsets
processing contrabandtypes
processing controltowerresources
processing conversationagents
processing conversations
processing corporationactivities
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing corporationrolegroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing corporationroles
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing crates
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing currencybuckets
processing dailychallengecampaigns
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing dailychallengepools
processing dailychallengerewards
processing dailychallenges
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing defaultoverviews
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing dogmaattributecategories
processing dogmaattributes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing dogmaeffects
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing dogmaunits
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing dynamicitemattributes
processing effects
processing effectsequences
processing epicarcs
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing essreservekeys
processing expertsystems
processing explosionbucketids
processing explosionids
processing factions
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing globaltypes
processing graphicids
processing graphiclocations
processing graphicmaterialsets
processing groupgraphics
processing grouprewards
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing iconids
processing marketgroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing menuhighlights
processing metagroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing musictriggers
processing newactivities
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing newfeatures
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing nodegraphs
processing nodetypes
processing npccharacters
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing npccorporationdivisions
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing npccorporations
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing operationcategories
processing operations
processing operationsites
processing operationtasks
processing operationtreestructures
processing petitionevents
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing petitionproperties
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing pvpfilamenteventdates
processing pvpfilamentmatchtypes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing races
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing recruitmentgroups
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing recruitmenttypes
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing requiredskillsfortypes
processing requiredskillsindex
processing schematics
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing schoolmap
processing schools
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing seasonalcampaignsloginrewards
processing seasonselection
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing skillplans
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing skilltreegroups
processing skyboxeffects
processing soundids
processing spacecomponentsbytype
processing spaceobjecthighlights
processing stationoperations
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing stationservices
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing stationstandingsrestrictions
processing surveys
processing tags
processing traveleffects
processing typedogma
processing typelist
processing typematerials
processing uielementtypes
processing uihelppointers
processing uihidingtemplates
processing uihighlights
Miner TraitMiner:
processing traits
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
Miner SqliteMiner:
processing app:/bin64/staticdata/mapObjects_celestials
unable to fetch data - ContainerNameError: container "res:/localization/localization_main" is not available for miner PickleMiner
processing app:/bin64/staticdata/mapObjects_npcStations
Miner PickleMiner:
processing app:/res/planetResources
unable to write data with JsonWriter - UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 7: ordinal not in range(128)
processing res:/graphics/character/paperdollcache
processing res:/localizationfsd/localization_fsd_de
processing res:/localizationfsd/localization_fsd_en-us
processing res:/localizationfsd/localization_fsd_es
processing res:/localizationfsd/localization_fsd_fr
processing res:/localizationfsd/localization_fsd_it
processing res:/localizationfsd/localization_fsd_ja
processing res:/localizationfsd/localization_fsd_ko
processing res:/localizationfsd/localization_fsd_main
processing res:/localizationfsd/localization_fsd_ru
processing res:/localizationfsd/localization_fsd_zh
processing res:/staticdata/asteroidtypesbysolarsystemid
processing res:/staticdata/blueprintsbymaterialtypeids
processing res:/staticdata/starmapcache
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.