def check(self,manager,manager1,listName,debugLimit):
        self.dumpListOfDicts(manager.__dict__[listName], debugLimit)
        self.dumpListOfDicts(manager1.__dict__[listName], debugLimit)
        self.assertEqual(manager.__dict__,manager1.__dict__)    

should work as well as

 def testRoyals(self):
        '''
        test Royals example
        '''
        royals1=Royals(load=True)
        self.assertEqual(4,len(royals1.royals))
        json=royals1.toJSON()
        print(json)
        types=Types.forClass(royals1, "royals")
        royals2=Royals()
        royals2.fromJson(json,types=types)
        self.assertEqual(4,len(royals2.royals))
        print(royals1.royals)
        print(royals2.royals)
        self.assertEqual(royals1.royals,royals2.royals)

the error message has a syntax error and fails and therefore does not display correctly.

needed e.g. to get access to SqlTable Info for UML diagram creation

fix API to:

def restoreFromJsonFile(self,jsonFile:str):
def restoreFromJsonStr(self,jsonStr:str):

``python
def restoreFromJsonFile(self,jsonFile:str)->list:
def restoreFromJsonStr(self,jsonStr:str)->list:

basically the storage package https://github.com/WolfgangFahl/ProceedingsTitleParser/tree/master/storage needs to move here

pydantic use will allow to use Json Schema and OpenAPI approaches.

add JSONAble serialization support

e.g. for http://wiki.bitplan.com/index.php/Dblpconf#Entities

The DDL For the view Record should be created as

CREATE VIEW Record AS
  SELECT KEY,MDATE,YEAR FROM article
UNION
  SELECT KEY,MDATE,YEAR FROM book
...

add to setup.py

e.g. when an invalid attribute name like py/object is being used

will simplify handling of table information

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145425674240 and this is thread id 123145442463744. 1

see https://stackoverflow.com/questions/48218065/programmingerror-sqlite-objects-created-in-a-thread-can-only-be-used-in-that-sa

https://stackoverflow.com/a/48234567/1497139

conn = sqlite3.connect('your.db', check_same_thread=False)

UnboundLocalError: local variable 'lookupResult' referenced before assignment

allow to return a dict/map by a given column of a list of Entities

use code from https://github.com/WolfgangFahl/DgraphAndWeaviateTest/blob/master/storage/sparql.py as a basis

allow LOD<->pandas data frame conversion

tabulate has nice functions that fit the pyLODStorage approach e.g. wikidata and latex table creation.
Refactor to use that library.

see e.g. https://stackoverflow.com/questions/32355685/join-a-list-of-strings-such-that-each-string-is-within-quotes-and-comma-separate so that SQL injection can be avoided when having dynamic queries.

There should be not exception but only a warning if debug is active.

with append=True it should be possible to append data to an existing table - the internal API has withDrop and withCreate for the SQLDB case.

raise Exception("Need %d sample records to createTable" % (sampleRecordCount))
Exception: Need 10000000 sample records to createTable

does not make sense - especially if number of records available is not shown

cacheDir in StorageConfig is bound to $HOME see

For testing purposes it would be great if another location for the cacheDir could be defined e.g. the temp folder
Possible use case

see also

• Latex example
• Mediawiki example

Examples:

15 Random substances with CAS number
Wikidata SPARQL query showing the 15 random chemical substances with their CAS Number
query

# List of 15 random chemical components with CAS-Number, formula and structure
# see also https://github.com/WolfgangFahl/pyLoDStorage/issues/46
# WF 2021-08-23
SELECT ?substance ?substanceLabel ?formula ?structure ?CAS
WHERE { 
  ?substance wdt:P31 wd:Q11173.
  ?substance wdt:P231 ?CAS.
  ?substance wdt:P274 ?formula.
  ?substance wdt:P117  ?structure.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
LIMIT 15

try it!
result

Ten largest cities of the world
Wikidata SPARQL query showing the 10 most populated cities of the world using the million city class Q1637706 for selection
query

# Ten Largest cities of the world 
# WF 2021-08-23
# see also http://wiki.bitplan.com/index.php/PyLoDStorage#Examples
SELECT DISTINCT ?city ?cityLabel ?population ?country ?countryLabel 
WHERE {
  VALUES ?cityClass { wd:Q1637706}.
  ?city wdt:P31 ?cityClass .
  ?city wdt:P1082 ?population .
  ?city wdt:P17 ?country .
  SERVICE wikibase:label {
    bd:serviceParam wikibase:language "en" .
  }
}
ORDER BY DESC(?population)
LIMIT 10

try it!
result

count OpenStreetMap place type instances
This SPARQL query
determines the number of instances available in the OpenStreetMap for the placeTypes city,town and village

query

# count osm place type instances
# WF 2021-08-23
SELECT (count(?instance) as ?count) ?placeType ?placeTypeLabel
WHERE { 
  VALUES ?placeType {
    "city"
    "town"
    "village"
  }
  ?instance osmt:place ?placeType
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
GROUP BY ?placeType ?placeTypeLabel
ORDER BY ?count

try it!
result

LOD containing a list as value leads to an error

Option to exclude lists of convert them to a single string to avoid the failure would be good

@Property handling won't work if not done this way.

Example:
CSV input:

pageTitle,name,label
page_1,Test Page 1,1
page_2,Test Page 2,2

LoD output:

[
  {"pageTitle": "page_1", "name": "Test Page 1", "label": "1"},
  {"pageTitle": "page_2", "name": "Test Page 2", "label": "2"}
]

Currently unsure where the type conversion should happen.
The LoD can be converted to JSONAble so the Type conversion could be done there.

restore functionality of 0.0.26 release

def storeToJsonFile(self,storeFilePrefix,tableName):
        '''
        store me with the given storeFilePrefix
        
        Args:
            storeFilePrefix(string): the prefix for the JSON file name
            tableName(string): the name of the attribute for which to store the type information
        '''
        JSONAble.storeJsonToFile(self.toJSON(), "%s.json" % storeFilePrefix)
        types=Types.forTable(self, tableName)
        JSONAble.storeJsonToFile(types.toJSON(), "%s-types.json" % storeFilePrefix)
       
 def restoreFromJsonFile(self,storeFilePrefix):
        '''
        restore me from the given storeFilePrefix
        
        Args:
            storeFilePrefix(string): the prefix for the JSON file name
        '''
        jsonStr=JSONAble.readJsonFromFile("%s.json" % storeFilePrefix)
        typesJson=JSONAble.readJsonFromFile("%s-types.json" % storeFilePrefix)
        types=Types(type(self).__name__)
        types.fromJson(typesJson)
        self.fromJson(jsonStr, types)

take the Server Example:

class Server(JSONAble):

to improve SQL table datatype handling if table is sparse

using the endpoint https://query.wikidata.org/sparql leads to a urllib.error.HTTPError: HTTP Error 403: Forbidden error.

see

• RDFLib/sparqlwrapper#139
• https://phabricator.wikimedia.org/T230135

try to avoid by setting user agent

will be needed for some (slow) storage backends like SPARQL and SMW via WikiPush

dtime=datetime.datetime.fromisoformat(value) TypeError: fromisoformat: argument must be str

The isCached code is currently awkwardly assuming that a table is not cached if it contains only 100 or less entries.
It should be ok to e.g. have an empty table in quite a few use cases.

 sqlQuery="SELECT COUNT(*) AS count FROM %s" % self.tableName
                try:
                    sqlDB=self.getSQLDB(cacheFile)
                    countResult=sqlDB.query(sqlQuery)
                    count=countResult[0]['count']
                    result=count>100
                except Exception as ex:
                    # e.g. sqlite3.OperationalError: no such table: Event_crossref
                    pass      

mergeSchema function is needed

If a class provides instance samples with getSamples use that information instead of creating a separate types-json file

e.g. if ther are blanks in a link

make sure roundTrip is possible by converting type names to string and back

refactor form https://github.com/WolfgangFahl/ProceedingsTitleParser/blob/master/ptp/plot.py

for SPARQL queries with optional values certation columns might not exist in all records. In this case the fixNone approach has to be applied as is already done for SQL queries and a dict entry with "None" as the value needs to be inserted.

currently to issue command like "CREATE VIEW" a code line like:

sqlDB.c.execute(viewDDL)

is necessary. It's better to offer a wrapper execute that will delegate the call so that for debugging and exception handling the execute calls can be traced and handled

sqlDB.execute(viewDDL)

would then be the recommended way for such calls

refactor from https://github.com/WolfgangFahl/ProceedingsTitleParser/blob/master/ptp/listintersect.py

in executeMany=False mode the culprit record should be shown if debugErrors=True

Attributes that are not represented in the samples should be excluded from the storing. (If enabled)

 def setNone4List(self,listOfDicts,fields):
        '''
        set the given fields to None for the records in the given listOfDicts
        if they are not set
        Args:
            listOfDicts(list): the list of records to work on
            fields(list): the list of fields to set to None 
        '''
        for record in listOfDicts:
            self.setNone(record, fields)
    
    def setNone(self,record,fields):
        '''
        make sure the given fields in the given record are set to none
        Args:
            record(dict): the record to work on
            fields(list): the list of fields to set to None 
        '''
        for field in fields:
            if not field in record:
                record[field]=None

but make sure a copy of the records is optionally used to keep the original list of dict as is.

http://ptp.bitplan.com uses https://github.com/WolfgangFahl/ProceedingsTitleParser/blob/master/storage/query.py
this would also be useful for https://confident.dbis.rwth-aachen.de/dblpconf/ - so migrate here to make it available for other projects.