photo / import Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 11.0 77 KB

Import tool companion for all of the export tools

Home Page: http://theopenphotoproject.org

License: Apache License 2.0

Python 100.00%

import's People

Contributors

Stargazers

Watchers

Forkers

samatjain patricksan dcancel olberger jordanbrock sneakypete81 notmarrco yetangitu unclebumpy aleksandarsavic jadeye

import's Issues

When displaying the status of a processing file it blocks

Ideally when running the script it displays Processing 1 of 10 1234567890.json... immediately as it starts processing the file and then appends OK or FAILED once complete. Right now the first part appears to wait till the second part is ready which means the entire line shows up only after it's finished.

I presume it's the comma after the first print command.

Importer getting the wrong filenames

Having a strange issue. I used the picasaweb exporter found here https://github.com/olberger/export-picasaweb.git. I was able to export my json files easily.

I have ~/import/import.py and I copied my fetched directory that the picasaweb generated. So my json files are in the correct location. Running import.py from the docs gives the following.

python import.py --host=gregfitz.trovebox.com --consumer-key=*** --consumer-secret=*** --token=*** --token-secret=****

Found a total of 8 files to process
Processing 1 of 8 20120304T200700-5716135985317781906.json ...
Traceback (most recent call last):
  File "import.py", line 91, in <module>
    import_into_openphoto(client)
  File "import.py", line 42, in import_into_openphoto
    resp = client.post('/photo/upload.json', params)
  File "/usr/local/lib/python2.7/dist-packages/openphoto-0.2-py2.7.egg/openphoto/openphoto_http.py", line 87, in post
    return self._process_response(content)
  File "/usr/local/lib/python2.7/dist-packages/openphoto-0.2-py2.7.egg/openphoto/openphoto_http.py", line 128, in _process_response
    response = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The reason it fails is because the file doesn't even exist

$ cat fetched/20120304T200700-5716135985317781906.json 
cat: fetched/20120304T200700-5716135985317781906.json: No such file or directory

I tried both trovebox.com and openphoto.me as the hostnames.

_Update:_ It appears it is getting the right file name, but it deletes the json file even if it fails. So, the fact that file doesn't exist isn't the issue. It is actually only failing to parse the json, but then deletes the file.

UnicodeEncodeError: 'ascii' codec can't encode character

Hello! Thanks for this tool. I just tried it and encountered a few problems. The 500 errors are a bit of an issue, but the last one gave a message I thought might be useful.... (keys and secrets omitted).

openphoto-import (master)$ python import.py --host=d6y.openphoto.me --consumer-key=xx --consumer-secret=xx --token=xx --token-secret=xx
Found a total of 1577 files to process
Processing 1 of 1577 1010505223.json ... FAILED: 500 - An unknown error occurred.
Processing 2 of 1577 1010506241.json ... FAILED: 500 - An unknown error occurred.
Processing 3 of 1577 1011363856.json ... FAILED: 500 - An unknown error occurred.
Processing 4 of 1577 1011364722.json ... FAILED: 500 - An unknown error occurred.
Processing 5 of 1577 1058682439.json ... FAILED: 500 - An unknown error occurred.
Processing 6 of 1577 1059547820.json ... FAILED: 500 - An unknown error occurred.
Processing 7 of 1577 1059549012.json ... FAILED: 500 - An unknown error occurred.
Processing 8 of 1577 1059550908.json ...
Traceback (most recent call last):
  File "import.py", line 79, in <module>
    import_into_openphoto(client)
  File "import.py", line 37, in import_into_openphoto
    resp = client.post('/photo/upload.json', params)
  File "/Library/Python/2.7/site-packages/openphoto-0.1-py2.7.egg/openphoto/__init__.py", line 41, in post
    body = urllib.urlencode(params)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1294, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 1: ordinal not in range(128)


openphoto-import (master)$ more fetched/1059550908.json 
{"description": "H\u00f4tel Colbert, I think.  \n<a href=\"http://en.wikipedia.org/wiki/Saint-Omer\">en.wikipedia.org/wiki/Saint-Omer</a>", "license": "", "title": "Saint Omer", "dateUploaded": "1186648402", "photo": "http://farm2.staticflickr.com/1395/1059550908_ef47d97191_o.jpg", "permission": "1", "longitude": "2.253087", "dateTaken": "1186648402", "latitude": "50.749005", "id": "1059550908", "tags": "flickr:place_id=Mw_SImRQULzqjJpumA"}

Here's an example of one of the 500:

 {"license": "", "tags": "", "dateUploaded": "1186265418", "photo": "http://farm2.staticflickr.com/1134/1010505223_e994225f4a_o.jpg", "title": "an end to slow cheese", "dateTaken": "1186265418", "permission": "1", "id": "1010505223"}

ValueError: No JSON object could be decoded

I'm running OSX Lion, with Python 2,7,2 installed using Homebrew. I get the above error when I run the import script. It borks on each one with ValueError: No JSON object could be decoded and then moves that .json file into the 'errored' folder.

Could really do with getting this import done ASAP, as my Flickr Pro account runs out in 3 days time and I don't want to have to fork out for a renewal. Here's the full terminal output, leading to the error:

Found a total of 3444 files to process
Processing 1 of 3444 20070515T130531-499365892.json ...
Traceback (most recent call last):
  File "import.py", line 91, in <module>
    import_into_openphoto(client)
  File "import.py", line 43, in import_into_openphoto
    result = json.loads(resp)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

oauth_problem=signature_invalid

I exported 8133 photos from Flickr using the OpenPhoto export-flickr script. No problems encountered and I ended up with 8133 JSON files ready for OpenPhoto import.

Running the import.py script, I'm finding that about 15% of files get rejected with a oauth_problem=signature_invalid error, like this (oauth tokens obfuscated):

Processing 534 of 3582 20090326T113724-3387718988.json ... FAILED: 403 - oauth_problem=signature_invalid&debug_sbs=POST&http%3A%2F%2Fphotos.ruk.ca%2Fphoto%2Fupload.json&dateTaken%3D1237738476%26dateUploaded%3D1238078244%26id%3D3387718988%26license%3DCC%2520BY-NC-SA%26oauth_consumer_key%3DXXXXXXX%26oauth_nonce%3DXXXXXXX%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1335375265%26oauth_token%3DXXXXXXX%26oauth_version%3D1.0%26permission%3DTrue%26photo%3Dhttp%253A%252F%252Ffarm4.staticflickr.com%252F3420%252F3387718988_2d6c8eb77a_o.jpg%26tags%3Dparis%252Cfrance%26title%3DArriving%2520in%2520Gare%2520de%2520l%255C%2527Est
{u'license': u'CC BY-NC-SA', u'title': u"Arriving in Gare de l'Est", u'dateUploaded': 1238078244, u'photo': u'http://farm4.staticflickr.com/3420/3387718988_2d6c8eb77a_o.jpg', u'tags': u'paris,france', u'dateTaken': 1237738476, u'permission': True, u'id': u'3387718988'}

The remaining 85% of photos are imported with no issue at all.

ValueError: No JSON object could be decoded

Hello, I'm trying to import my flickr photo using this script and I'm facing the following error :

    Found a total of 48 files to process
    Processing 1 of 48 20090228T140409-3316349518.json ...
    Traceback (most recent call last):
      File "./import.py", line 91, in <module>
        import_into_openphoto(client)
      File "./import.py", line 42, in import_into_openphoto
        resp = client.post('/photo/upload.json', params)
      File "/home/sushi/pymod/openphoto-python/openphoto/openphoto_http.py", line 125, in post
        return self._process_response(response, content)
      File "/home/sushi/pymod/openphoto-python/openphoto/openphoto_http.py", line 178, in _process_response
        json_response = json.loads(content)
      File "/usr/languages/python/2.6/lib/python2.6/json/__init__.py", line 307, in loads
        return _default_decoder.decode(s)
      File "/usr/languages/python/2.6/lib/python2.6/json/decoder.py", line 319, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/languages/python/2.6/lib/python2.6/json/decoder.py", line 338, in raw_decode
        raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded

The JSON comes from the flickr export script.

Do you have any idea ?

consumer-key?

Where do I get consumer-key, consumer-secret, token and token-secret?
I created an app on openphoto.me, but I can't find this information anywhere.

All photos returning 500 Unknown Error

When trying to import 93 photos (JSON from flickr-export) I get FAILED: 500 - An unknown error occurred. for every single one.

Sample output:

Processing 93 of 93 20111111T000358-6332826962.json ... FAILED: 500 - An unknown error occurred.
{u'description': u'The fully carved heel and inner heel, with the offcuts against them.\n\nMore information + sound clips of this guitar at <a href="http://waterpigs.co.uk/lutherie/catalogue/acoustic-guitars/001/" rel="nofollow">waterpigs.co.uk/lutherie/catalogue/acoustic-guitars/001/</a>', u'license': u'', u'tags': u'', u'dateUploaded': 1320955438, u'photo': None, u'title': u'2 Nov 2011 17:40', u'dateTaken': 1320912238, u'permission': True, u'id': u'6332826962'}

It's not a problem with the oauth tokens (I've tried with invalid tokens and get an error 403).

Running under debian (the same server the app is hosted on, if that makes any difference)

Missing copyright / license information

The import.py lacks copyright / license details.

Thanks in advance.

Rate-limit import

For importing photos from Flickr, Flickr has an (unknown) rate limit on API queries. @jmathai mentions 3600/hr, and this page mentions API queries should average no more than 1 per second.

The importing script should import no faster than 1 photo/second.

oauth_problem=consumer_key_unknown

So I fetched all the JSON, I got my credentials from flickr. I looked in the database to get my credentials from a self-installed copy of openphoto. But every single photo bombs with this:

Processing 2 of 6157 20110310T133451-5515373763.json ... FAILED: 403 - oauth_problem=consumer_key_unknown

I've double and then triple checked the keys. I created a new flickr app and tried the keys from that, still no luck.

Help

Invalid MIME type when running import

When I run the import script, I get the following error:

Processing 1 of 2511 20060110T173838-84880548.json ...
Traceback (most recent call last):
  File "import.py", line 91, in <module>
    import_into_openphoto(client)
  File "import.py", line 42, in import_into_openphoto
    resp = client.post('/photo/upload.json', params)
  File "/usr/local/lib/python2.6/dist-packages/openphoto/openphoto_http.py", line 87, in post
    return self._process_response(content)
  File "/usr/local/lib/python2.6/dist-packages/openphoto/openphoto_http.py", line 142, in _process_response
    raise OpenPhotoError(error_message)
openphoto.errors.OpenPhotoError: Code 500: Invalid mime type

The contents of 20060110T173838-84880548.json are:

{"license": "CC BY-NC-SA", "title": "dividing", "dateUploaded": 1136914718, "photo": "http://farm1.staticflickr.com/39/84880548_8836262062_o.jpg", "permission": true, "longitude": -1.8705940000000001, "dateTaken": 1124637608, "latitude": 50.763545999999998, "id": "84880548", "tags": "flashbacks,bronzelake,flickr:place_id=3Hf1ls5SUrwCDw"

Any idea what the problem could be? I'm not sure if it's objecting to the MIME type of the JSON file or the JPEG that it references, but the JSON was written by the export-flickr script, and the JPEG loads fine in my browser.