Resumable uploads errors if (optional) name is omitted about gcp-storage-emulator HOT 6 CLOSED

Thanks for the great test case! Unfortunately files.pythonhosted.org is currently broken (at least for me) and I can't test properly with gcsfs.

$ pip install gcsfs
...
ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/6f/ad/46e12b207068000bfa76ab5e07154d975b0e1977cc0786299344e6c90925/gcsfs-2022.2.0-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)')))
...

I made a quick fix. #148 Could you test with it and see if gcsfs works as expected with the change?

If you don't want, or can't, build from the source, the latest Docker image follows the main branch.

docker pull oittaa/gcp-storage-emulator
docker run -d --rm \
  -e PORT=9023 \
  -p 9023:9023 \
  --name gcp-storage-emulator \
  oittaa/gcp-storage-emulator

from gcp-storage-emulator.

Wow, things look good on my end too so I'll close out the issue! Thank you for the super quick fixes! 🙌

from gcp-storage-emulator.

Must be a CDN problem or something, because GitHub doesn't have any issues fetching the packages, but I get some self-signed certificate every time. pypa/pip#10963

from gcp-storage-emulator.

Thanks for the quick look (and very odd on pypi 🙃)! Now I got a UnboundLocalError: local variable 'object_id' referenced before assignment (and apparently a non-json requests.data) with:

import gcsfs
import json

fs = gcsfs.GCSFileSystem()
fs.mkdir("test")

gcsfs.GCSFileSystem().touch("test/a") # I think this one just doesn't have the name in either the body nor query param
# UnboundLocalError: local variable 'object_id' referenced before assignment

with fs.open("test/b", "w") as file:
    json.dump(5, file) # this just writes the requests data with b"5"
# AttributeError: 'bytes' object has no attribute 'get'

For the optional file name, I'm wondering if GCS just generates a uuid or similar upload_id and gets the name on subsequent requests? It looks like we use the object name from the first request for the upload_id (which seems very reasonable), but maybe it should generate a uuid or something if no name... and I guess store a mapping of {uuid: upload session} sort of thing?

from gcp-storage-emulator.

Hmm... From the API documentation I understood that you have to pass either the query parameter or the metadata, but you shouldn't omit both. I need to take a closer look how gcsfs works when the CDN starts working again.

2. Optionally, create a JSON file that contains the metadata you want to set on the object that you're uploading.

...

OBJECT_NAME is the name you want to give your object. For example, pets/dog.png. This is not required if you included a name in the object metadata file in Step 2.

from gcp-storage-emulator.

Ok, I've tried to figure out what's going on. I'm about 99% sure that gcsfs uses a wrong method. They should use PUT method to write the data, but they're using POST. https://cloud.google.com/storage/docs/performing-resumable-uploads#single-chunk-upload, but for some reason the official Google API accepts it anyway, if the request includes upload_id parameter. So I added a workaround for the undocumented Google API behavior. #149

This should work now.

import os

import gcp_storage_emulator.server
import gcsfs
from google.cloud import storage

server = gcp_storage_emulator.server.create_server("localhost", 9023, in_memory=True)
server.start()
host, port = server._api._httpd.socket.getsockname()
os.environ["STORAGE_EMULATOR_HOST"] = f"http://{host}:{port}"

try:
    gcs_fs = gcsfs.GCSFileSystem()
    gcs_fs.mkdir("test_bucket")
    gcs_fs.touch("test_bucket/file")
    with gcs_fs.open("test_bucket/file", "w") as file:
        file.write("testing")

    client = storage.Client()
    bucket = client.bucket("test_bucket")
    for blob in bucket.list_blobs():
        content = blob.download_as_bytes()
        print(f"Blob [{blob.name}]: {content}")
finally:
    server.stop()

from gcp-storage-emulator.