cloudevents / sdk-python Goto Github PK
View Code? Open in Web Editor NEWPython SDK for CloudEvents
Home Page: https://pypi.org/p/cloudevents/
License: Apache License 2.0
Python SDK for CloudEvents
Home Page: https://pypi.org/p/cloudevents/
License: Apache License 2.0
from cloudevents.http import from_http
from_http({"ce-id":"1", "ce-source":"s", "ce-type": "t"}, "123")
should raise MissingRequiredHeader hinting at specversion
raises int object has no attribute get because we don't check instance of raw_ce after json loading here on line 54
Might want an if statement checking the value of raw_ce, and error handlinlg appropriately
from cloudevents.sdk.converters.binary import NewBinaryHTTPCloudEventConverter
cnvtr = NewBinaryHTTPCloudEventConverter()
assert not cnvtr.can_read({"ce-specversion": "1.0"})
cnvtr.can_read({"ce-specversion": "1.0"})
will return true because it only checks for ce-specversion and not other required versions.
Automatic releases should be setup for this repo, including publishing when pushing to master
. Use tooling where possible.
Release tooling not setup. There's a release.sh
file, but I'm not sure it it's automated.
Can we use GitHub Actions?
See how the Google python client uses it:
.github/workflows
folder..github/workflows/release-please.yml
releaseType: python
and python in other places in the yamlmaster
to trigger the bot.Contact @busunkim96 from Yoshi, @grant, go/python-chat for possible help.
Read this: https://github.com/actions/setup-python, possibly setup on a personal repo
from cloudevents.http import to_binary, to_structured
Because these functions are within the http module, we don't need to have the suffix _http
from cloudevents.http import to_binary_http, to_structured_http
README badges for quick status of the repo
No badges
https://github.com/GoogleCloudPlatform/functions-framework-python
Need to validate headers in structured CloudEvents calls and automatically toggle between Binary and Structured CloudEvents
from cloudevents.http import CloudEvent
Simple path to import stuff
from cloudevents.sdk.http import CloudEvent
sdk path doesn't actually hold anything therefore we would want to either sub-package http folder or possibly expose it top level by importing it within a cloudevents.__init__.py
Assuming we need the breadth of data encapsulation as seen in the events folder, we should rewrite all of the setter/getter methods into python properties using the @properties decorator
It feels a little heavy to install sphin alongside this package. Maybe this could be only installed in a different target (dev maybe) ?
MultiDict (https://github.com/aio-libs/multidict) is a better option to work with HTTP headers
from cloudevents.http import from_http
headers = {
"Ce-Source": "<my-source>",
"Ce-Specversion": "1.0",
"Ce-Type": "my-type",
"Ce-Id": str(uuid4())
}
from_http("", headers)
should pass despite upper-cased attributes names.
from cloudevents.http import from_http
headers = {
"ce-source": "<my-source>",
"ce-specversion": "1.0",
"ce-type": "my-type",
"ce-id": str(uuid4())
}
from_http(None, headers)
likewise, this should return an event with data set to none.
None data causes _json_or_string to fail, Recommend addding a None if statement in that function.
Capitalized ce-specversion isn't read by from_http. Instead, from_http should make all keys lower case before passing it into future functionality.
We should probably create sample code showing both non-json data and base64 encoded data
from_http
call on an invalid request raises CloudEvent.InvalidAttributesError
from_http
call on an invalid request raises ValueError which is vague
Hi,
I tried the sample cloudevent_to_request
script and I got this:
binary CloudEvent
content-type: application/json
ce-specversion: 0.2
ce-type: cloudevent.event.type
ce-source: <event-source
ce-id: my-id
Traceback (most recent call last):
File "./cloudevent_to_request.py", line 74, in <module>
run_binary(event, url)
File "./cloudevent_to_request.py", line 32, in run_binary
print(binary_data.getvalue())
AttributeError: 'str' object has no attribute 'getvalue'
structured format worked well
I got an error with the pattern:
m = marshaller.NewDefaultHTTPMarshaller()
@app.route("/")
def handle():
event = m.FromRequest(....)
print(event)
Invoking this a second time, I got the following exception:
File "env/lib/python3.5/site-packages/cloudevents/sdk/marshaller.py", line 71, in FromRequest
content_type, self.__converters))
cloudevents.sdk.exceptions.UnsupportedEventConverter: Unable to identify valid event converter for content-type: 'No registered marshaller for application/json; charset=utf-8 in <generator object HTTPMarshaller.__init__.<locals>.<genexpr> at 0x7fbb573c6db0>'
I can look into this later this week.
def from_http(
headers: typing.Dict[str, str],
data: typing.Union[str, bytes],
data_unmarshaller: types.UnmarshallerType = None,
):
This order of parameters would make cloudevents.http.from_http more consistent with the other methods which either expect some header like object first, or return headers followed by data
def from_http(
data: typing.Union[str, bytes],
headers: typing.Dict[str, str],
data_unmarshaller: types.UnmarshallerType = None,
)
Error at
sdk-python/cloudevents/sdk/event/base.py
Line 157 in f51b5e4
I get:
cloudevents/sdk/event/base.py", line 157, in MarshalBinary
for key, value in props.get("extensions"):
ValueError: too many values to unpack (expected 2)
I think it should be props.get("extensions").items().
I think I'm hitting this because I'm creating one event from another and specifically passing on the extensions using SetExtensions rather adding each as an item:
revent = (
v02.Event().
SetContentType("application/json").
SetData(responseStr).
SetEventID(resp_event_id).
SetSource(self.model.event_source()).
SetEventType(self.event_type).
SetExtensions(event.Extensions())
)
from cloudevents.http import is_binary, is_structured
from cloudevents.sdk.converters import is_binary, is_structured
These methods at the moment comform to the http spec but aren't generalized to other specs. Therefore we would want to move these methods into the http module
We may want to add the following to pypi_release.yml:
with:
skip_existing: true
to enable readme adjustments pushed to master. Unsure if this is the best way of handling this, or if the workflow should be changed to only allowing tagged commits
Should we unit test the sample code and if so how should we standardize this? Unsure if we should make http samples modular such that we can directly test the logic, or if tests should simply reflect at a high level what you expect CloudEvents to do.
A json CE without required attributes can still pass the FromRequest
, e.g. without id
data = { "specversion": "string", "datacontenttype": "string", "data": "Unknown Type: object,string", "data_base64": "string", "time": "2020-04-24T10:58:14.315Z", "dataschema": "string", "subject": "string", "type": "string", "source": "string" } marshaller.FromRequest( v1.Event(), dict(request.headers), io.StringIO(json.dumps(data)), lambda x: x )
meanwhile,
the v1.Event()
doens't enforce the value of specversion
in structured mode.
the data_base64
is put into extensions
instead of treating as an optional attribute.
Following code should work using the pypi package.
pip install cloudevents
python
>>> from cloudevents.sdk.http import CloudEvent
pip install cloudevents
python
>>> from cloudevents.sdk.http import CloudEvent
Throws error because module http does not exist due to pypi being outdated.
Ultimately i'm asking if everyone thinks branch v1.0.0-dev should be considered stable now, and if not what else can be done.
Should we add further testing to address #32
from cloudevents.sdk.http_events import CloudEvent
attributes = {"source": "<source-url>", "type": "com.issue.extensions", "example-extension": "ext1"}
data = 'Hello'
event = CloudEvent(attributes, data)
headers, body = event.to_http()
print(body)
The above code will produce the following:
b'{"specversion": "1.0", "id": "fc713795-93f6-44a8-b67e-7b8bd7071e2a", "source": "<source-url>", "type": "com.issue.extensions", "time": "2020-07-16T22:41:48.222788+00:00", "extensions": {"example-extension": "ext1"}, "data": "Hello"}'
When in fact we expectthis to be outputted:
b'{"specversion": "1.0", "id": "fc713795-93f6-44a8-b67e-7b8bd7071e2a", "source": "<source-url>", "type": "com.issue.extensions", "time": "2020-07-16T22:41:48.222788+00:00", "example-extension": "ext1", "data": "Hello"}'
Current output is readable by python CloudEvents, but possibly not by other cloudevent systems unless I'm misunderstanding extensions.
As mentioned here in the spec the data must be serialize/deserialize to/from data_base64
when the data is determined to be binary.
For serialization examples, see the go implementation, or the C# implementation
For deserialization examples, see the go implementation, the java implementation, or the C# implementation
http_events.CloudEvent.from_http method currently only marshalls v1 events. This should probably be more dynamic in order to support multiple event versions. Perhaps we could restore specversion detection as seen in #36 or implement some other workaround.
Line 70 in https://github.com/cloudevents/sdk-python/blob/v1.0.0-dev/cloudevents/sdk/http_events.py
event = marshaller.NewDefaultHTTPMarshaller().FromRequest(
v1.Event(), headers, data, data_unmarshaller)
Some standardized auto formatting pre-commit such that developers can spend less time manually modifying code format and more time implementing logic.
Tox provides warnings but doesn't exactly auto-format. There were several complaints against tox, and developers having to create several additional commits to fix lint issues in the CI.
https://pypi.org/project/black/ for code formatting
https://pypi.org/project/isort/ for import path ordering
https://pypi.org/project/autoflake/ to remove unused imports
https://pypi.org/project/flake8/ for pep8 conformance
Seems like you guys have a lot going here for HTTP, I've got a need to include the AMQP bindings for processing cloud events in Python.
I'll be trying to resolve this issue myself, is that alright?
to_json and from_json in cloudevents.sdk.http.event.py raises notimplemented error
I prefer using Travis but any CI pipeline integration with pull requests would be helpful. Is Travis the right place for this? Want me to put in a config?
Hi! Is there any release soon for V1 implementation at PyPi?
from_http(data="", headers={})
This should throw CloudEventMissingFields("Invalid data field") or similar descriptive error
First, from_http attempts to read this event as a structured event.
Because there is nothing inside data, it fails to unmarshal'. Therefore it runs json.loads("")
because it expects data to be a json string with cloud event extensions.
Finally you get a json decoder error which is difficult to debug
Changelog should explain why versions v02 and v01 were removed
Changelog wasn't updated in that PR
View the changelog
from cloudevents.http import from_http
headers = {"ce-specversion": "1.0", "type": "me.type", "source": "<my-source>"}
from_http(headers, None)
I would expect the above code to raise cloudevents.exceptions.MissingRequiredFields("...")
Instead it does raise cloudevents.exceptions.InvalidStructuredJSON("...")
I believe it does this because line 48 we check whether headers is binary and if not we assume this is a structured request and try to read as json.
Perhaps instead of raising InvalidStructuredJSON we should set specversion to None and outside the if statement raise MissingRequiredFields('can't find specversion and couldn't decode json').
Right now there is a poorly defined PR template and no issue template.
This can make it hard to clearly communicate issues (as seen in some issues in this repo).
We should improve this.
Example:
https://github.com/google/new-project/tree/master/.github
Following crashes as CloudEvent attempts to unmarshall this as a binary event
from cloudevents.sdk.http_events import CloudEvent
import json
data = json.dumps({"id": "123", "source": "<source url>", "type": "com.sample.type", "specversion": "1.0"})
headers = {}
event = CloudEvent.from_http(data, headers)
from cloudevents.sdk.http_events import CloudEvent
import json
data = json.dumps({"id": "123", "source": "<source url>", "type": "com.sample.type", "specversion": "1.0"})
headers = {"Content-Type": "application/cloudevents+json"}
event = CloudEvent.from_http(data, headers)
We have not registered this content type with the IANA yet as seen here therefore we shouldn't expect all requests to have this header. We should instead check the http headers to determine whether event is structured or binary.
Hi all, first: thanks for providing this library. It's going to be really useful to have a standard way to work with CloudEvents from Python.
I took a preliminary look at it and my first impression was that the interface provided by this library isn't very "Pythonic", meaning that it doesn't provide an interface in a way that (I believe) would be expected by most Python developers. There are some small things (like using method names like FromRequest
instead of from_request
, using New
in class/method names) but also larger things (like the use of method chaining, setters/getters, etc.)
I'm hoping that since this project is still in alpha, there's still time to improve the interface here (and that y'all are open to suggestions). I'll try to provide some examples below of what I'm talking about but I'm happy to clarify if anything's unclear. I'm also not intimately familiar with the CloudEvents spec, so I may have some misunderstandings as well.
Event
from HTTP RequestThe current example for turning an HTTP request into an event:
import io
from cloudevents.sdk.event import v02
from cloudevents.sdk import marshaller
m = marshaller.NewDefaultHTTPMarshaller()
event = m.FromRequest(
v02.Event(),
{
"content-type": "application/cloudevents+json",
"ce-specversion": "0.2",
"ce-time": "2018-10-23T12:28:22.4579346Z",
"ce-id": "96fb5f0b-001e-0108-6dfe-da6e2806f124",
"ce-source": "<source-url>",
"ce-type": "word.found.name",
},
io.BytesIO(b"this is where your CloudEvent data"),
lambda x: x.read()
)
HTTPMarshaller
, and if it's the default maybe we should just get it by default when creating an Event
, instead of making the user initialize it every time?data_unmarshaller
is maybe not necessary? We're taking a bytestring, turning it into a BytesIO
object to pass as data
, and then using data_unmarshaller
to turn it back into a bytestring. Why not just have the user do any data unmarshalling themselves before constructing the event?Instead, consider:
from cloudevents.sdk import Event
event = Event(
headers={
"content-type": "application/cloudevents+json",
"ce-specversion": "0.2",
"ce-time": "2018-10-23T12:28:22.4579346Z",
"ce-id": "96fb5f0b-001e-0108-6dfe-da6e2806f124",
"ce-source": "<source-url>",
"ce-type": "word.found.name",
},
body=b"this is where your CloudEvent data",
)
Event
, detects which spec it matches, and gives back a corresponding Event
subclass. (If the user knew which spec they were supporting, instead of constructing an Event
they could construct a v02.Event
and skip the event detection)HTTPMarshaller
is used by default. This also helps avoid issues like #12.data_unmarshaller
field is removed (or at the very least, optional)The provided example is:
from cloudevents.sdk.event import v01
event = (
v01.Event().
SetContentType("application/json").
SetData('{"name":"john"}').
SetEventID("my-id").
SetSource("from-galaxy-far-far-away").
SetEventTime("tomorrow").
SetEventType("cloudevent.greet.you")
)
Instead, consider the following based on the same Event
class proposed above:
from cloudevents.sdk import Event
event = Event(
data=b'{"name":"john"}',
content_type="application/json",
event_id="my-id",
event_time="tomorrow",
event_type="cloudevent.greet.you",
source="from-galaxy-far-far-away",
)
Event
object can be created with a single method call. If the user needs to modify the event after initialization, they can act on the attributes directly (e.g. event.event_time = "yesterday"
).Currently, after creating an event, if the user wanted to get one of the fields such as the event time, the user would have to do something like:
event = ...
data = event.ce__eventTime.value
ce
-prefix is a bit unexpected and the double-underscore is unconventional.value
to get the value is also unexpectedInstead, consider something like:
event = ...
data = event.event_time
I realize this is proposing a pretty big overhaul of the current interface, but I think doing so would probably go a long way to lend towards the usability and maintainability of this library, and it'd be better to do it sooner than later. I'm happy to help out here, including designing/implementing these changes and helping maintain them afterwards, if it makes sense. Thanks!
Merge v1.0.0-stable branch into master updates pypi
Must run release.sh with proper permissions.
We should setup a github action using:
https://github.com/pypa/gh-action-pypi-publish
An approximate hook in .github/workflows/main.yml could be:
- name: Publish package
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.pypi_password }}
This assumes we have a github secret from pypi. We need 2 things for this automated step to work:
pypi_password
hereUsing a very minimalistic event results in a non-worky cloud events:
vaikas-a01:sdk-python-master vaikas$ python3 ./app2.py
{'content-type': 'application/cloudevents+json'}
b'{"specversion": "1.0", "id": "my-id", "source": "from-vaikas", "type": "vaikas.test", "datacontenttype": "application/json", "data": "{\\"HELLO\\": \\"FROM VILLE\\"}", "extensions": {}}'
Workaround:
https://github.com/cloudevents/sdk-python/blob/master/cloudevents/sdk/event/v1.py#L31
Removing this line makes it work (but then you can't use extensions):
in v1.py
Repro code:
event = (
v1.Event().
SetContentType("application/json").
SetEventID("my-id").
SetSource("from-vaikas").
SetEventType("vaikas.test").
SetData('{"HELLO": "FROM VILLE"}')
)
m = marshaller.NewHTTPMarshaller(
[
structured.NewJSONHTTPCloudEventConverter()
]
)
#broker url
#url = 'http://localhost:8080/'
headers, body = m.ToRequest(event, converters.TypeStructured, lambda x: x)
print(headers)
print(body.getvalue())
#send the request
response = requests.post(url, headers=headers, data=body)
Should the sdk be handling datacontenttype CloudEvent headers?
We should add further testing to have 100% test coverage
Recommend adding the following into .tox:
PYTESTARGS = -v -s --tb=long --cov=cloudevents --cov-report term-missing --cov-fail-under=100
At this moment ToRequest(event, "structured", lambda x: x)
will return the data as io.StringIO
, but ToRequest(event, "binary", lambda x: x)
will return with respect to the returning datatype of a data_unmarshaller
.
There's not suppose to be any kinds of differences. That's said, data
must be returned as io.BytesIO
in both cases.
event = CloudEvent(headers, data)
event.specversion
event = CloudEvent(headers, data)
event['specversion']
from cloudevents.sdk import converters
from cloudevents.sdk.http_events import CloudEvent
import requests
attributes = {
"Content-Type": "application/json",
"type": "README.sample.binary",
"id": "binary-event",
"source": "README",
}
data = {"message": "Hello World!"}
event = CloudEvent(attributes, data)
event['specversion']
Need discussion on how to define the namespaces for http methods (e.g. #49 (comment)).
from cloudevents.sdk import http
http.binary.to_json(event)
from cloudevents.sdk import converters
headers, body = event.to_http(converters.TypeBinary)
The above is just an example naming schema we could use.
We probably want to test version bumps using test pypi in pypi-release.yml
May also want to add stable branch creation to pypi_packaging.py and RELEASING.md
samples/http-json-cloudevents and samples/python-requests need tests. Additionally, samples/http-image-cloudevents needs to test with a flask test server
The SDK spec states we don't have to support v0.2 v0.1 versions. We should remove support and document current support here.
event = from_http(request.get_data(), request.headers)
event1 = from_http(request.get_data(), request.headers)
assert event == event1
I would expect this to pass as we are passing in the same fields into both objects
Assertion error because cloudevent does not support equality overload
from cloudevents.sdk.http import CloudEvent, to_binary_http
event = CloudEvent({"source": "<my-url>", "type": "issue.example"})
headers, body = to_binary_http(event)
Raises TypeError object of type NoneType has no len() when trying to marshall a None data type
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.