Database created with Python 3.5 cannot be read with Python 2.7 about stalker HOT 25 CLOSED

For completeness I also paste @fredrikaverpil's comment here:

@jasperges Am I correct to believe the sqlalchemy/pickle issue when using both Python 2 and 3 > comes down to when you store data using PickleType?:
http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.PickleType

It seems you could define which protocol to use. In our case, perhaps we should look into specifying protocol 2, even when using Python 3?:

PickleType(protocol=2, pickler=None, comparator=None)
protocol – defaults to pickle.HIGHEST_PROTOCOL.

As far as I can see, Stalker currently only uses PickleType for the following:

• WorkingHoursMixin.working_hours
• Studio.last_schedule_message

I don't use this, which is perhaps why I haven't stumbled upon any issues when using both Python 2 and 3 to read/write to/from the same db (?).

Perhaps, in case this is an issue, we could switch out PickleType for something else and create a new alembic revision which will take care of upgrading the db?

from stalker.

The problem is indeed caused due to the use of PickleType. When you create a studio it gets default working hours, which uses WorkingHoursMixin.working_hours. So if you create a studio in Python 3.5, you can't use the database anymore in Python 2.7. @fredrikaverpil I guess you initialized the database and created a studio in Python 2.7 and after that used the database in Python 3.5. As far as I can see that should work.

As @fredrikaverpil suggested, the fix is quite simple. You can just change PickleType to PickleType(protocol=2) and it works. @eoyilmaz Shall I make a pull request for this?

from stalker.

This is correct. I initialized the db using Python 2.x.

About setting the protocol to version 2; Somehow I'm not sure about holding on to old versions is the best solution. Let's discuss. @eoylimaz what do you think?

Is there any chance we can use another data type?

from stalker.

I do agree the best solution is not to hold on to old versions.

As you suggest another data type is viable option. Or the protocol version could be a setting the user can control somehow. But that might make matters needlessly complicated.

from stalker.

PickleType is really unnecessary for this data. And because with my next push, Stalker will be mainly using PostgreSQL for its database backend, I think I should feel free to convert the data type to something like JSON.

from stalker.

Sounds good to me!

from stalker.

from stalker.

@eoyilmaz will stalker still work with SQLite?

from stalker.

Just answering my own question... yes, it seems SQLite is working just fine with the latest 0.2.18 release 👍

from stalker.

from stalker.

Aha. It was actually quite convenient to be able to run a SQLite db for development purposes. But I could spin up a Postgres Docker container instead.

I'm running Postgres 9.1 in production. Would you advise me to upgrade to a newer version due to these new features?

I see the docs still mention SQLite. I'd propose the docs should be updated to reflect these new changes.

On a similar note, I noticed that you can now use SQL for Postgres in Google Cloud. I wish to explore the possibilities to run Stalker either locally (using our own production database) or in the cloud, e.g. using the new SQL for Postgres in GCE. Most features are still in beta, but I'm hoping that it'll be possible to sync a local database with the GCE one somehow (replication?).
The reason behind this is we could then have freelancers work remotely using Stalker and this would be very useful. Have you explored any such options yourself?

from stalker.

Considering how we connect to a database is to give the username, password and an address, reaching to a database on the internet should not be problem (other than being slow may be).

The main problem should be the pipeline tools, I believe. The tools that freelancers should have installed on their system (i.e. anima for our studio).

from stalker.

Ahh yes I should update the Docs also yes.

from stalker.

Is this still an issue with 0.2.18?

I'm asking because I've been running 0.2.18 in development for quite some time now with Python 3.5.

from stalker.

Yep, it still is. The studio working hours are stored as the Python class in the database. If it was stored as a json object (or something similar), there would be no need for pickle and the problem would be solved I think.

from stalker.

Ok, I've released Stalker v0.2.19, but didn't have time to fix this one.
And I plan to fix this, in one of the next couple of commits that I'm planning to do soon.
FYI.

from stalker.

Ok, I've released Stalker v0.2.19

Wow, that's an impressive release! 👍
I'm updating here in my dev environment.

from stalker.

Just double-checking; the 0.2.19 release doesn't require an alembic upgrade, correct?

from stalker.

Awesome!
On a sidenote: do you have any news on the development of Stalker Pyramid? Is there any way I can help to make it fully open source?

from stalker.

Oh, I've completely unnoticed these messages for the last 2 days.

@fredrikaverpil yes it doesn't require an alembic upgrade.

@jasperges Oh boy (sigh!), we're still working on Stalker Pyramid actively (by we I mean me and my wife). The only reason that I do not release the source is that it contains HTML and CSS templates that we've bought (for 15 bucks actually!) and build the code on top of it. And the code in its current form (both the development and master branch) is crap and I don't like it.

On the other hand, for the last 1 and a half year or so I've created another branch called angularjs, where I treated the code like I do in Stalker, so I've beautifully coded it with TDD. What we've planned with this branch is to completely convert the server side code in to a RESTFul service and leave everything else (rendering the HTML templates and running the JavaScript codes) to client side with AngularJS. The server side is 80-90% completed, and I liked it a lot.

But the client side is around %5-10 percent coded. I can release the source code at its current form without the HTML and CSS templates (and probably need to remove a lot of *.js files that comes with the template). And then if you like the idea, you can help us on developing Stalker Pyramid.

from stalker.

I've forgot to mention that I've fixed this issue in v0.2.20 by properly deriving the WorkingHours class from Entity class. So no need to store it in Studio instance in a PickleType column (and the other PickleType column was the Studio.last_schedule_message which was really unnecessary).

But I don't want to get in to the hassle of writing a proper migration script which extracts the old Pickled WorkingHours data and creates a proper new WorkingHours instance.

So what do you say, do you mind the migration script to throw away the data which is very simple to recreate later on? or should I try to find a way to read the Pickled data and recreate the WorkingHours instances (which will take time) inside the migration script.

from stalker.

I've forgot to mention that I've fixed this issue in v0.2.20

Awesome!

So what do you say, do you mind the migration script to throw away the data which is very simple to recreate later on?

We actually don't use this feature of Stalker (not yet, at least) in our studio. So this is fine by me.

from stalker.

@eoyilmaz Wow, thanks! For me it's also perfectly fine to throw away the data.
And about Stalker Pyramid: I will reply after the weekend. Don't have much time right now.

from stalker.

This has been fixed in v0.2.20 (fd6f14d)

NOTE: After the alembic upgrade, if there is a Studio instance created before the upgrade, connecting the database will raise an AttributeError due to the missing WorkingHours instance of previously created Studio instance, just ignore it and create a new WorkingHours instance and assign it to the Studio instance. Next time you connected to the database, everything will work fine.

from stalker.

Because we've started talking about Stalker Pyramid here in this issue, I'm going to send a one final message about it. I've created a new repository for Stalker Pyramid and uploaded the stripped down version. So it doesn't have any closed source code dependency in it.

Now we can start talking about it there (probably in a new issue).

from stalker.