Sparse Matrix's on POMDP model about ai-toolbox HOT 22 CLOSED

Additionally I would like to know if there is way to save objects to file like POMDP.Policy in the python version.
I've tried using pickle but I don't think that its supported. Is there any other way around it ?
Perhaps if I could export the values of POMDP.Policy into lists or dataframes I could serialize that instead and manually rebuild POMDP.Policy later

from ai-toolbox.

from ai-toolbox.

Hi, thank you for you help first of all

I've seen the SparseModel class but I didn't fully understand it.
My first question starts in the description where it says that it makes the observations sparse and I'm not sure if the transition and reward matrix is also sparse since they're pretty huge as well.
My second problem with it is the fact that at least in python it seems that i need to feed it with a full matrix and I would like to give it a already sparse matrix because for some problems i literally don't have enough ram to build the full matrix (either observation, transition or reward matrix)

Hi, I'll try to answer a bit more in depth later; in the meantime have you seen the class SparseModel in the POMDP namespace? It should hopefully be what you are looking for. Let me know if the documentation is not clear and you cannot manage to use it.
…
On Sat, Jul 10, 2021, 6:07 PM Cassandro Micael Gonçalves Martinho < @.***> wrote: I'm not sure if it is a issue or if it is even supposed to be a feature but I haven't been able to create a POMDP model with sparse transitions, rewards or observations. I'm working on a pursuit domain environment and since the states scale so much with the size of the problem I wanted to try and use some sparse matrix's since most of the entries are empty anyway but I haven't been able to do so. Would appreciate any kind help on the problem, thank you for your time PS: I'm using the Python version — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#51>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMI4DEQOPV4C3LXSU5WSMLTXBV3XANCNFSM5AELQO5Q .

from ai-toolbox.

I'll try to answer to all the questions in order.

The POMDP model classes are basically wrappers around an MDP model, adding the observations and observation function to it (since every POMDP has an MDP inside). The POMDP sparse model that is exported to Python uses both a sparse MDP inside (so sparse transition and reward functions, see the MDP::SparseModel docs) and a sparse observation function (see the POMDP::SparseModel).

Unfortunately initializing a POMDP sparse model from Python with sparse matrices is not very easy, and indeed currently there is no code that does this. This is because internally my classes use Eigen sparse matrices, while in Python you probably will have a separate sparse representation (maybe numpy?). Thus you will need to write some code to copy from your sparse representation to the internal Eigen one, which will need to be in C++.

For saving the policy to a pickle, I think it currently can't be done but it should be a pretty easy fix, I'll try to add it in the next few days.

from ai-toolbox.

Ok, next few days I'll try to implement the sparse POMDP and get back at you with any questions.

The pickle issue is to restrain from solving the models everytime they are useful. To do that it would be amazing if we could pickle both the model class and the policy class. This way we can save them to file and pick them up when needed

Thank you for your time, you've been extremely helpful

from ai-toolbox.

I'm working on the pickle issue now.

Regarding the sparse models, which methods did you want to try with them? I'm afraid that many POMDP methods implemented here do not currently take advantage of the sparse representation. A lot of code would need to be added to specifically iterate over sparse matrices. This is for example for belief updates, belief search, computing projections, cross-summing and so on. You might need to add support for sparse beliefs as well, depending on what you need to do.

from ai-toolbox.

Hey, I just pushed to master, policies should now pickle correctly :) Let me know if it works!

from ai-toolbox.

I'm working on the pickle issue now.

Regarding the sparse models, which methods did you want to try with them? I'm afraid that many POMDP methods implemented here do not currently take advantage of the sparse representation. A lot of code would need to be added to specifically iterate over sparse matrices. This is for example for belief updates, belief search, computing projections, cross-summing and so on. You might need to add support for sparse beliefs as well, depending on what you need to do.

Hey, I'm not sure if I understand the question.
I'm not sure what methods I'm using in the current POMDP.Model implementation (before using pickle) but the methods that I would like to access after pickle serialization would be the following: model.sampleSOR(s,a); model.getObservationProbability(i, a, o); POMDP.updateBelief(model, b, a, o).
I'm not experiencing too big of run times so I'm not worried about the efficiency when it comes to that, my goal is to try to reduce the amount of memory needed instead.

from ai-toolbox.

Hey, I just pushed to master, policies should now pickle correctly :) Let me know if it works!

I've tried out pickle serialization and ran some tests on the POMDP.Policy and it worked wonders, thank you so much. However, would it be possible to also serialize the model class, either POMDP.Model or POMDP.SparseModel ?

from ai-toolbox.

I'm not sure what methods I'm using in the current POMDP.Model implementation (before using pickle) but the methods that I would like to access after pickle serialization would be the following: model.sampleSOR(s,a); model.getObservationProbability(i, a, o); POMDP.updateBelief(model, b, a, o).

Ah, I see. Then there shouldn't be any problems :) I thought you might need some solver method like incremental pruning, which might not be happy to process large models.

I've tried out pickle serialization and ran some tests on the POMDP.Policy and it worked wonders, thank you so much. However, would it be possible to also serialize the model class, either POMDP.Model or POMDP.SparseModel ?

My bad, I thought they were already pickable. I'll try to fix them asap. Thanks for making me notice :)

from ai-toolbox.

Hey, I've finished adding pickling for all models (MDPs and POMDPs).

One note that you should probably aware of, is that to simplify things I have made pickling use the same mechanism I use to serialize the models in C++ using the operator<< and operator>>. You can find them in the two MDP/IO.hpp and POMDP/IO.hpp files (and respective .cpp files)

At the moment unfortunately this serialization is quite naive and does the same thing for both dense and sparse models. That is, it iterates over all S,A,S' pairs (and S,A,O for POMDPs) and just dumps all the values into a string. If in your case these spaces are very large, it might take a while to create and load a pickle, and also the pickle might take a lot of space.

This can be fixed relatively easily by having the serialization code be a bit smarter (just don't write any zeroes for sparse models and note for each non-zero entry where it is supposed to be). I'm not sure however how much time I'll have to do this in the next few days.

I have opened issue #52 to track this issue. If this is a serious problem for you I can help you make a patch to the library if you want. If you are interested we can discuss on the #52 issue directly.

from ai-toolbox.

Hey, sorry for the late reply.

I'm currently implementing the pickle of model and policy, I'll keep an eye to see if everything goes well and check the size of the object file to see if it is a problem, I'll get back to you if that is the case.

Hey, I've finished adding pickling for all models (MDPs and POMDPs).

One note that you should probably aware of, is that to simplify things I have made pickling use the same mechanism I use to serialize the models in C++ using the operator<< and operator>>. You can find them in the two MDP/IO.hpp and POMDP/IO.hpp files (and respective .cpp files)

At the moment unfortunately this serialization is quite naive and does the same thing for both dense and sparse models. That is, it iterates over all S,A,S' pairs (and S,A,O for POMDPs) and just dumps all the values into a string. If in your case these spaces are very large, it might take a while to create and load a pickle, and also the pickle might take a lot of space.

This can be fixed relatively easily by having the serialization code be a bit smarter (just don't write any zeroes for sparse models and note for each non-zero entry where it is supposed to be). I'm not sure however how much time I'll have to do this in the next few days.

I have opened issue #52 to track this issue. If this is a serious problem for you I can help you make a patch to the library if you want. If you are interested we can discuss on the #52 issue directly.

from ai-toolbox.

Hey,

After some testing with the model serialization I've found that the POMDP.SparseModel is behaving a little bit weirdly. I've compared a model before and after serialization + load and it seems that they have different probabilities, comparing with getTransitionProbability and getObservationProbability.
What seems weird is that this isn't actually happening for the base model POMDP.Model

from ai-toolbox.

from ai-toolbox.

I've made a file to try and replicate the issue, here's the link
https://github.com/CassandroMartinho/TestingModel/blob/e681878080d1b0819190d0d86eba3926f7e66ecf/model_test.py

from ai-toolbox.

I can't see the file. Could it be that the repo is private? If you can just copy-paste it here :)

from ai-toolbox.

It was indeed private, I'll just put the file here instead.
model_test.py.zip

from ai-toolbox.

I took a further look at the models before and after serialization. I've found that the method POMDP.Model.sampleSOR(s, a) is also returning different values before and after serialization, even when all matrix's are the same (transitions, observations and rewards). I'm attaching an update to the previous file with an additional comparison
model_test.py.zip

from ai-toolbox.

Hey, thanks for the additional work :)

I think I've found the problem; it's actually two separate bugs. One was in the sparse setTransitionFunction where I forgot to zero the transition matrix before copying the new values in. The other is in the actual IO part that serializes the class.

However since I noticed that the IO part is actually pretty badly written (I made it a loong time ago and it hasn't been used much since), I'm just going to rewrite them in a better way, essentially solving #52. It's fairly easy work but it takes a bit of time.

Thanks a ton for all the help!

from ai-toolbox.

Hey, I've finally pushed to master :)
I've basically revamped the whole IO system, so it should be much more resilient and easy to debug, solving #52. This way, sparse models are also pickled in sparse form which is an added bonus.

I've also tested your python script and it seems to not give any errors anymore, thankfully! Please do let me know if it works for you.

from ai-toolbox.

Hey, I've been testing it and it seems to be working very well.

Thank you for the added effort this will help my project a lot

from ai-toolbox.

Perfect! I shall close this issue then. Feel free to reopen if you find something does not work :)

from ai-toolbox.