tiger_antelope.py: ValueError: Input transition matrix does not contain valid probabilities. about ai-toolbox HOT 6 CLOSED

On Tue, Oct 27, 2020 at 1:48 AM Eugenio Bargiacchi ***@***.***> wrote: As a sanity check, you could try to add at line 264 (in the loop that constructs the probabilities) the following: for state in range(len(S)): coord = decodeState(state) T.append([[getTransitionProbability(coord, action, decodeState(next_state)) for next_state in range(len(S))] for action in A]) print([sum(x) for x in T[-1]]) # <-------------------- ADD THIS LINE The last line will perform the sums for you; all printed numbers should be 1.0, otherwise something is going wrong. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM2IICKR7UV4S2P7AEMGFTSM2CNJANCNFSM4TAEWADQ> .

On Tue, Oct 27, 2020 at 8:36 AM Eugenio Bargiacchi ***@***.***> wrote: Ah, I see. Something is going wrong in the creation of the transition function. What the Python code in the example is doing is creating a 3-dimensional matrix, with dimensions SxAxS, where each entry T[s][a][s'] corresponds to the probability of transitioning from state s to state s', given action a. What this means is that, to be correct probability distributions, all these numbers must be valid probabilities (between 0 and 1), and that the sum for all possible transitions from T[s][a] must be 1.0. The error is related to the fact that these assumptions are not true, so the program aborts. The prints are to check that all these sums are indeed 1.0, and in your case they are not, which explains it. That said, I think I found the problem. It's a subtler change on how integer divisions between Python 2 and Python 3. I'm going to do a patch soon that makes the example more "cross-compatible" against both versions. As a quick fix, you need to replace the decodeState(state) function in your example file with: def decodeState(state): """ Convert from state_index to coordinate. Parameters ---------- state: int Index of the state. Returns ------- coord: tuple of int Four element tuple containing the position of the tiger and antelope. """ coord = [] for _ in range(4): c = state % SQUARE_SIZE state = state // SQUARE_SIZE # This is the changed line that forces an integer division in Python 3. coord.append(c) return tuple(coord) ------------------------------ Regarding your problem, there's no direct way to do this "cleanly" without modifying the internals of the library (depending on which algorithm you plan to use). The best solution that will maintain compatibility with everything else is to simply use the same transitions for all actions in your particular states. So, for all actions, the transition probabilities will be the same. This makes picking the action does not affect the environment, which is what you want. This will allow planning algorithms to work correctly (like for example value iteration). If you plan to use reinforcement learning (for example Q-learning), then you might also want to force your agent to pick a specific action (say, 0) in those states, as it will prevent unnecessary exploration in states where picking an action does not do anything. Let me know if what I wrote makes sense to you :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM2IIFJIZBWXL4L3HB64XDSM3SGPANCNFSM4TAEWADQ> .