Efficient memory ordering for DiscreteDP

[oyamad]

(This is to summarize my comments #109 (comment), #109 (comment), #109 (comment).)

• n: number of states
• m: number of actions (Assume it is constant over the states)

Consider bellman_operator, where we compute

• vals = R + beta * Q * v and
• Tv = s_wise_max(vals).

Let me denote R + beta * Q * v by U(s, a) as a function of (s, a), to distinguish it from a Julia array; let me denote the probabilities by q(sa, s'), where sa is a state-action pair, while s' is a state tomorrow.

To compute s_wise_max, the data for U should be ordered in memory as

U(1, 1), ..., U(1, m), U(2, 1), ..., U(2, m), ..., U(n, 1), ..., U(n, m)

i.e., action should change faster.

So the same applies to R: action should change faster.

To compute Q * v, the data for q should be ordered in memory as

q(sa_1, 1), ..., q(sa_1, n), ..., q(sa_L, 1), ..., q(sa_L, n)

(where the state-action pairs are listed as sa_1, ..., sa_L)

i.e., state-tomorrow should change faster.

Given 1. and 2., for Q, state-tomorrow should change first, action second, and state-today third.

[sglyon]

This was very helpful, thank you for summarizing here.

From the very last statement, it sounds like in Julia we need to order Q(s', a, s) and in python we would need to order it Q(s, a, s'). Do you agree?

[oyamad]

Yes, that's my opinion.