Tessero\Mdp\MarkovDecisionProcess

fromDense(mixed $P, mixed $R, bool $validate = true): self

fromSparse(array $perAction, mixed $R, bool $validate = true): self

fromTransitions(int $states, int $actions, iterable $transitions, iterable $rewards, bool $validate = true): self

Build from transition triples, e.g. read from a database: [[state, action, nextState, probability], ...] and rewards [[state, action, reward], ...].

states(): int

actions(): int

rewards(): Tessero\NDArray

Expected immediate rewards (S, A).

valueIteration(float $gamma, float $epsilon = 1.0E-6, int $maxIter = 10000, mixed $initial = null): Tessero\Mdp\MdpResult

Value iteration to an epsilon-optimal policy.

policyIteration(float $gamma, int $evalSweeps = 0, float $epsilon = 1.0E-6, int $maxIter = 1000, mixed $initialPolicy = null): Tessero\Mdp\MdpResult

Policy iteration. $evalSweeps = 0 evaluates each policy exactly (dense LU, O(S^3): fine to a few thousand states); k > 0 is modified policy iteration with k evaluation sweeps (large models).

evaluate(mixed $policy, float $gamma): Tessero\NDArray

Exact value of a fixed policy: V = (I - gamma P_pi)^-1 R_pi.

qValues(mixed $V, float $gamma): Tessero\NDArray

Action values Q (S, A) for state values V.

finiteHorizon(int $horizon, float $gamma = 1.0, mixed $terminal = null): array

Backward induction over $horizon decisions.

simulate(mixed $policy, int $start, int $steps, ?Tessero\Random\Generator $rng = null): array

Sample a trajectory following $policy from $start.