Loading repository data…
Svalorzen / repository
A C++ framework for MDPs and POMDPs with Python bindings
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
This C++ toolbox is aimed at representing and solving common AI problems, implementing an easy-to-use interface which should be hopefully extensible to many problems, while keeping code readable.
Current development includes MDPs, POMDPs and related algorithms. This toolbox
was originally developed taking inspiration from the Matlab MDPToolbox, which
you can find here, and from the
pomdp-solve software written by A. R. Cassandra, which you can find
here.
If you are new to the field of reinforcement learning, we have a few simple tutorials that can help you get started. An excellent, more in depth introduction to the basics of reinforcement learning can be found freely online in this book.
If you use this toolbox for research, please consider citing our JMLR article:
@article{JMLR:v21:18-402,
author = {Eugenio Bargiacchi and Diederik M. Roijers and Ann Now\'{e}},
title = {AI-Toolbox: A C++ library for Reinforcement Learning and Planning (with Python Bindings)},
journal = {Journal of Machine Learning Research},
year = {2020},
volume = {21},
number = {102},
pages = {1-12},
url = {http://jmlr.org/papers/v21/18-402.html}
}
// The model can be any custom class that respects a 10-method interface.
auto model = makeTigerProblem();
unsigned horizon = 10; // The horizon of the solution.
// The 0.0 is the convergence parameter. It gives a way to stop the
// computation if the policy has converged before the horizon.
AIToolbox::POMDP::IncrementalPruning solver(horizon, 0.0);
// Solve the model and obtain the optimal value function.
auto [bound, valueFunction] = solver(model);
// We create a policy from the solution to compute the agent's actions.
// The parameters are the size of the model (SxAxO), and the value function.
AIToolbox::POMDP::Policy policy(2, 3, 2, valueFunction);
// We begin a simulation with a uniform belief. We sample from the belief
// in order to get a "real" state for the world, since this code has to
// both emulate the environment and control the agent.
AIToolbox::POMDP::Belief b(2); b << 0.5, 0.5;
auto s = AIToolbox::sampleProbability(b.size(), b, rand);
// We sample the first action. The id is to follow the policy tree later.
auto [a, id] = policy.sampleAction(b, horizon);
double totalReward = 0.0;// As an example, we store the overall reward.
for (int t = horizon - 1; t >= 0; --t) {
// We advance the world one step.
auto [s1, o, r] = model.sampleSOR(s, a);
totalReward += r;
// We select our next action from the observation we got.
std::tie(a, id) = policy.sampleAction(id, o, t);
s = s1; // Finally we update the world for the next timestep.
}
The latest documentation is available here.
We have a few tutorials
that can help you get started with the toolbox. The tutorials are in C++, but
the examples folder contains equivalent Python code which you can follow
along just as well.
For Python docs you can find them by typing help(AIToolbox) from the
interpreter. It should show the exported API for each class, along with any
differences in input/output.
Cassandra's POMDP format is a type of text file that contains a definition of an MDP or POMDP model. You can find some examples here. While it is absolutely not necessary to use this format, and you can define models via code, we do parse a reasonable subset of Cassandra's POMDP format, which allows to reuse already defined problems with this library. Here's the docs on that.
The user interface of the library is pretty much the same with Python than what
you would get by using simply C++. See the examples folder to see just how
much Python and C++ code resemble each other. Since Python does not allow
templates, the classes are binded with as many instantiations as possible.
Additionally, the library allows the usage of native Python generative models (where you don't need to specify the transition and reward functions, you only sample next state and reward). This allows for example to directly use OpenAI gym environments with minimal code writing.
That said, if you need to customize a specific implementation to make it perform better on your specific use-cases, or if you want to try something completely new, you will have to use C++.
The library has an extensive set of utilities which would be too long to enumerate here. In particular, we have utilities for combinatorics, polytopes, linear programming, sampling and distributions, automated statistics, belief updating, many data structures, logging, seeding and much more.
| Models | ||
|---|---|---|
| Basic Model | Sparse Model | Maximum Likelihood Model |
| Sparse Maximum Likelihood Model | Thompson Model (Dirichlet + Student-t distributions) | |
| Algorithms | ||
| [Dyna-Q][dynq] | [Dyna2][dyn2] | [Expected SARSA][esar] |
| [Hysteretic Q-Learning][hqle] | [Importance Sampling][imsa] | [Linear Programming][m-lp] |
| [Monte Carlo Tree Search (MCTS)][mcts] | [Policy Evaluation][mpoe] | [Policy Iteration][mpoi] |
| [Prioritized Sweeping][mprs] | [Q-Learning][qlea] | [Double Q-Learning][dqle] |
| [Q(λ)][qlam] | [R-Learning][rlea] | [SARSA(λ)][sarl] |
| [SARSA][sars] | [Retrace(λ)][retl] | [Tree Backup(λ)][trel] |
| [Value Iteration][vait] | ||
| Policies | ||
| [Basic Policy][mpol] | [Epsilon-Greedy Policy][megr] | [Softmax Policy][msof] |
| [Q-Greedy Policy][mqgr] |
Selected from shared topics, language and repository description—not editorial ratings.
knapeczadam /
🧟 Gameplay Programming course project (DAE, 2023). Focused on AI agents, decision-making, steering behaviors, and pathfinding using a custom C++ framework. Final project: Zombie Survival Game — an autonomous agent surviving through dynamic AI behavior.
76/100 health| [PGA-APP][pgaa] |
| [Win or Learn Fast Policy Iteration (WoLF)][wolf] |