Brain-Inspired Robot Navigation Using Reinforcement Learning

This folder contains the implementation of three different reinforcement learning agents for my Master's research project: "Brain-Inspired Robot Navigation using Reinforcement Learning with Hippocampal and Striatal Contributions"

Webots 3D World with our Experiments Maze and Controller Python code on right side	Our E-puck Differential Drive Robot with Camera and Wheel Encoder sensors

Project Overview

The project demonstrates how different RL algorithms can model brain-inspired navigation strategies, specifically exploring the computational roles of the hippocampus and striatum in spatial navigation and decision-making.

This implementation provides a comparative framework for understanding how different RL paradigms can model various aspects of biological navigation strategies. It is an original work and personally-designed architechture that I developed in full as part of my Master's project thesis.

Agent Implementations

1. Deep Q-Network (DQN) Agent

File: DQN_train_epuck_rl.m

• Type: Value-based, discrete action space.
• Brain Analogy: Models striatal decision-making through action-value learning.
• Key Features:
  • Epsilon-greedy exploration strategy.
  • Experience replay buffer (1M samples).
  • Double DQN for stable learning.
  • Discrete actions: Forward, Turn Left, Turn Right.
• Network Architecture: Single-input network processing 504-dimensional observations (500 camera pixels + 4 odometry features).

2. Proximal Policy Optimization (PPO) Agent

File: PPO_train_epuck_rl.m

• Type: Policy-based, stochastic policy.
• Brain Analogy: Models hippocampal-like probabilistic decision-making.
• Key Features:
  • Stochastic actor network with softmax output.
  • Separate value function (critic) for advantage estimation.
  • Generalized Advantage Estimation (GAE).
  • Policy clipping for stable updates.
• Network Architecture: Dual networks (actor-critic) with shared feature extraction fed by 507-dimensional observations.

3. Twin Delayed Deep Deterministic Policy Gradient (TD3) Agent

File: TD3_train_epuck_rl.m

• Type: Actor-critic, continuous action space.
• Brain Analogy: Models smooth motor control similar to cerebellar-cortical loops.
• Key Features:
  • Deterministic policy with continuous actions.
  • Twin critic networks for reduced overestimation.
  • Target policy smoothing.
  • Delayed policy updates.
• Network Architecture: Multi-input critic networks with state-action concatenation fed by 504-dimensional observations.

Supporting Files

• ROS_e_puck_socket_controller.py: Webots-MATLAB interface.
• e_puck_reset_fcn.m: Environment reset function for episodic training.
• *.mexw64 and *.c: S-Functions for fast communication between Simulink and Webots.

Environment Setup

The agents operate in a Webots 3D simulation environment with:

• Robot: E-puck differential drive robot.
• Sensors: Camera (25×20 downsampled), GPS, wheel encoders (Odometry).
• Task: Navigate to target while avoiding obstacles.
• Observation Space: 504-dimensional normalized vector (or 504 + 3 elements one-hot vector of past actions).
• Reward Structure: Distance-based with goal achievement bonus.

Brain-Inspired Design Rationale

1. DQN (Egocentric Striatal Model): Discrete action selection mimics basal ganglia winner-take-all dynamics.
2. PPO (Allocentric Hippocampal Model): Probabilistic policy resembles place cell activation patterns.
3. TD3 (Motor Control Model): Continuous actions model smooth motor cortex output.

Training Configuration

• Episodes: 140 per agent.
• Episode Length: 12,000 steps (10 minutes at 50ms timestep).
• Environment: Webots simulation with real-time communication.
• Hardware: GPU acceleration supported for neural networks.

Usage

1. Execute the Webots simulator running the e-puck world \worlds\rosWebots3D.wbt.
2. Ensure the Python controller \controllers\ROS_e_puck_socket_controller.py is running there.
3. Run the desired training script in MATLAB found in folder \RL_agents_models.
4. Trained agents are saved as .mat files for evaluation.