Optimizing Inventory Management with Reinforcement Learning: A Hands-on Python Guide | by Peyman Kor | Oct, 2024

The present state is represented by a tuple (alpha, beta), the place: alpha is the present on-hand stock (objects in inventory), beta is the present on-order stock (objects ordered however not but obtained), init_inv calculates the overall preliminary stock by summing alpha and beta.

Then, we have to simulate buyer demand utilizing Poisson distribution with lambda worth “self.poisson_lambda”. Right here, the demand exhibits the randomness of buyer demand:

alpha, beta = state
init_inv = alpha + beta
demand = np.random.poisson(self.poisson_lambda)

Word: Poisson distribution is used to mannequin the demand, which is a standard selection for modeling random occasions like buyer arrivals. Nevertheless, we will both practice the mannequin with historic demand knowledge or reside interplay with atmosphere in actual time. In its core, reinforcement studying is about studying from the info, and it doesn’t require prior information of a mannequin.

Now, the “subsequent alpha” which is in-hand stock will be written as max(0,init_inv-demand). What meaning is that if demand is greater than the preliminary stock, then the brand new alpha can be zero, if not, init_inv-demand.

The value is available in two components. Holding value: is calculated by multiplying the variety of bikes within the retailer by the per-unit holding value. Then, we’ve one other value, which is stockout value. It’s a value that we have to pay for the circumstances of missed demand. These two components kind the “reward” which we attempt to maximize utilizing reinforcement studying methodology.( a greater method to put is we wish to decrease the price, so we maximize the reward).

new_alpha = max(0, init_inv - demand)
holding_cost = -new_alpha * self.holding_cost
stockout_cost = 0
if demand > init_inv:
stockout_cost = -(demand - init_inv) * self.stockout_cost
reward = holding_cost + stockout_cost
next_state = (new_alpha, motion)

Exploration — Exploitation in Q-Studying

Selecting motion within the Q-learning methodology includes a point of exploration to get an outline of the Q worth for all of the states within the Q desk. To try this, at each motion chosen, there may be an epsilon probability that we take an exploration strategy and “randomly” choose an motion, whereas, with a 1-ϵ probability, we take the very best motion attainable from the Q desk.

def choose_action(self, state):
# Epsilon-greedy motion choice
if np.random.rand() < self.epsilon:
return np.random.selection(self.user_capacity - (state[0] + state[1]) + 1)
else:
return max(self.Q[state], key=self.Q[state].get)

Coaching RL Agent

The coaching of the RL agent is finished by the “practice” operate, and it’s observe as: First, we have to initialize the Q (empty dictionary construction). Then, experiences are collected in every batch (self.batch.append((state, motion, reward, next_state))), and the Q desk is up to date on the finish of every batch (self.update_Q(self.batch)). The variety of episodes is restricted to “max_actions_per_episode” in every batch. The variety of episodes is the variety of occasions the agent interacts with the atmosphere to study the optimum coverage.

Every episode begins with a randomly assigned state, and whereas the variety of actions is decrease than max_actions_per_episode, the accumulating knowledge for that batch continues.

def practice(self):
self.Q = self.initialize_Q()  # Reinitialize Q-table for every coaching run
for episode in vary(self.episodes):
alpha_0 = random.randint(0, self.user_capacity)
beta_0 = random.randint(0, self.user_capacity - alpha_0)
state = (alpha_0, beta_0)
#total_reward = 0
self.batch = []  # Reset the batch at the beginning of every episode
action_taken = 0
whereas action_taken < self.max_actions_per_episode:
motion = self.choose_action(state)
next_state, reward = self.simulate_transition_and_reward(state, motion)
self.batch.append((state, motion, reward, next_state))  # Gather expertise
state = next_state
action_taken += 1
self.update_Q(self.batch)  # Replace Q-table utilizing the batch