There is a reproducibility crisis in science.

In computational science this is completely avoidable. If we can tell a computer what to do, we can tell it to do it again. So we can tell other people how to tell their computer to do it again.

In the domain of discrete event simulation, the software and methodologies that are traditionally used can make it difficult to do this. In a recent preprint I have submitted with Dr. Vincent Knight, Prof. Paul Harper, and Asyl Hawa, we compare simulation approaches in terms of reproducibility and best practice, and introduce a Python library for discrete event simulations of queueing networks, Ciw, that is designed with these in mind.

A conference paper by Richard A. Kilgore lists three properties as minimum requirements for reproducibility in simulations:

  • Readability
  • Modularity
  • Extendibility

We add to that best software practices.

The Problem

The current landscape of discrete event simulation practices is largely reliant on commercial stand alone software, which are often GUI-heavy. There’s also a small proportion of simulation practitioners who work in Excel (small, but significant enough to be included in most simulation textbooks).

Let’s elaborate:

  • Commercial software have high costs. This limits model sharing to those who can pay.
  • Closed source software restricts access to source code, inhibiting model understanding and flexibility.

These properties directly go against the concept of open science.

  • Stand alone packages tend not to be modular or extendible, hindering model re-usability.
  • Model testing is difficult, and the use of GUIs may encourage bad model validation and verification.
  • Binary model files make version control troublesome.
  • Models hidden behind GUIs are not readable, crucial parameters and behaviours are hidden behind a hierarchy of menus.

A Solution?

One solution to this that we propose is the Ciw library, used in a Python ecosystem.

All three properties of reproducible simulation modelling (Readability, Modularity, and Extendibility), along with best practices, can apply to well written Python, and the use of Ciw:

  • Open source so free to download, use, modify, and contribute.
  • All of Ciw’s source code is available, modifiable, and extendible. It is also fully tested.
  • A simulation model is a script, so is testable, shareable, and version controllable.
  • We believe Ciw has clear, readable syntax.*
  • Seeds of random number streams are clearly set, and data collection is transparent, which means everything can be saved.
  • Python is renowned in the scientific community, as it has readable syntax and an abundant collection of other scientific libraries.
  • Conducting simulations in this ecosystem would then allow seamless integration with other methodologies, such as data analysis, machine learning, and other algorithms.
  • Object-orientation, which Python is known for, and on which Ciw is built, is very agreeable with discrete event simulation. This also forces modularity, and encourages extendibility.

*Here’s a sample of Ciw’s syntax, for obtaining the average waiting time of an M/M/1 queue, with arrival rate \(\lambda = 4\), service rate \(\mu = 5\), after simulating for 20 time units, with warm-up time 2 time units, over 5 trials:

>>> import ciw

>>> N = ciw.create_network(
...     Arrival_distributions=[['Exponential', 4.0]],
...     Service_distributions=[['Exponential', 5.0]],
...     Number_of_servers=[1]
... )

>>> average_waits = []
>>> for trial in range(5):
...     ciw.seed(trial)
...     Q = ciw.Simulation(N)
...     Q.simulate_until_max_time(20)
...     recs = Q.get_all_records()
...     waits = [r.waiting_time for r in recs if r.arrival_date >= 4]
...     average_waits.append(sum(waits) / len(waits))

>>> sum(average_waits) / len(average_waits)
0.379274988243813

The paper compares a range of simulation frameworks: Ciw, SimPy, SIMUL8, AnyLogic, custom scripts written in Python and C++, and spreadsheet modelling in Excel.

In addition to comparing these in terms of best practices and reproducibility, performance comparisons were made. Ciw did fall behind slightly in this respect, being around 320 times slower than the custom C++ script when run with a CPython, and 50 times slower than C++ when run with PyPy. However, speed is not prioritised in the development of Ciw, and the developers would never compromise its main purposes (model readability and enabling reproducible scientific research) in favour of speed.

In summary: current practices in discrete event simulation do not lend themselves well to reproducibility. We introduce an environment that fosters reproducibility of simulations, namely Python and the Ciw library.