From start to finish, best practices and practical advice for doing a simulation-based project.
This post is part 6 from a 7-part series about the Simulation Model LIfe cycle. You can catch up on the previous post here.
Life cycle of a simulation model
Analyze the Results
Report on the Findings
This post focuses on the sixth step and the next post will focus on the final step - you can start watching the lecture from step 6 here, or continue reading the blog post.
Note: Although one does go "back" in later steps to do some activities during previous steps, you by no means redo the entire step. The steps listed above are guidelines of the general steps that a typical project follows. On all steps, you can go backward and revisit previous steps, but unlikely that you will skip a step or move forward before completing a step to at least 80%-90%.
For a more insightful discussion on the matter read here
6) Analyze the Results
Analyzing the output of your simulation model can be an entire project on its own. Often very large models of complex systems have massive output files containing tons of data for various parts of the system modeled. This post aims to provide a high-level set of guidelines to assist you during this phase and get more value out of this phase of the simulation model life cycle.
1) Remember the model objective
Keeping the model objective at the forefront of every piece of analysis is key. Remember we are not in the verification and validation phase. We will not be looking at every single piece of output and scrutinizing the results unless it is needed to explain some anomaly found in the main output.
In the example model used during this series our objective was to measure the impact on queue length, thus our analysis should focus on this metric. All other analyses are by-products trying to explain the changes we see in the queue length.
Below is an example of a very succinct table showing what is the delta moving from multiple queues & multiple servers setup to a single queue & multiple servers setup.
As we can see the single queue & multiple servers setup is showing a decrease in both the maximum and average queue length.
If there are questions regarding this result, especially if it is counter-intuitive to what was expected we can do a deep dive into other outputs
2) Look at the data from various angles
This is especially true when the results are counter-intuitive or you are not able to explain why the specific results were achieved.
Having your data in an easily available format and able to open in Excel makes this a lot easier. Here you can easily create pivot tables, draw charts and do simple statistical analysis. If however, your data is too large (e.g. more than 1.4M lines of data) you will need to use some specialized tool like R, Python, Tableau or any others.
In our example, this can be for example comparing the queue length with the time of day and seeing that it matches your expectations.
3) Use various analysis techniques
Building on the previous point; as you look at the data in different ways you need to analyze it in different ways as well.
The first step would be to analyze the data using simple rudimentary statistics. Look at the min, average and max of each output that was identified as being related to the model objective. Do these values make sense? Are they aligned with what you expected?
The next step can be to draw some plots, for example, a histogram, so that you can see the distribution of data. Next, you can for example draw a time series chart for one of the iterations
4) Double-check results with the “experts”
It is also best practice to have another set of eyes look at your data. The first prize would be other team members, especially if you can get some of the operational people that have a better feel for the process and the data to advise on what comparisons to create from your data.
I specifically put the word expert in quotes to highlight the fact that you can get value from having your results, analysis and findings double-checked by not only real experts, but also by those who believe they are. Often people completely separated from the process can ask the most fundamental questions that help you structure your analysis better, go and look for errors inside your model, or just add that one additional model output that helps make sense of it all.
5) Ensure reproducibility in analysis
A key requirement, often overlooked by many, is to ensure that your analysis is reproducible. Although this is a key requirement for the scientific method, there are also many practical reasons why you should make your analysis reproducible. Very often in projects, there is a change in the design or maybe you find a small bug in your model and now you need to redo your analysis. If your analysis is truly reproducible then this should not be a problem. Simply a case of rerunning your model, taking the output and putting it into your (hopefully automated) pipeline of analysis.
The best practice for making your analysis reproducible is to code your analysis. If you use a tool like Excel for your analysis this can be achieved through well-structured data analysis using formulas and pivot tables or even VBA code. If you use a tool like Python or R the analysis steps should be coded into a script.
Summary
In the post, we looked at some guidelines for analyzing your simulation model output data. This is definitely a decrease in intensity from our previous post where we looked at the experimentation phase. In the next post, the last in this series, we will be bringing it home with another short post looking at some guidelines for presenting your results.
PS! Remember that during this phase you are more than likely to go back to your model and add more outputs and even summarise some outputs inside your model and only output the aggregated version. This can often same a lot of time if you keep on doing the same analysis outside your model with the data.
Looking forward to the next post on best practices for building your model?
Remember to subscribe, so that you can get notifications via email or you can also join the mobile app here!
Watch the full lecture here, the video below starts at step 6
What next?
If you liked this post, you are welcome to read more posts by following the links above to similar posts. Why not subscribe to our blog or follow us on any of the social media accounts for future updates. The links are in the Menu bar at the top or the footer at the bottom.
If you want to contact us for some advice, maybe a potential partnership or project or just to say "Hi!", feel free to get in touch here and we will get back to you soon!