This is the first of a new series (tag) Dancing with Wolfram. Occasionally, I need to work through programming strategies in Wolfram, and I need a place to store my ideas. Perhaps they will be useful to others.
The problem
When collecting data from a sensor, one wants to generate a list of {x, y} pairs where x is typically time and y is the sensor reading. One way of doing this is to create an empty list and then using Mathematica’s AppendTo function to add elements to the list. The problem with this approach, however, is that it is not very efficient. The function call makes a copy of the original list each time, and when the list of data gets very large, the time it takes to store a data point increases. Below, I’ve plotted the average time needed to store a datapoint as a function of the list size. For comparison, I’ve collected data (times measured on a Raspberry Pi v2) using AppendTo with a list (blue dots) and adding key -> value pairs to an association (orange dots).
After about 1000 data points, the AppendTo a list approach starts to take increasingly longer times. I was unable to collect any data beyond 10,000 data points since AppendTo started running into memory issues. It is not unreasonable to expect a sensor data set to contain in excess of 1000 data points, so the performance of AppendTo is not acceptable.