I'm teaching a theory-focused randomized algorithms course at the undergraduate level, and have included a section on streaming algorithms. It would be great to get students to implement these algorithms, but I do not know how to simulate a data stream for them. I am aware that some companies have APIs with varying degrees of openness (e.g. Twitter), and I'm aware of big things like IBM InfoSphere Streams, but this seems too hard, since the course is mostly focused on theory. Ideally, I'd like something that continuously produces a stream of numbers or words, and have students write Python code to access the stream, then compute on the sample they take. Are there any freely available tools of this sort? Even, like, a free toy version of IBM InfoSphere Streams?

  • $\begingroup$ Welcome back! Do you just want a stream of random numbers? If so, I can build an answer off of something like this. If not, what data characteristics would be helpful? $\endgroup$ – Ben I. Sep 19 '18 at 18:43
  • $\begingroup$ @BenI., that sounds like overkill. Why aren't pseudo random numbers enough if all you really want is to illustrate the theory. $\endgroup$ – Buffy Sep 19 '18 at 20:06
  • $\begingroup$ Pseudo random numbers are perfectly fine. I just want them to be coming in a continuous stream, rather than stored on the machine. I think InfoSphere stream is basically an environment you program in, that handles the continuous streams of data in the background. So, you would write code that would start "listening" to the stream (i.e. take a sample) and then compute on what you get during that sample. $\endgroup$ – David White Sep 19 '18 at 21:19
  • $\begingroup$ Forgive my ignorance, but I would like to see a description added to the question of the types of operations that the stream processing would perform, especially when you mentioned "stream of words" rather than only numbers. Maybe there is a James Joyce emulator out there. Stream of consciousness, anyone? You could do an Eliza analysis on the stream... $\endgroup$ – Scott Rowe Sep 19 '18 at 23:04
  • 1
    $\begingroup$ I think I'll just create a giant txt file of numbers (or words, if we want to do a streaming algorithm on word data) and then have the students read it in line by line. To get practice with multiple streams, I can give them several different txt documents. That way, we don't actually need to interface with the internet or rely on an external API, both of which always take resources to learn. $\endgroup$ – David White Sep 20 '18 at 15:10

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.