I'm teaching a theory-focused randomized algorithms course at the undergraduate level, and have included a section on streaming algorithms. It would be great to get students to implement these algorithms, but I do not know how to simulate a data stream for them. I am aware that some companies have APIs with varying degrees of openness (e.g. Twitter), and I'm aware of big things like IBM InfoSphere Streams, but this seems too hard, since the course is mostly focused on theory. Ideally, I'd like something that continuously produces a stream of numbers or words, and have students write Python code to access the stream, then compute on the sample they take. Are there any freely available tools of this sort? Even, like, a free toy version of IBM InfoSphere Streams?
-
$\begingroup$ Welcome back! Do you just want a stream of random numbers? If so, I can build an answer off of something like this. If not, what data characteristics would be helpful? $\endgroup$– Ben I. ♦Commented Sep 19, 2018 at 18:43
-
$\begingroup$ @BenI., that sounds like overkill. Why aren't pseudo random numbers enough if all you really want is to illustrate the theory. $\endgroup$– BuffyCommented Sep 19, 2018 at 20:06
-
$\begingroup$ Pseudo random numbers are perfectly fine. I just want them to be coming in a continuous stream, rather than stored on the machine. I think InfoSphere stream is basically an environment you program in, that handles the continuous streams of data in the background. So, you would write code that would start "listening" to the stream (i.e. take a sample) and then compute on what you get during that sample. $\endgroup$– David WhiteCommented Sep 19, 2018 at 21:19
-
$\begingroup$ Forgive my ignorance, but I would like to see a description added to the question of the types of operations that the stream processing would perform, especially when you mentioned "stream of words" rather than only numbers. Maybe there is a James Joyce emulator out there. Stream of consciousness, anyone? You could do an Eliza analysis on the stream... $\endgroup$– Scott RoweCommented Sep 19, 2018 at 23:04
-
1$\begingroup$ I think I'll just create a giant txt file of numbers (or words, if we want to do a streaming algorithm on word data) and then have the students read it in line by line. To get practice with multiple streams, I can give them several different txt documents. That way, we don't actually need to interface with the internet or rely on an external API, both of which always take resources to learn. $\endgroup$– David WhiteCommented Sep 20, 2018 at 15:10
1 Answer
Java has a Streams API that works with all of its collections. You create a stream with a source, transform or filter it with intermediate operations, and do something with it using a Consumer.
All Collections in Java have a .stream() method that creates a stream from the collection. You can also create streams from files, as well as creating streams of random objects from a Random.
Intermediate operations include filter, which accepts a Predicate as an argument, and map and friends that can transform objects into objects of other types or primitives.
Consumers include facilities to place the contents of a Stream into any Collection, System.out::println, and other pure side-effect operations.
This requires an understanding of lambdas and functional interfaces (our second-trimester course in Java covers these). This mechanism is powerful and succinct, and it does not alter the collection the stream is attached to.
