It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of access-log file into IP addresses and size of requests and fork off separate processing threads: one for obtaining unique set of IP addresses and another for summing up.
It seems like all that is needed for describing a pipeline is:
(a) a queue for input
(b) a processing program that's connected to the queue
(c) (possibly multiple) queues for output
(d) a topology connecting the processing programs
(e) a job scheduler.
At the face of it, it looks similar to Apple's Automator and Matt Welsh's PhD thesis on SEDA:
Here is some older material on pipeline coding in Lua you might be interested in that describes this problem in terms of Filters, Sources, and Sinks: http://lua-users.org/wiki/FiltersSourcesAndSinks
It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of access-log file into IP addresses and size of requests and fork off separate processing threads: one for obtaining unique set of IP addresses and another for summing up.
It seems like all that is needed for describing a pipeline is:
At the face of it, it looks similar to Apple's Automator and Matt Welsh's PhD thesis on SEDA: EDIT: Formatting