We use a multitude of models - linear and non-linear, supervised and unsupervised - to surface the best user-generated content from social for our clients.
Our ML stack is written almost entirely in Python - because of the readily available excellent libraries that give you many of the tools you'll need in your ML toolbox (matplotlib, numpy, pandas, scikit-image, scikit-learn, scipy to mention just a few). We have some more sophisticated image processing in C++. We find ourselves attacking quite a number of machine learning problems so to make our workflow more efficient we have built a layer of functionality above these libraries to manage the flow of data and speed up model prototyping and assessment. Processing input data and model fitting is usually done locally, with any hyper-parameter grid search or other computationally intensive tasks being run on a remote, cloud hosted cluster spun up on-demand.
Our tech is built around lambda architecture principles - we have a live path that processes new social content to the extent required to make content immediately available to relevant clients, and a batch path that processes complete datasets daily. Our ML models get used on both paths.
With a neat set of Python libraries for data manipulation and model definitions and our own model prototyping and assessment rig the biggest challenge is always building big, quality datasets. This continues to be our biggest learning and we are always keen to explore how others tackle this. We rely on a combination of in-house turking (which we consider a vital model development stage), external turking services (for volume) and have additionally looked at some knowledge transfer techniques.
We use a multitude of models - linear and non-linear, supervised and unsupervised - to surface the best user-generated content from social for our clients.
Our ML stack is written almost entirely in Python - because of the readily available excellent libraries that give you many of the tools you'll need in your ML toolbox (matplotlib, numpy, pandas, scikit-image, scikit-learn, scipy to mention just a few). We have some more sophisticated image processing in C++. We find ourselves attacking quite a number of machine learning problems so to make our workflow more efficient we have built a layer of functionality above these libraries to manage the flow of data and speed up model prototyping and assessment. Processing input data and model fitting is usually done locally, with any hyper-parameter grid search or other computationally intensive tasks being run on a remote, cloud hosted cluster spun up on-demand.
Our tech is built around lambda architecture principles - we have a live path that processes new social content to the extent required to make content immediately available to relevant clients, and a batch path that processes complete datasets daily. Our ML models get used on both paths.
With a neat set of Python libraries for data manipulation and model definitions and our own model prototyping and assessment rig the biggest challenge is always building big, quality datasets. This continues to be our biggest learning and we are always keen to explore how others tackle this. We rely on a combination of in-house turking (which we consider a vital model development stage), external turking services (for volume) and have additionally looked at some knowledge transfer techniques.
See https://www.periscope.tv/w/1RDGlYXeDpOJL for a quick talk where I expanded on why and how Twizoo uses ML.