World hunger, political conflict, business competition and other complex problems cannot be solved with mathematical algorithms measuring probabilities alone. However, by combining together human intelligence with the best artificial intelligence, the company Quid has built software that experts are calling the worlds first augmented intelligence platform. Using superior speed and storage capacity of computation, the process by which human beings typically acquire the deep pattern recognition of expertise is accelerated. The software does more than run simple prediction algorithms, it allows users to interact with data in an immersive, visual environment to better understand the world at a high resolution so that they can ultimately shape it and change it.
Founded in 2010, Quid is addressing a new class of problems to help organizations make strategic decisions around business innovation, public relations, foreign policy, human welfare, and more. Through advanced visualizations that interpret massive amounts of diverse internal and publicly accessible external data sets, Quid tells a unique and compelling story about the complexity of our world – trends, comparisons, multi-dimensional relationships, etc. – to change the direction of decision making.
For Quid, it’s not about man battling it out with machines, but rather, man working with machines when entering a new level of complex problem solving. For example, military intelligence may one day be able to change the direction of future conflicts by working with Quid software to analyze millions of data points from war logs and reports, news articles, and social media about the most recent casualties of war. The intelligence teams plugged into Quid would be able to see the war unfold as it happens across multiple data dimensions and uncover the mathematical patterns hidden in the data that are shaping the direction of the conflict.
I spoke with Quid Co-founder and CTO Sean Gourley to explain how Quid is helping organizations leverage Big Data and augmented intelligence to tackle the Bigger Problems they are facing in a fast moving world.
1. Quid applies Data Intelligence to Big Data – a very different concept than applying Data Science to Big Data. Please explain.
Data Science is the dominant view of Big Data characterized by applying predictive models to highly structured data sets. Its roots come largely from social networks, as Jeff Hammerbacher and DJ Patil coined the term ‘Data Science’ in 2008 while working at the two leading social network companies – LinkedIn and Facebook.
These companies are collecting all types of information about their users to build database profiles to better predict user actions- whether it is clicking on an advertisement or a suggested social connection. So the heart of Data Science philosophy is being able to analyze millions of clean data points to meet a very clear measurable objective that can be A/B tested. You don’t care why a prediction happens, but only that it happens. Theory here is much less important.
Data intelligence on the other hand focuses on the ‘why’ something is happening. The why is important as it is needed if we are to step beyond prediction and drive real change. For example, Facebook has found that liking ‘curly fries’ also means that you are highly intelligent. This correlation will help advertisers sell more products associated with intelligence, but it won’t tell you why a person likes curly fries, which may be useful in addressing obesity or heart disease. Data intelligence helps you understand all of the elements driving someone to like curly fries in order to perhaps change the behavior (curly fries are bad for you!). This understanding requires extracting intelligence from more complex, unstructured data sets, as well as a human component to stitch all of the information together to gain insight. Quid is focused on augmenting this type of human interaction with machine number crunching to change the direction of behaviors and shape new worlds.
Just to summarize, Data Science might collect everyone’s Facebook likes in a structured database to predict whether a person is highly intelligent whereas Data Intelligence will piece together information from patents, scientific papers, news articles, and more to build an ensemble of information that might help you understand what intelligence really is.
Click here for a GigaOM article where Sean provides further explanation on the difference between Data Science and Data Intelligence.
2. Quid is a software application for strategic decision makers to explore and gain insight from complex data sets. Walk us through how one can get started with this application and use it successfully. Does it require extensive set up, training, and expertise?
Quid is designed to be a plug and play experience – all of the most relevant data sets are available within the software for analysis based on the strategic questions users want to ask. In contrast, other tools will require to you to spend a large amount of time to first acquire, clean, and reformat the data before you can even start the analysis. But Quid isn’t spending an enormous amount of time preparing the data either as we hook into data sources both inside and outside the organization. We have integrated data streams from third party data providers such as CapIQ for financial data, GNIP for social media, various news sources, and other data streams through our own web crawler.
Once a person is logged into Quid, users that are well versed in tools like MS Excel can easily start analysis. We also provide two training sessions, 90 minutes each so we can help users start to think about Data Intelligence and how to weave information together. The training also goes through network theory so users practice identifying patterns of relationships among entities in a social space. It is this line of thinking that leads you to answer seemingly complex questions like “How long before synthetic biology becomes a technology that can be easily hacked to pose a threat to US security” or “How is the conversation around privacy unfolding over time and where should we as a company position ourselves” or “What is the internal HR structure of apple and how would I replicate it” or “what are the most significant political events unfolding in japan surrounding the nuclear restart”.
We also have a highly skilled set of embedded engineers on call to help users understand and enhance analysis and visualizations. So it is really a partnership we have with our clients so that they best able to use our software to be successful in their strategic decision-making.
3. The key to insight is not about building the right model, but also about asking the right questions. How does Quid facilitate this paradigm?
Because Quid is focused on augmenting human interaction with data, we have designed our software accordingly to facilitate analysis at the speed of the human brain. The interface easily allows you to pivot between different data sets and projections of the information, allowing you to explore data to uncover questions you didn’t even know existed. You might begin your quest for information with a defined set of questions, but then ask different questions after uncovering new data points since Quid is pulling information from a vast number of data streams (news sources, documents, patents, etc.) to highlight new relationships. With Quid, users can easily navigate, analyze, and drill down into information as quickly as they move through a thought process or question after question.
4. Who are the ideal candidates for a Quid solution? What sets Quid apart from other Big Data analytics and visualization players in the market?
The ideal candidates are those that have difficult questions to answer, who are humble enough to admit that they don’t have all the answers, and who are open to using the power of data to make smarter decisions. The ideal user is one that is not trying to simply predict events, but rather to gain insight into them. Merger and Acquisition teams from some of the worlds largest companies are using Quid to gain a better understanding of where technology is heading, who their competitors are, and who they should partner with to change the direction of the company and meet strategic goals. Marketing teams are using Quid to change the direction of consumer conversations around various topics such as Wearable Computing and Privacy. It’s not about just monitoring positive or negative consumer sentiment, but rather understanding the ‘why’ around the sentiment to help influence the conversation.
What sets Quid apart from other vendors is that are focused on strategic decision makers and design our software specifically for them. We put our users front and center and design a visual interface to big data that is intuitive to use and allows for deep insights to emerge quickly. The software helps users to see signals, make connections, and recognize relationships between objects. The software is also one of the few platforms out there that allows users to deal with messy unstructured data in a clean and elegant fashion.
5. I know you are trying to make the world a better place with Quid, but your technology can also help corporations grow their business? Can you provide us with an example of how you have helped organizations innovate? Manage risk?
Let’s take a Product Launch as an example. Before a launch, a Marketing team can use Quid to understand conversations and messaging around relevant technologies and competitive products, helping to ensure the market responds in a positive way to the product. For a public company, if the market responds positively, the company stock price goes up. If not, the stock price could drop translating into millions of dollars lost depending on the size of the company.
Taking this a step further, by capturing the right messages for the product launch, you can also subsequently change the direction of the product messaging based on the strategy of the company since the market has already gained your attention and responded in a positive way. This same technology can be extended to political campaigns, or any form of public messaging strategy.
Users plug into quid on many smaller but also valuable questions, such as sales people meeting clients for the first time and wanting to know in 5 minutes all the significant events that have unfolded within their clients industry. Or keeping track of the political situation in China with an eye towards their global supply chain.
6. What technologies power Quid?
We’re fortunate to have the talented Tim Heath onboard as our VP of engineering. He’s led the team here to build out a wide variety of technologies that power the Quid software stack. Most of our stack is hosted in the Rackspace cloud and includes hundreds of server images. The life cycle of data at Quid starts with our data acquisition stack. This is where we interface with 3rd party data providers, scrape the web, and pull in data from other sources. This data is then parsed and transformed in our elastic compute stack which is primarily written in Python and leverages technologies such as Rabbit MQ and Storm. After we transform and cleanse the data, we archive it (for safety) and index it via Elastic Search. Because we can’t predict the kinds of questions users will ask from their data, so elastic computing and search is great because they provide the flexibility and power to support these requests. The highly parallelizable architecture can process 100,000 operations per minute on incoming data streams. Once the data is indexed, our users can access it via our search UI. Quid’s unique algorithmic layer does the heavy lifting to process the data to find implicit connections, clusters, anomalies and trends. The algorithms stack is written in Python and leverages open source packages such as NumPy, SciPy, and NLTK.
For the frontend we have designed an interactive user interface that is a core part of the Quid experience. This requires rendering tens of thousands interactive objects at a rate of 60 frames per second — which really pushes the limits of what you can do in a browser. To make all this possible we have built a custom visualization and physics engine from the ground up. We leverage technologies such as HTML5, Canvas, CoffeeScript and Backbone.js to push the visual performance. Underlying all of this technology is our devops stack where all server images are codified and managed via Chef. Deploying new clusters or making changes to our topology is both simple and safe. We’re also big fans of Vagrant so developers code locally on their laptop with the exact same server image you get in the cloud.
7. Is there a specific global issue that is personal to you that you feel Quid will be able to help solve one day?
Through immersing myself in the study of the mathematical structure of insurgency for the best part of a decade, I discovered the power of data to reveal the signatures that underlie modern war to better understand conflict in these complex war zones. Quid was soon founded in 2010 because I wanted to share this power of data and apply it to other complex issues the world is dealing with. With that, I not only want to see Quid being used one day to prevent war by better understanding all the dynamics leading up to the edge of war. In the very near term I’m also really excited about some of the environmental groups such as Greenpeace using Quid today to boost their campaign effectiveness and push the boundaries of messaging strategy.