A Chance Meeting: Social Media Monitoring

I was at a Business School today to attend an event and met a few people from the industry. Here is an interesting conversation I had with somebody I met. Let’s call him Mr. X.

Mr. X – So what do you do?
Ram – I am an entrepreneur and run a software product company called Circus Social. We do social media monitoring and analytics; and have offices in Singapore and Bangalore.

Mr. X – What exactly is social media monitoring?
Ram – In simple terms, we fetch data, and a lot of it, from multiple social networks like Facebook, Twitter, Instagram, Pinterest, Reddit etc. and also from several blogs, forums and news sites. We then process and augment this data (with sentiment, gender, removing spam etc) and allow marketers to get insights from this data. We show them trends, insights, a lot of charts and graphs and allow them to make better business decisions.

Mr. X – So you are a big data company. Sounds like tough work. What kind of business decisions can be made?
Ram – Yes, we are indeed a big data company. Currently, our clients are enterprises, typically marketers, who would like to know their customers better, research their markets, analyse their competitors, identify top influencers, get real time alerts on topics of interest etc. We are also launching our SME product shortly.

Mr. X – So, do you invade people’s privacy? Isn’t this data private?
Ram – We only crawl data that is already publicly available. Anything that is private or protected by you is completely out of bounds for us. We do an incredibly good job of collating all of this data in our platform and marketers find tremendous value in quickly viewing trends from 10,000 feet but also in the ability to drill down to understand the “why” and “how” behind the trends.

Mr. X – Will this be useful for my company?
Ram – If people are talking about your company, your brands or topics that you are interested in tracking, you will definitely find it useful.

Mr. X – Where can I get more information?
Ram – Here’s my business card. Please visit our website https://www.circussocial.com where you can find a lot more information about what we do and a list of some of our clients. You can also sign-up for a free demo.

Mr. X – Sounds good. Thank you!
Ram – You are welcome!

Check out how the 20/Twenty Social Monitoring platform has grown!

Growth of 20/Twenty Social Monitoring & Intelligence Platform – Under the Hood

Growth of 20/Twenty Social Monitoring & Intelligence Platform – Under the Hood

We have often been asked about the challenges we faced in scaling up our technology stack to manage big data. I have attempted to address this in this post which is the first of a series of blog posts on this and similar topics.

20/Twenty was created ground-up as the most intuitive and easy to use cloud based (SaaS) Social Monitoring & Intelligence platform in the world.   Based on our deep understanding of what marketers needed and the awesome designs we created, we signed up our first client even before the product was officially launched. The pressure to quickly deliver the first version of the product was intense 🙂

From an engineering point of view, there’s a huge amount of data that we pull (Think Big Data!), process, augment and then visualize in the platform all on a near real-time basis. Imagine someone tweeting and it appears on our platform within a few seconds along with augmented information including Gender, Sentiment, Engagement, Spam score etc.

The evolution of 20/Twenty has already seen a few stages of growth. The graph below shows how 20/Twenty data has grown over the last 2 years since our product launch. This is a really cool growth for a startup like Circus Social both from a business perspective as well as from an engineering standpoint. We used several tricks from the books as well as a few practical hacks to ensure our ability to fetch, process, augment and visualize high volumes of data continued to become better, though this journey was not without pain!

social-intelligence-20twenty-big-data-growth

 

Stage 1

We created over 200 custom marketing applications in our previous avatar at Circus Social working with some of the biggest brands in the world. We used the same open source technologies (PHP / MySQL) to create the first version of 20/Twenty. This worked well and as our data grew in the first few months, we continued to grow vertically by adding more capacity (CPU/RAM).

Most of the queries from the application were read queries whereas a bulk of “write operations” were being performed by our data crawlers. We therefore created an efficient master-slave architecture where the application would read from the slaves and the crawler scripts would write into the master. This worked well in general but the exponential increase in the volume of data meant that certain queries were running extremely slow and impacting the user experience.

Stage 2

Since our data volume was growing exponentially and the relational aspects of the database were not the core of our application, we realized that sooner or later, we would have to move to a NoSQL database. However, the performance issues that were cropping up had to be sorted quickly and without a downtime. We quickly realized that we needed a dedicated search engine and MySQL was not good enough for this purpose.

We explored several options and Elasticsearch came to our rescue here. Elasticsearch is a distributed, RESTful search and analytics engine that centrally stores your data in a manner which can be retrieved / read really fast by your applications. Our awesome tech team deployed this in a matter of days. The improvement in performance was remarkable. The plan worked and we cheered!

Stage 3

Word spread in Singapore and Asia about how good our platform was (and our sales team did a good job too!) and we continued to sign up new clients. The volume of data continued to grow for existing clients as well as new clients. The tech stack of MySQL and ElasticSearch did not let us down but we wanted to create an architecture that would scale infinitely, if there’s a thing like that.

In Stage 3, we moved the core of our database from MySQL to Cassandra (Elasticsearch was now interacting with Cassandra) and the backend code from PHP to Node.js. We also migrated most of our front end code to Angular.js for better performance. This was a major architectural change on a live application being used by several clients so we created a parallel production like environment and ran it parallelly for several weeks to ensure everything was working as desired before switching over.

While we did the above, we continued to work on cool new features on the product and opened up our data API’s to a few clients who wanted a deeper integration with their own applications. Other tools we used during this and other stages were Postman, Github and JIRA.

As we scale further from here, we will probably have newer and more exciting technology challenges and we will keep posting about them. If you are excited to work on some of these, do write to us at careers@circussocial.com

 

CIRCUS SOCIAL AND VOCANIC INTEGRATE PLATFORMS TO TAKE DIGITAL INTELLIGENCE & SOCIAL CARE TO NEW HEIGHTS

The state-of-the-art partnership will allow customers to combine the prowess of VSocial and 20/Twenty all on one dashboard

Singapore, 2016-Oct-04 — Circus Social, a leading social data analytics company in Asia, announced today a groundbreaking product integration with Vocanic, Asia’s largest full service social media marketing agency offering customers a full suite of social media management capabilities.

This collaboration allows clients to enjoy the best of Circus Social’s 20/Twenty, a next-generation social intelligence platform, and Vocanic’s VSocial, an enterprise social media marketing suite, all under one roof and on either of platforms.

Customers of both platforms now have the chance to listen to and analyse powerful social and news data through 20/Twenty, and use VSocial’s superior publishing features to publish and manage their social presence.

“We are excited about the opportunities this integration will present to clients,” Shreeniwas Iyer, CEO at Vocanic, said. “This enables clients to enjoy a platform that goes across the social spectrum, including listening, proactivity, publishing, engagement and Social CRM. Circus Social has a wonderful product that greatly compliments the capabilities of VSocial. We are delighted about the prospect of accessing new markets and prospective clients through this partnership with Circus Social.”

Ram Bhamidi, CEO at Circus Social, added: “This partnership is unique because it leverages on the social intelligence and data analytics capabilities of Circus Social as well as the publishing and collaboration features of VSocial. This means we now have joint big data capabilities to create really compelling offers for customers of both companies.”

Some of the innovative new features include a full social CRM capability that allows brands to search for and track their customers in real-time through ‘Listen in Social & Key Opinion Leaders’, store details and interaction histories in the Data Warehouse, and schedule content optimized for them through learnings from the Social & Behavioural Profile Insights tool.

About Circus Social Pte. Ltd

Circus Social is a social data analytics company that specializes in digital & social intelligence and analytics platforms. With offices in Singapore and Bangalore, India, Circus Social offers a full suite of social analytics services, including but not limited to, social media & digital listening and monitoring, analytics and reporting, and social intelligence and predictive technology.

Our solutions include:

20/Twenty: 20/Twenty is the world’s most comprehensive social intelligence platform that helps marketers make better business decisions by collecting, augmenting and integrating social, online, offline and premium News with Dow Jones Factiva, on one easy-to-use platform.
Beacon: our proprietary predictive algorithm that allows brands to spot and predict crises, leads and queries.

For more information, visit: www.circussocial.com

About Vocanic

Vocanic, founded in 2005, is Asia’s leading business focused on Social Media. With capabilities in Strategy, Digital Solutions and Creative, Singapore headquartered Vocanic has offices in Bangkok, Kuala Lumpur, and Jakarta. Operating under the GroupM umbrella, Vocanic can roll out regional programs with on the ground support for each major market.

VOCANIC supports clients to develop and implement strategies that deliver measurable business benefits. Offering services in Social Media Asset Management, Social CRM and Social Media Crisis Management, VOCANIC develops custom branded content designed especially for the social era.

For more information, visit: www.vocanic.com