Alerti est une solution de veille et d'analyse des médias sociaux that want to control what is said about them online.
A big part of the Ruby on Rails application is a crawling system that gathers all information needed for the users. With a constant growth, this system started to work slowly with some database tables of several hundred million rows.
My mission was to make this app scale to handle more and more users. I started doing some research on different technologies, benchmarking for different databases and optimizing db indexing and db structure.
The second part was building the new architecture for the crawler that could scale whatever the charge. We used for that the following technologies:
- TokuDB as a sql engine
- Sidekiq for background parallel job processing
- rabbitMQ for messaging service between apps
- Redis for caching data