The Tale of Creating a Distributed Web Crawler
Around 6 million records with about 15 fields each. This was the dataset that I wanted to analyze for a data analysis project of mine. But »
The Case of the Mysterious Python Crash
It was almost 11PM. My distributed web crawler had been running for a few hours when I discovered a very weird thing. One of its log »
Using Uber's Pyflame and Logs to Tackle Scaling Issues
Here I was again, looking at my screen in complete disbelief. This time, it was different though. My distributed web crawler seemed to be slowing down »