What I've Learned From 10+ Years of Personal Projects

The desire to work on personal projects has grown on me pretty much unexpectedly.

About 10 years ago, I had a great job. But as my skills increased, I realized that an entire world of possibilities existed beyond my day-to-day duties.

So I started working on and off on a first project in my spare time, out of pure curiosity. Then one idea led to another and I started working on a second project. Fast forward to 2021, and I've been doing this for more than 10 years!

To tell you the truth, this has been the single greatest differentiator in my programming skills. From implementing specifications to building web apps and interpreters, I've become a much better programmer in every possible way.

So let me quickly introduce each one of my personal projects. I'll follow up with a summary of what I've learned in the process.

Hopefully, this will be a source of inspiration to start your own!

The Projects

Every now and then, I'll publish a longer-form post describing each project in greater details. But for now, here's a TDLR-like summary of each one.

1. An airfare and car rental price predictor (2010-2011): think about a predecessor of Google Flight's or Skyscanner's price prediction algorithm. I saved a lot of money!
2. A time tracking app (2011): I automated the logging of my time into my employer's project management system.
3. A universal connector framework (2011-2012): I created a no-code tool capable of querying XML and JSON web services transparently.
4. My Toastmasters club's website (2013-2014): I completely redesigned my Toastmasters club's website without much experience with either HTML, CSS or JavaScript.
5. A better customer support tool (2014-2015): I explored an idea of a radically better customer support tool.
6. A budget tracker (2014-2016): I created my own budget tracking tool, similar to Mint, but tailored to aspiring entrepreneurs.
7. An online toolbelt (2016): I worked on an idea of a web-based command-line interface aggregating many common tools (Base64 conversion, testing out regular expressions, etc.)
8. My technical blog (2016-present): I started the blog that you're currently reading to become a better writer.
9. A distributed web crawler (2016-2017): what started as a huge data analysis project turned into developing my own distributed web crawler.
10. A developer-oriented search engine (2017): I entertained the idea of developing my own developer-focused search engine.
11. A new Python interpreter (2018-2019): I developed a CPython-compatible interpreter supporting built-in and user-defined types, classes, metaclasses, functions, decorators, comprehensions, generators, exceptions, closures, imports, etc.
12. A new Python type checker/inferrer (2020-present): I'm developing a type checker capable of inferring types without the need of type annotations.
13. An automated documentation tool (2019-present): I'm developing a next-generation documentation tool for Python.

What I've Learned

Working on the above projects allowed me to learn at a much greater pace than what would have been otherwise possible.

For convenience, I distilled everything I learned into 3 categories: idea-related, startup-related and technical-related.

Idea

1. My best project ideas came organically - I didn't actively try to come up with those ideas, but instead I just followed my interests and tried to solve my own problems. My Online Toolbelt project is a textbook example of a made up idea (not an organic one).
2. Generally, I need to scratch my own itch in order to stay motivated and keep going. The technical challenges will be numerous down the road, so I don't want any single issue to completely paralyze or demotivate me. Alternatively, if my sole objective is to learn something new, then that may be sufficient to keep me motivated.
3. If I'm asked "Would you use your own tool?" and my answer is "no", then why would anyone else want to use it? However, I found that sometimes I'm biased and I may just reply "yes". The trick here is to ask myself this instead: if someone else developed the same tool, would I use it? A good example of this is my budget tracking tool. I did have a need for that tool, but I certainly didn't want to use any such tool developed by someone else due to privacy and security concerns.
4. In retrospective, I realize that I had to go through stupid ideas to develop a sense of what is viable and what isn't. A good example of one such idea was my Online Toolbelt - it was deeply unrealistic and had no chance of success. Besides, I keep reminding myself that nobody is immune to unrealistic ideas - even Paypal's early Palmpilot project and Marc Andreessen's plans for an online Nintendo platform were!
5. Personal projects are perfect to work on things that I'm genuinely curious about, but that a regular job wouldn't allow me to persue (because it would be judged irrelevant or too risky).
6. I found that focus is extremely important. It's easy to get lost in technical details, or to deviate substantially from an idea because I somehow think that I have an Even Better Idea™. I confess that I was guilty of this with my Better Customer Support Tool project and my huge data analysis project. The main issue here is that thinking that the Even Better Idea™ is really better is pure speculation. The solution to that is a mix of:
   a. working first and foremost on problems that I have myself so that I understand the problem-space
   b. working only on problems that interest me deeply
   c. testing my hypotheses more effectively by speaking early on with potential users or customers.

Startup

1. I learned to not focus too much on the startup thing. It doesn't matter whether one of my projects becomes one or not; what matters is that I don't launch a startup based on a bad idea that struggles for many years before it shuts down. For me, this was hard to do because around 2012, I read a couple of startup-related books (including Lean Startup) and I was hooked - I was literally dreaming about this!
2. While validating my ideas¹, I realized that speaking directly with people is helpful to get valuable feedback, but I need to be careful with the kind of people I talk to. If I introduce my idea to people who don't have a clue of what I'm talking about (i.e. people not in my target segment, like friends, family or even strangers), then whatever feedback I receive is at best useless and at worst dangerous.
3. As a programmer, I'm not naturally business-oriented². So when I need to substantially deviate from coding for a long time, I feel that something is missing. For example, when I was on a roll of speaking with several potential customers a day, it required careful preparation, reviewing of notes and follow-ups. I was progressing rapidly from a business perspective, but in my programmer's mind, I wasn't creating anything of value. The solution I found is to mix these business-related tasks with some coding on the side (for projects that may or may not be related).
4. Building an MVP (minimum viable product) is a great tool to quickly rule out stupid ideas. Sometimes, I don't even need an MVP though - I just need to explain my idea to somebody who's knowledgeable enough (about my target segment).
5. I realized that recognizing when to pivot is very important. Sometimes, the current form of an idea isn't viable or realistic, but tweaking it just a little bit may bring it a little bit further. At other times, though, knowing when to completely give up on an idea or project is even more important. This is exactly what happened with my Universal Connector Framework, my Online Toolbelt and my huge data analysis project³.
6. I learned that startup accelerators/incubators are valuable to learn the basics of startups and entrepreneurship, but they can be a huge echo chamber and they can bring you in undesirable directions. All that disruption and seeking-investment-at-all-costs talk is pretty damaging, honestly, because you start focusing on the wrong things.
7. Reading books allowed me to understand so many things about business and startups in general, be it about customer development, psychology, marketing or business law. There's just no way that I could have come up with those insights on my own.
8. Going through a succession of failures (i.e. ideas that don't work) can be really depressing at times. I found that the best way to circumvent that feeling is to work exclusively on problems that I personally have and that I'm genuinely curious and passionate about. Hence the motivation will follow and I won't feel down if it ever fails based on the unwritten rules of startup success. Because, well, I'm having fun, right?

Technical

1. I've noticed that the kinds of projects that pique my interest are increasingly more complex and challenging. For example, I started working on simple things like a time tracker in 2011 and now I'm working on a static type checker for Python. I'm certainly not doing that on purpose; instead, I suspect that this is a consequence of me feeling much more comfortable with uncertainty and becoming more confident in my abilities.
2. To be productive, I generally stick with my favorite programming language and I use what I already know. If nothing I know allows me to do what I want, then I just learn whatever I need and I use it. For example, I initially just knew C++ and C#. While I used the latter for my first few projects, I then realized that web-based projects would need something better than ASP.NET. So I learned HTML, CSS, JavaScript, Python, Flask and React.
3. Building things from scratch is useful when I want a deep understanding of how things work under the hood. However, in a regular project (or in the context of a job), it very rarely makes sense to build things from scratch, unless they're an intrinsic part of the project's core functionality or if no alternative exists. A good example of this was my Universal Connector Framework.
4. I found that designing good-looking user interfaces is hard, very hard. It may be fun and challenging, but it's often not worth the effort, as it's not my specialty. So while I was able to develop a good-looking website for my Toastmasters club, I decided to stick with React Bootstrap for my budget tracking tool.
5. As primarily a Windows developer, it's easy to become rusty with Linux. So I found that the best solution is to a) deploy my projects to Linux VMs and b) regularly use WSL on the side. The end result is that I build cross-platform projects without too much effort.
6. Writing requires an entirely different set of skills than programming. It's why I started this technical blog and why I maintain a personal programming journal. I found that the habit of writing is a great way to clarify my thoughts and to keep track of my findings and design decisions. Simultaneously, it's a great way to share what I'm working on with other people.
7. I learned that using low-level tools (e.g. WinDbg or gdb) is not much harder or scarier than using an IDE. Over time, using them becomes second nature.
8. I've become much better at evaluating a project's requirements and at scoping things, thanks to working on so many personal projects (e.g. deciding whether something is a must-have or a nice-to-have).
9. I found that spending more time on analyzing a problem (or a new technology) is almost always worth it. My understanding of the said thing increases substantially, and I thus reduce the technical (or business) risk dramatically.
10. Brainstorming can be quite challenging when working alone, but I found a very effective solution. First, I write down the problem at hand in the clearest way possible. Second, I write down every potential idea or solution as a numbered list, along with the pros and cons of each one. Finally, I pick the best idea or solution, typically the one with the fewest cons. This approach allows me to really think through the problem.

Conclusion

If you had told me 10 years ago the projects that I'd be working on in the future, I probably wouldn't have believed you, thinking that many of those were sort of inaccessible.

This is the magic of personal projects - you never know where they will lead you. And if you persevere, things will probably work out better for you than if you hadn't started anything.

If I had one piece of advice to give to my early 20s-ish self, it would be: "Start a personal project now!" Of course, I'm far from being done. After all, personal projects are for me a way to push my boundaries and learn new things.

So what does the future hold for me? I can't tell you for sure, but I certainly love what I'm currently working on.

Update (22/03/2021): this post was featured on Lobsters, Reddit and Hacker News. Thanks to everyone for your support and feedback!

Disclaimer: this post contains an Amazon affiliate link that will help me pay for the hosting of this blog.