Tracking Down Bugs in Tens of Millions of Lines of Code in 5 minutes

TLDR; Find a unique bit of text that’s near or related to your issue, google it, and keep thinking about if the result really relates to your issue.

You’re a Mozilla fan, you open a new tab and see an ad for the Lockbox, their new password storage app! (well I did and I can’t get it to show up again, here’s what it looks like)

So you eagerly put in your email address and hit refresh while waiting for the mail to show up, it finally does!

You click on the “Get It on Google Play” button and it goes to…

The Apple App Store?

Going back and clicking the iOS icon in the hope of maybe them having interchanged them doesn’t help. The links are different but they both go to the apple store.

So at this point I’d feel a mild panic, I’ve just hit go on a marketing push that’s going to send everyone looking for an Android app, to the itunes store.

Where is the bug?

The broken link was in the email that came to me. I’ve no idea what Mozilla uses to construct and send those mails. Looking at the urls ”https://click.e.mozilla.org/?qs=733be7fbb3eb288b9bce0582db5c24d7dc74f4c6a129dcd4345e1a94c269cfb9bb2f94cac2a95379981d5e67df59db83156453c2f59441f2” it looks like they’re custom for analytics tracking and also the iOS and Android ones are different from each other.

What do we know?

Whatever system they use to send that email, it’s likely open source itself. There’s a good chance there’s a template somewhere for it that anything that refers to it will use.

So let’s find the mail system!

Looking at that email again, for an easily googlable phrase the subject stands out pretty well! “Your Lockbox download link” is likely going to be an usual term. Even if it was common, there’s plenty of copy in the mail to search on if we get a few million irrelevant results.

Since we’ve assumed it’s going to be open source, we can cut down on false positives even further by only searching Github.

So let’s type it in https://github.com/search?q=Your+Lockbox+download+link&type=Code

Not a 100% match but already we can see we’re on the right track. The first result is in the mozilla/bedrock repository. Which makes sense. However the filename is “whatsnew-fx64-base.html” which means it’s likely just a description and not in any kind of mailer that’s going to be sent when you ask for a link.

Scrolling down a bit more I came across a possible result. This has HTML and references a mobile app, it’s also in a path called templates.

Diving into that file, ctrl+F Android and we see:

”{{ googleplaybutton(href=’https://app.adjust.com/kq8zhhz’) }} ”

Ok, now let’s see if that goes to the Google App Store aaaannnd NOPE. It goes to the Apple one! We’ve found our issue! (or one instance of it)

Reporting the bug

Now that we’ve found it, it’s time to let someone know by creating a Github issue.

Opening up an issue in the repository of the file with the wrong link, we add one more bit of detail. Copying the permalink gives you a direct pointer to the issue and when you paste that entire link into a github issue, they expand it like so.

Have we really found it?

One thing we can do to check if we’ve caught all instances of it, is to use this much more unique url segment and search again to see if it shows up anywhere in the codebase, we search restricting the scope to only the mozilla user on github https://app.adjust.com/kq8zhhz user:mozilla

https://github.com/search?l=&q=https%3A%2F%2Fapp.adjust.com%2Fkq8zhhz+user%3Amozilla&type=Code

The only results are two what’s news and the file in the template folder we’ve reported! Looks kinda okay. Of course the issue could still be elsewhere but we’ve done what we could.

Epilogue

Turns out, it wasn’t really the kind of bug I thought it was! But it did still reveal unexpected behavior. If I’d clicked the google app badge to share the link to someone else I’d have been pretty confused! Collaboraton is really important to get to the bottom of real world complexity.

Intrested in hiring a Senior Android Developer or have suggestions for a topic I should cover? Send me a dm at @AniketSMK or email me at hello@[firstname][lastname].com

Written by Aniket Kadam - building useful things. Follow me on Twitter