Converting a codebase from JavaScript to TypeScript

Convoy recently celebrated its 5th birthday. In those five years, the company has undergone significant growth, including the number of engineering employees and the size of our codebase. Like many startups, Convoy had a single repository in the beginning. As the company scaled, new repositories were added, and they were added with the latest and greatest technologies. Much of the original (and critical) code, however, continued to exist in a growing repository that lagged behind in features and best practices. One of those best practices was migrating from JavaScript to TypeScript. This was not an easy process for our main repository, and we hope others can learn (and maybe laugh a little) from our mistakes. The story of converting our codebase is one of passion, patience, and most of all, persistence.

Why TypeScript?

TypeScript is a programming language developed by Microsoft that is an extension of JavaScript which introduces typing. To understand why Convoy shifted towards TypeScript, it is important to understand why JavaScript was chosen in the first place. In 2015, the company settled on JavaScript as its programming language because of its popularity, usability, and versatility. Engineers could hop between projects with minimal setbacks since the language can be used for both frontend and backend development.

However, the inadequate code safety and the lack of documentation had caused major bugs and multiple outages. Since TypeScript was fairly new when Convoy started, JavaScript was used to reduce risk. As TypeScript matured, some engineers began experimenting with it in our codebase. Over time, the language became preferred due to its additional benefits, such as static types and improved readability. These features help engineers discover more bugs and write code with more confidence. By the end of March 2019, most engineers were eager to convert our main repository.

Before the project started, many engineers at the company considered the task monumental. Our main repository, Shipotle (a combination of “Shipping” and a burrito-making company frequented by one of our founders), had more than 75,000 lines of production code in JavaScript — many of which were legacy and business critical. To accomplish this goal, we applied our four-step migration plan:

1. Infrastructure: We added a TypeScript compiler to the build process. This allowed TypeScript files to exist in the codebase.
2. Education: We explained the benefits of the new language to engineers and we encouraged new files to be written in TypeScript.
3. Guardrails: We then blocked new JavaScript files from entering the codebase. This prevented our remaining work from expanding.
4. Migration: We explored and implemented multiple ways to convert the legacy JavaScript files to TypeScript.

After step three, with all new code being written in TypeScript, migrating files became a prerequisite for any major refactoring change. To expedite the conversion process and to split the work across engineers, an informal meeting was proposed to leadership. These “meetings” would address the technical debt in Shipotle in an incremental, collaborative, and enjoyable way. And thus, TypeScript Conversion Parties were born!

Conversion parties

The parties turned out to be hugely successful. Engineers were passionate about cleaning up the codebase and a new Slack channel was born: #typescript-conversion. Anyone could post their pull request for a review or ask questions about confusing chunks of code. With this steady effort, we began to chip away at the monolithic repository. We also started highlighting bugs, documenting best practices, and creating a giant spreadsheet to track our progress. The best part was that everyone found different reasons to attend. New employees participated because the changes were low risk and they wanted to familiarize themselves with the codebase as well as TypeScript (possibly a new language for them). Others enjoyed the opportunity to stay connected with people outside of their team. Above all else, people were motivated by the mission and the free food. We provided pizza and beverages at every party, and the parties always took place later in the workday.

Eventually, though, participation diminished and enthusiasm dwindled. People were busy with their feature work and the events were optional. Many files were also avoided for being too large, too unfamiliar, too important to mess up, or a combination of all three. Despite Shipotle being shared across many different teams and dozens of engineers, it was difficult for people to volunteer their time towards tedious tech debt cleanup. After a year, it was clear that our pace was insufficient (we were only converting a few files each week), and we would need to come up with a new plan.

Seeking other solutions

Ideas began circulating about how we would tackle the rest of the JavaScript files. One proposal was incorporating friendly competition across the engineering organization. Employees or teams that converted and merged the most lines of code could be compensated with company-wide recognition, free lunches, gift cards, or maybe even T-shirts. We ultimately abandoned this approach because many engineers at the company did not consistently use Shipotle, and thus would be adversely affected and disadvantaged by the rewards. Around the same time, people started investigating how we could automate the conversion process. We explored multiple third-party tools, and one engineer who was interested in learning more about transpilers created their own tool that could automatically infer and insert simple types. These tools, however, proved to be incomplete, and they all required manual intervention or validation. Consequently, we found ourselves persevering with the existing parties.

With the momentum of TypeScript conversion parties waning and COVID-19 forcing employees to work from home, progress on the conversion project practically screeched to a halt. JavaScript remained, though, and it was still causing problems in Shipotle. Developers worked around partial conversions, irritated as they tried to add new feature code in a safe way. Simple typing bugs went undetected such as referring to nonexistent fields or passing invalid arguments, and some linting rules (e.g. enforcing await) could not be enabled due to our mixed codebase. Yet, typing the remaining production files seemed like a monumental task.

The Shipotle Infrastructure team began to discuss how to complete the migration. Any-typing (the process of assigning ‘any’ to all variables missing a type) could quickly unlock some of the linting benefits. However, this technique would fail to identify bugs and improve developer confidence. Another approach contemplated was using automatic conversion tools. This option fell short because, as mentioned earlier, they needed supervision and nontrivial refinements. Lastly, advanced manual typing would resolve even the most complicated issues, but this method required significant effort from our developers. The following table roughly summarizes our findings from the different approaches considered:

The final push

After many discussions with engineers working in Shipotle, one engineer from the Infrastructure team analyzed how long it would take to convert a handful of files with basic typing (i.e. adding types when obvious and applying ‘any’ everywhere else). By extrapolating the time it took to convert several files to the total number of files in the codebase, it was estimated that a three-person team could manually migrate the remaining 40,000 lines of production code in approximately three weeks. The final proposal fell somewhere between any-typing and advanced typing, and included the following characteristics:

• Manually convert the remaining JavaScript files to TypeScript
• Add simple and complex types when easily identifiable
• Discover and document type mismatches or bugs
• Use the type ‘any’ for those issues and add a TODO-FIX comment

This approach would require significant development cost but it would finish the job with consistency and confidence. When approval was given, the spreadsheet for tracking JavaScript files was regenerated so the three developers could coordinate their work and progress. Excited by the momentum, other engineers began to pitch in where they could.

In the end, the final conversion push took four weeks instead of three but otherwise was a complete success. All of the production code was now in TypeScript! Looking back, we learned a few things along the way.

1. Partial conversions are inadequate. Sometimes a final push is required to reach the desirable state of completion and consistency.
2. Using tools for automatically converting types seems obvious. However, the tools will likely need oversight, adjustments, and significant investment (especially for inferring complex types).
3. Limit the scope of the project. Migrating does not mean resolving all bugs. Label some issues rather than further delay the mission.
4. Love Problems, Not Solutions (one of our company core values). To achieve success, focus on the goal rather than the process, without compromising on quality.

Finally, this endeavor could not have finished without the patience and persistence of everyone involved. Every time work stumbled and lost momentum, someone else was there to pick up the torch. Now that the migration work is complete, we can ship code at a faster speed, in greater confidence, and best of all, with fewer bugs. The seemingly insurmountable task was in fact possible and we hope this story inspires you to take on your own difficult migrations. Thank you for reading and good luck!