This article was written by Jackie Davalos and Dina Bass. It first appeared on Bloomberg Terminal.
When software developer Nikolai Avteniev released a preview version of Microsoft Corp.’s Copilot coding assistant in 2021. When he got his hands on it, he quickly recognized its potential.
The assistant was developed by Microsoft’s GitHub coding platform and was based on a version of OpenAI’s generative artificial intelligence. He wasn’t perfect and sometimes made mistakes. But Avteniev, who works for ticket seller StubHub, was surprised at how well it completed lines of code with just a few prompts. All he had to do was press tab and Copilot would fill in the rest.
“Instead of 15 keystrokes, there were three,” he recently recalled. “A little burst of speed was nice.”
Three years later and now powered by the latest version of OpenAI’s GPT-4 technology, GitHub’s Copilot can do much more, including answering questions from engineers and converting code from one programming language to another. As a result, the assistant is responsible for an increasingly larger part of the software written and is even used to program critical systems in companies.
Empower your work with monthly insights into Enterprise Data & Technology
Sign up for the newsletter
Along the way, Copilot is gradually revolutionizing the working lives of software developers – the first professional group to use generative AI on a mass scale. According to Microsoft, Copilot has attracted 1.3 million customers so far, including 50,000 companies ranging from small startups to corporations such as Goldman Sachs, Ford and Ernst & Young. Engineers say Copilot saves them hundreds of hours per month by tackling tedious and repetitive tasks and frees them up to focus on more complicated challenges.
GitHub was acquired by Microsoft for $7.5 billion in 2018 and dominates its market. The company is betting that Copilot has the AI it needs to combat rival services like Tabnine, Amazon’s CodeWhisperer and Google-backed Replit Ghostwriter. GitHub’s AI assistant is also a kind of beta test for a variety of other copilots that Microsoft is building into Office, Windows, Bing and other areas of the business.
As with AI in general, GitHub Copilot has limitations. Developers say it sometimes pulls outdated code, provides unhelpful answers to questions, and generates suggestions that are broken or may infringe copyright. Because the tool is trained on public and open code repositories, engineers run the risk of reproducing security issues or introducing new ones into their work, especially if they blindly accept Copilot’s recommendations.
GitHub emphasizes that the tool is an assistant, not a replacement for human programmers, and has placed the onus on customers to use it wisely. Solid policies are needed to prevent lazy programmers from simply accepting Copilot’s suggestions, said GitHub CEO Thomas Dohmke. He expressed confidence that the engineers would remain honest with each other.
“The social dynamics of the team will ensure that those who cheat by accepting code too quickly and not actually going through the process defined by the team will not make it to production,” he said in an interview.
Generative AI is the latest in a long line of innovations that have transformed computer coding over the years. Over the last century, program compilers accelerated software development by quickly translating instructions into ones and zeros that computers can understand. More recently, Linux has popularized open source coding, allowing programmers to leverage other people’s work rather than writing everything from scratch.
Programming assistants like GitHub’s Copilot could be even more revolutionary, as generative AI has the potential to automate large parts of what software developers currently do.
At the moment it mainly makes them more efficient. StubHub’s Avteniev, who also teaches software engineering at the City College of New York, says Copilot’s predictive ability helps programmers stay “in the flow” because they no longer have to stop to look things up. Avteniev has been programming for more than 20 years, but even he sometimes forgets programming languages, forcing him to waste time Googling them. “Copilot prevents you from having to stop your current coding process,” he said. “Even if it produces nonsense, it’s still easier to just accept what it’s doing and then correct it yourself.”
Aaron Hedges, a developer for more than 15 years, was already burned out before Copilot arrived. Hedges works for ReadMe, a startup that helps companies create technical descriptions of their application programming interfaces, or APIs. Like Avteniev, he makes good use of Copilot’s autocomplete feature. “Because I’m a pretty experienced engineer, I can look at this and say, ‘Oh yeah, that looks right.'” He also likes that he can ask questions without leaving his programming window. “I don’t have to go away and open a browser, which can be really disruptive,” he said.
At $10 a month, a Copilot subscription is a bargain that Hedges is happy to pay for himself. After work, he creates websites for Dungeons & Dragons fans. When you have a toddler and another baby on the way, free time is precious. “These two hours that I have in the evening to program are super important to me,” he said. “The more efficient I can be, the better.”
Few tasks are more tedious than debugging software – a process that can take up to 50% of an engineer’s time. According to Figma, which helps developers design app or website interfaces, Copilot can create bug testing programs in minutes instead of hours. “This is the true value of AI,” said Abhishek Mathur, the company’s vice president of engineering. “It doesn’t replace our work, but gives us time to develop creative solutions.”
Some companies are starting to use Copilot to create code for critical systems. Brewer Carlsberg is writing code for an existing tool that supports the field sales force in planning, preparing and documenting sales discussions. The beer maker is aware of Copilot’s limitations and uses its own quality assurance process to verify that the code it creates works as intended, according to Chief Information Officer Sarah Haywood. At some point, she said, companies might outsource this task. “Over time, people will build more trust in AI,” she said. “I don’t think we need to double-check everything AI does, otherwise we won’t add any real value.”
Discover, capture and access data.
To assess the technology’s accuracy, Canada’s University of Waterloo published an experiment last year. The researchers collected a data set consisting of code snippets with known errors and fixes for those errors. The researchers asked Copilot to create these exact snippets to see if it would spit out the erroneous versions. The assistant replicated the faulty version 33% of the time, less often than a human. In a quarter of the cases, the AI spat out the fix code. “Copilot was generally better at avoiding basic errors than more complex ones,” said Mei Nagappan, a computer science professor at the school and one of the study’s authors.
“The analogy here is that we are currently in the age of driver assistance and not yet at the autonomous driving stage,” he said.
Software developers are slow to change their work habits. Many welcome Copilot but are concerned about relying on it too much. A recent study funded by GitHub found that developers accepted the wizard’s suggestions only 27% of the time.
Engineers can also be quick to blame Copilot when something goes wrong. When Etsy’s website briefly crashed last October and December, some of the company’s developers blamed Copilot for the outage. Etsy confirmed the incidents but denied that Copilot was responsible. “While we certainly understand engineers discussing how Copilot could theoretically play a role in failures or issues, we have no evidence that the tool has actually resulted in any customer impact,” a spokesperson said.
Copilot is expected to improve significantly in the coming years. GitHub is already rolling out improvements, including an enterprise version that can answer questions based on a customer’s own programming code. This is intended to help new engineers get up to speed and allow experienced programmers to work faster. In the coming months, GitHub will also give engineers the ability to use their employer’s own codebase to help autocomplete programs they are working on. This makes the generated code more individual and helpful.
GitHub cannot afford to sit still. At least a dozen startups want to revolutionize the market. Some are using new models that have dramatically increased the amount of information code wizards can quickly access, making it easier for them to create entire programs. “An AI programmer who can see all of your code will be able to make much better decisions and write much more coherent code than one who can only look at your code through a paper towel roll, one small amount at a time. “,” said Nat Friedman, an investor and former GitHub CEO.
Friedman is backing a startup called Magic AI that plans to create a “superhuman software developer.” The company Cognition AI, supported by Peter Thiel, is now working on an assistant that can work on software projects independently. Princeton University released an open source model for an AI software engineering agent this month, and it seems not a week goes by without a new startup popping up.
In interviews, few programmers expressed fear that AI will replace them. Like many industries, they say, automation will free them to focus on more challenging and interesting tasks. But Jensen Huang, CEO of red-hot AI chipmaker Nvidia Corp., has a less rosy outlook. He recently predicted that the programming profession was doomed. Now that AI makes it possible to program in plain English, anyone can become a programmer, Huang said.