I have just tried GitHub Copilot with Clojure and here is what I think about it.

Editor Integration

I decided to give Copilot a try because a few friends of mine spoke really high about it. When I wanted to integrate it into my Emacs though, I saw that there’s no official release for it. Luckily, some great people built this unofficial plugin, and I was ready to use Copilot in my Emacs within minutes. There was just this one insistent warning about Copilot not being able to infer indentation offset for Clojure mode that kept popping up, which I still do not know how to fix, but as it didn’t really effect my usage and I was just trying Copilot out, for the trial period I added a line to my config to avoid seeing that warning as a temporary solution.

(setq copilot-indent-offset-warning-disable 1)

Intro

While I’m writing code, I am often in either one of these 3 states:

1. I have no idea what to write and I need to look something up

  • While writing a map with translations and you need to add a translation for a language you don’t speak. Example:
{:title
 {:en "Chess"
  :tr "Satranc"
  :fr ""}}
;; I'm supposed to add French translation but I don't speak French, so I have to look it up
  • Another example is I would need to write a script to sort my response from ElasticSearch, but I don’t really know how to write a script in Painless so I need to read some documentation.

2. I have a pretty good idea on what to write, I just need to think for a few seconds.

  • Imagine you need to transform some data and you need to think how for a brief moment.

3. I’m pretty sure on what to do, I’ll just click on a key on the keyboard.

Copilot

In my experience, Copilot was really useful for the situations in case 1. It provided very good translations that when I sent it over to product team for a review, they edited only a few lines in a one hundred line translation document.

However, the situation is different for case 2 and 3.

First, for situations in case 3, Copilot is just confusing me. It’s like you are pretty sure on what to do, and someone keeps telling you “Nah, don’t do that! Do this”, and they don’t even tell why your idea is not good enough, and there is no clue their idea is good, but they are very confident. Inevitably I read the code it suggests, and it hinders my thinking process, and when that happens too often, it starts being annoying and decreases the productivity.

And for case 2, when I write down the name of a function, Copilot fills that in. That sounds good, but the problem with that is, now instead of writing the code myself, I have to review Copilot’s code. Although it is sometimes fine, it is not rare it suggests a code block with bugs in it, and I end up editing the code, and that’s more time and brain power consuming than me writing the code without any suggestion myself. Maybe only for generating tests, would I prefer CoPilot for this case. Per my experience, it did a good job generating the tests I named and saved me some time.

All in all, although Copilot proved itself helpful in case 1, for situations in case 2 and 3, which I’m in most of the time while writing the code, I feel like it’s not being helpful and decreasing my productivity. As a natural result of this, I won’t be renewing my subscription to it. Though I plan to retry it next year to see how it performs then.

Last words:

  1. I read somewhere it is possible to configure Copilot to filter its suggestions based on a confidence level with a custom script that you can integrate to your editor so that I’d maybe get very simple advices that I can validate easily in a reasonable time with minimal risk (scenarios in case 1), but I didn’t pursue it as I don’t think it worths the effort as of now.

  2. I tested it only with Clojure, so maybe it performs better in languages in which it’s fed with more data.