OpenAI x GitHub Copilot: Preliminary thoughts
If you missed my post yesterday, it was about OpenAI announcing a partnership with Github Copilot. OpenAI plans on releasing a new model as well as a part of the OpenAI Beta this summer called, “OpenAI Codex” which is a descendant of GPT-3.
I was really excited to hear the news, but just like last summer when GPT-3 came out, I am frustrated again because I don’t have access. This time, it’s entirely up to Github to decide who gets access, and just like before, I have no idea what their criteria is regarding who gets access and who doesn’t. I signed up for the waitlist, but I’m not sure if/when I’ll be able to access it. These concerns are still mirrored on the OpenAI corporate website to this day. I still don’t know what criteria OpenAI uses to give some people access over others, and it appears they don’t talk about it anywhere either.
It was a bit disappointing to be a part of the OpenAI beta, but not have immediate access to Github Copilot automatically. I believe this is something OpenAI could have negotiated to represent the community’s interests as a part of their deal. The community is still small and I believe Github would have the resources to handle approximately ~100K potential OpenAI beta users on launch day. It would have been fun!
I woke up and created this meme to describe how I feel:
It sucks waiting, but in the meantime, I did start mulling over the potential implications of AI driven coding. It’s too early to form any predicitions, but in broad strokes, I have many questions about OpenAI Codex:
how many parameters is the model? Was the training data processed any differently? Is it a fine-tuned version of GPT-3?
were there any interesting scaling law discoveries which differentiate code from other mediums like images or just text from books?
was the model trained with a different objective function than something like SuperGLUE? How is OpenAI evaluating code effectiveness overtime?
what is the long term plan for the model? What capabilities will it have 2-3 generations for now? Are we estimating higher quality code or straight up, no-code, few shot, text to android app generation in a few years?
I understand the model has existing limitations like single-file reading only (not the whole codebase) and character limits, but what is the plan to address these kinds of things overtime?
I’m also interested in trying the VSCode extension myself once I get access, I look forward to:
coding a VueJS app from scratch using Github Copilot
writing some flask endpoints/unit tests with it
trying to see if it can handle grunt work tasks like error handling, generating copy for error messages, and more.
I also want to test its few shot code writing abilities beyond just simple computer programs you’d see in an undergrad CS textbook. I’m interested to see how “no code” github Copilot can get.
More thoughts to come!