October 2023 Summaries

Codeium Chat, a tool already integrated into Visual Studio Code and JetBrains IDEs, is now available for Eclipse, providing developers with a ChatGPT-like experience directly within their IDE. This integration is particularly beneficial for Java developers due to the language's verbosity, allowing for enhanced code explanations, docstring generation, language translation, and function refactoring. The chat feature complements Autocomplete by assisting with more complex or undefined tasks, leveraging Eclipse's built-in visual real estate for a chat panel. The tool already supports tens of thousands of developers who send hundreds of thousands of chat messages daily, indicating its effectiveness and popularity among users.

August and September saw significant enhancements to the Chat experience, particularly in VS Code, where improvements included introducing separate chat histories for different workspaces, allowing users to seamlessly switch between them. The update also enabled multi-window Codeium Chat, facilitating parallel threads of thought. Additionally, the formatting of code blocks in Chat responses was refined to address earlier issues with markdown handling, ensuring better presentation of inline variable and method names. The platform now boasts hundreds of thousands of developers, with further improvements anticipated, as the team expresses gratitude for user support and optimism for concluding the year successfully.

The text critiques the effectiveness of certain metrics used to evaluate AI code assistants, emphasizing that statistics like acceptance rates and percentages of AI-generated code can be misleading due to the unique nature of software development across different contexts. It suggests that qualitative feedback and detailed analytics dashboards that provide transparency are more valuable for assessing the real impact of these tools for individual users and enterprises. The text also describes an evaluation method for an autocomplete language model that involves using public repositories to find and test functions, simulating the completion of deleted snippets, and running unit tests to assess performance. It argues for a data-driven approach in rollout processes and underscores the importance of balancing various metrics, such as latency and bytes completed, to ensure that users derive more value from the evolving autocomplete system. It concludes with a promise to address further questions in a follow-up blog post.