July 2023 Summaries

In this paper reading session, Sally-Ann DeLucia and Amber Roberts discuss the paper "Improving Language Model Retrieval with Query-Aware Contextualization" by OpenAI's team. The paper focuses on improving retrieval performance in large language models (LLMs) by manipulating the context given to them. Key takeaways from this discussion include: 1. Encoder-decoder models have a bidirectional encoder that allows for better understanding of context based on preceding and future tokens, which can be leveraged to improve retrieval performance in LLMs. 2. Placing the query or question before and after the document can significantly improve retrieval performance in LLMs. 3. The architecture of transformers may change as more research is conducted into understanding how these models use context. 4. Pushing relevant information to the top and returning fewer documents are promising strategies for improving retrieval performance in LLMs. 5. Observability tools can be helpful in understanding how these models use context and can aid in experimentation with different architectures.

The collaboration between Snowflake and Arize aims to enhance the machine learning (ML) toolchain by streamlining data access, analysis, and insights. This partnership enables customers to use Arize's advanced AI observability features with Snowflake's simplified data management capabilities. By integrating these two platforms, users can create a fully manageable, simple, and scalable data pipeline that automatically extracts model insights and boosts ROI without compromising security or governance standards. The integration also allows for real-time, continuous monitoring of ML models to ensure optimal performance and proactive troubleshooting.

Orca is a 13-billion parameter model that learns to imitate the reasoning process of large foundation models (LFMs) like GPT-4, surpassing state-of-the-art models by over 100% in complex zero-shot reasoning benchmarks. The paper addresses challenges faced by smaller models such as limited imitation signals, homogeneous training data, and lack of rigorous evaluation. Orca leverages rich signals from GPT-4 to enhance model capabilities and skills through learning from step-by-step explanations generated by humans or advanced AI models.

Mark Scarr is the Senior Director of Data Science at Atlassian, where he leads the Core Machine Learning Team. The team works on various projects across the organization, including marketing and growth/product teams. They have worked on recommendation engines, propensity modeling, and generative AI space. Atlassian's primary machine learning use cases include harvesting keywords for performance marketing and search bidding optimization, customer lifetime value modeling, and building a framework to augment existing keyword pools. Cloud migration has been a game-changer for the business opportunities, providing richer data sets for model training. The team collaborates closely with business stakeholders and analytics teams to ensure models are aligned with business objectives. Mark Scarr is excited about the adoption of LLMs and their potential applications in Atlassian's products, particularly in areas like text summarization, auto-completion, and generating tickets in Jira. He emphasizes the importance of flexibility, adaptability, and embracing new technologies in machine learning. The team is always open to hiring candidates with robust backgrounds in different realms of machine learning.

In this paper reading session, we discussed "GLoRA: Parameter-Efficient Fine-Tuning for Vision and Language Models" by Zhang et al. The main takeaways from the paper are as follows: 1. GLoRA is a parameter-efficient fine-tuning method that builds upon six previous efficient fine-tuning methods, including LoRA, AdapterFusion, VPT, Scaling & Shifting features, and RepAdapter. 2. The main advantage of GLoRA over other fine-tuning methods is its ability to both fine-tune the weight space and the feature space, addressing some limitations of previous methods. 3. GLoRA can be easily expressed as a unified mathematical equation, allowing for an expanded search space without significantly increasing the number of parameters. 4. Experimental results show that GLoRA outperforms other fine-tuning methods in terms of performance and efficiency on both vision and language tasks. 5. The main benefits of using GLoRA are its flexibility, adaptability to a variety of tasks and data sets, and the ability to make more nuanced adjustments during fine-tuning. 6. However, there is still room for improvement in terms of reducing training time and exploring new domains for GLoRA. 7. The paper also highlights that parameter-efficient fine-tuning methods like LoRA and GLoRA are becoming increasingly popular due to their ability to save money and time while achieving better performance than traditional fine-tuning methods.