Large Language Models (LLMs) are being explored for their potential to handle tabular data tasks traditionally dominated by models like Gradient Boosting Machines (GBMs). The "TabLLM" paper investigates the feasibility of using LLMs for tabular classification by converting data into text prompts, allowing LLMs to process it as natural language. The study found that while LLMs can perform well, especially in low-data scenarios, they face challenges such as limited context length and reliance on meaningful column semantics. The experiments revealed that LLMs could match or exceed the performance of GBMs in some fully fine-tuned settings, particularly on datasets with fewer features, though GBMs remain preferred for larger, data-rich tasks due to their efficiency and cost-effectiveness. The analysis underscores the strengths and limitations of LLMs, suggesting they are a viable option for tabular tasks when data is scarce, but their suitability depends on factors like data richness and feature nature.