December 2022 Summaries

Document understanding algorithms leverage an encoder-decoder pipeline that integrates computer vision (CV) and natural language processing (NLP) techniques to analyze documents, where the CV component treats the document as an image to create a representation for processing by a transformer. Vision transformers (ViTs), an emerging alternative to convolutional neural networks (CNNs), split images into patches, convert them into linear embeddings, and feed these into a transformer encoder, offering advantages such as better global relation grasp and resilience to adversarial attacks, though they require more training data due to fewer inductive biases. These transformers are computationally intensive, but pre-training with large datasets can mitigate some challenges. The Hugging Face Vision Encode Decoder models and Donut models exemplify how these technologies can transform input images directly into structured outputs like JSON, which is advantageous for document understanding tasks such as processing receipts. However, unlike models such as LayoutLMv3 that use preprocessing, Donut's direct conversion lacks bounding box information, limiting location context for extracted data. The Unstructured team is developing pipelines using Donut for extracting structured data from documents and plans to release these models soon on platforms like GitHub and Huggingface.

Document understanding algorithms leverage an encoder-decoder pipeline that integrates computer vision (CV) and natural language processing (NLP) methods to analyze document content, treating documents as input images to produce representations for multimodal transformers. Vision transformers (ViTs), which resemble NLP architectures like BERT, have emerged as an alternative to traditional convolutional neural networks (CNNs), offering advantages such as enhanced global relation grasping and resilience to adversarial attacks, though they require more training data and computational resources. HuggingFace's Vision Encoder Decoder models incorporate ViTs, facilitating the development of document understanding models like Donut, which processes input images to generate structured representations without requiring preprocessing steps like OCR. Despite omitting bounding box information, Donut efficiently converts input images directly into structured outputs, such as JSON, and is part of ongoing efforts to extract structured data from receipts and invoices, with models soon to be available on platforms like GitHub and Huggingface.