Company
Date Published
Author
David Millman
Word count
1467
Language
English
Hacker News points
None

Summary

The process of extracting ASCII files with fixed (csv/tsv) and variable (EDI/HL7) formats into a supported destination involves determining the input or load options based on the scenario, automating the transformation logic in dbt, and running the sample projects in Fivetran. The final target is a table called 'EDI_DOCUMENTS' that is configured in the transformation dropdown. To achieve this, one must create a pipeline that creates a table called EDIRAW in the schema EDITEST_AUG31, update the project to use a different source file, add new segments, and refine the UNION statements. The dbt project files need to be altered, including model files, segment mappings, and data files. VSCode can be used to run, modify, and test the dbt project in a standard developer environment without having to configure Fivetran. The samples provided are for demonstration purposes only, not meant for production, and further work could be performed to make use of other Fivetran and dbt related functionality.