Company
Date Published
Author
David Millman
Word count
956
Language
English
Hacker News points
None

Summary

Fivetran provides automated date integration for HL7 files, which are similar to EDI files but with an extra twist of nested records. To load HL7 data into a data warehouse, changes are required in the Fivetran S3 connector configuration, such as selecting the pipe field delimiter instead of asterisk and enabling the headerless files option. The data is then parsed using SQL views, which extract values from the pipe-delimited column using string functions specific to each CDP platform (e.g., Snowflake's `SPLIT_PART`, Databricks' `element_at`). The parsed data can be used to derive segment types, such as the MESSAGE_TYPE field, which contains a nested record structure. This approach allows for late binding solutions, which can parse data once in the CDP landing zone without rejecting it early on, making it ideal for analytic/innovation use-cases.