May 2014 Summaries

The concept of self-describing JSONs, proposed by Snowplow Analytics, aims to address gaps in data schema knowledge by embedding schema information directly within JSON data, thus simplifying schema evolution and historical data management. This approach involves enhancing JSON Schema with a "self" property that details the vendor, object type, format, and version, and modifying individual JSON objects to include a "schema" field that uniquely identifies the associated JSON Schema. This methodology facilitates a two-pass validation process, ensuring that JSON instances are verified against their claimed schemas. The initiative is inspired by the need for clearer documentation and validation of unstructured events and custom contexts, with potential applications in programmatically generating database and schema objects. Snowplow intends to utilize this system internally and invites community feedback to refine the draft proposal before full implementation in future releases.

SchemaVer is a new versioning system proposed by Snowplow for data schemas, diverging from the traditional Semantic Versioning (SemVer) used in software development due to the unique requirements of schema versioning. While SemVer focuses on versioning software changes with increments based on compatibility, SchemaVer introduces a MODEL-REVISION-ADDITION structure to address the backward compatibility of data schemas with historical data. The MODEL is incremented for breaking changes incompatible with historical data, REVISION for changes that may affect some historical data, and ADDITION for changes fully compatible with all historical data. Unlike SemVer, SchemaVer uses hyphens to separate version numbers and starts from version 1. This system is designed not only for Snowplow's internal use but also for broader applications, such as in JSON Schema versioning, RESTful APIs, and other document-oriented software systems, with the draft proposal open for feedback to refine and formalize its application.