LOCATION: Bellevue, Washington
PREVIOUS SOLUTION: None
REQUEST A DEMO TODAY
OfferUp saves big with CompilerWorks Lineage
According to Forbes magazine, OfferUp leads the race to reinvent Craigslist. The Seattle-area unicorn’s mobile app-based marketplace is one of the fastest-growing channels for people buying and selling almost anything.
OfferUp’s data warehousing team had migrated 100 upstream tables from PostgreSQL to Amazon’s fully managed and highly scalable DynamoDB database service. However, the downstream consequences of the move meant the team had to quickly and accurately refactor data warehouse, ETL, and BI workloads. This included over 200 Airflow jobs and 2,000 Tableau visualizations.
The prospect of tracing all of the dependencies between Airflow, Tableau, and the Postgres tables was daunting. OfferUp dedicated seven data engineers to the task, but it quickly became apparent that the tight timeline and high priority of the project made a manual approach untenable.
OfferUp’s team understood that speed and accuracy were essential to completing the project on time. They turned to CompilerWorks for help.
CompilerWorks Lineage solution automatically generated column-level lineage information for all of OfferUp’s data processing activities. Lineage takes input from data processing logs, data processing code, BI reports, and ETL pipelines to produce a unified model: the lineage fabric. For OfferUp, the fabric mathematically represented all their data processing code.
CompilerWorks Lineage enabled OfferUp to complete their migration in record time, shaving months off the projected timeline. In addition, Lineage was able to identify individual database columns unused by any downstream processes. The team was able to retire these fields, saving resources, improving optimization, and reducing complexity.
CompilerWorks rigorous automated approach gave the OfferUp team confidence. Using CompilerWorks Lineage meant the migration to DynamoDB had a level of accuracy unrivaled by any other software or manual method.
CompilerWorks Lineage enabled OfferUp to complete their migration in record time, shaving months off the projected timeline. In addition, Lineage was able to identify individual database columns unused by any downstream processes.
Automatically analyze, convert, and optimize enterprise data processing code
Learn more about how our two core applications can change how you migrate and maintain enterprise-wide data processing.