INDUSTRY: Retail

LOCATION: Bellevue, Washington

PREVIOUS SOLUTION: None

 

REQUEST A DEMO TODAY

    OfferUp saves big with CompilerWorks Lineage

    Company Overview

    According to Forbes magazine, OfferUp leads the race to reinvent Craigslist. The Seattle-area unicorn’s mobile app-based marketplace is one of the fastest-growing channels for people buying and selling almost anything.

    Challenges

    OfferUp’s data warehousing team had migrated 100 upstream tables from PostgreSQL to Amazon’s fully managed and highly scalable DynamoDB database service. However, the downstream consequences of the move meant the team had to quickly and accurately refactor data warehouse, ETL, and BI workloads. This included over 200 Airflow jobs and 2,000 Tableau visualizations.

    The prospect of tracing all of the dependencies between Airflow, Tableau, and the Postgres tables was daunting. OfferUp dedicated seven data engineers to the task, but it quickly became apparent that the tight timeline and high priority of the project made a manual approach untenable.

    The Solution

    OfferUp’s team understood that speed and accuracy were essential to completing the project on time. They turned to CompilerWorks for help.

    CompilerWorks Lineage solution automatically generated column-level lineage information for all of OfferUp’s data processing activities. Lineage takes input from data processing logs, data processing code, BI reports, and ETL pipelines to produce a unified model: the lineage fabric. For OfferUp, the fabric mathematically represented all their data processing code.

    CompilerWorks Lineage enabled OfferUp to complete their migration in record time, shaving months off the projected timeline. In addition, Lineage was able to identify individual database columns unused by any downstream processes. The team was able to retire these fields, saving resources, improving optimization, and reducing complexity. 

    The Results

    CompilerWorks rigorous automated approach gave the OfferUp team confidence. Using CompilerWorks Lineage meant the migration to DynamoDB had a level of accuracy unrivaled by any other software or manual method.


    CompilerWorks Lineage enabled OfferUp to complete their migration in record time, shaving months off the projected timeline. In addition, Lineage was able to identify individual database columns unused by any downstream processes.


    Automatically analyze, convert, and optimize enterprise data processing code

    Learn more about how our two core applications can change how you migrate and maintain enterprise-wide data processing.