S3 Glacier Migration Program
Designed and executed pilot program migrating Basin's RAW storage to Glacier IR, delivering $2.9M annual cost avoidance.
The Challenge
Basin's RAW data storage costs were growing proportionally with data ingestion volumes, representing one of the largest line items in the platform's infrastructure budget. The current S3 Standard storage class was over-provisioned for actual access patterns—analysis showed that the vast majority of RAW data was rarely accessed after initial processing, yet it was being stored at premium tier pricing. With data volumes scaling alongside the platform's 965% growth, this inefficiency was compounding rapidly and needed a solution that could reduce costs without impacting query performance or SLA commitments.
My Approach
Designed a controlled pilot program across three strategically selected test regions, each representing different data volumes and access patterns. Built comprehensive monitoring dashboards to track retrieval latency, access frequency, and error rates throughout the migration. Conducted detailed access pattern analysis to validate that Glacier Instant Retrieval met the performance requirements for downstream consumers. Established clear rollback criteria and SLA validation checkpoints before recommending full production rollout across all regions.
Technologies & Tools
Related Projects
Basin: Amazon Security's Data Lake
Platform processing 9PB daily from 350,000+ sources supporting ML workloads and security analytics across AWS.
Fangorn Coral Collector Deployment
Deployed security log collectors on 350,000+ hosts across all AWS regions, achieving 97% security coverage with <1% CPU impact.
Want to discuss this project?
I'd love to share more details about my approach and results.
Get in Touch