I work with the Facebook CDN team and (amongst other things) maintain the data pipelines that log requests, the tailers that fetch/annotate the requests, and populate the Hive tables that we used for this research and other improvements to serving content.
There is certainly a lot of infrastructure that this is built on that many small teams don't immediately have access to in other companies. Whether it's self-service hardware provisioning, Scribe logging infrastructure, tailer frameworks with checkpointing and retries (and job systems to schedule them), and large amounts of available space on Hive for experimentation. But most of the software parts are available as open source, so it doesn't need remain unavailable.
This is my first team at Facebook that I've been heavily involved in this scale of data capture and analysis, but it only took a few days to get up to speed through a combination of great tools and good documentation. Being able to drop a Python file in a code repo to ensure that some complex data warehousing task takes place every day after that is pretty powerful.
There is certainly a lot of infrastructure that this is built on that many small teams don't immediately have access to in other companies. Whether it's self-service hardware provisioning, Scribe logging infrastructure, tailer frameworks with checkpointing and retries (and job systems to schedule them), and large amounts of available space on Hive for experimentation. But most of the software parts are available as open source, so it doesn't need remain unavailable.
This is my first team at Facebook that I've been heavily involved in this scale of data capture and analysis, but it only took a few days to get up to speed through a combination of great tools and good documentation. Being able to drop a Python file in a code repo to ensure that some complex data warehousing task takes place every day after that is pretty powerful.