Market data pipeline
Every week I look at how big traders are positioned in currency futures. I wrote a Python pipeline that pulls the CFTC and CME data and builds those tables for seven currencies, so I don't have to put them together by hand.
Then I wrote the setup to run it in the cloud, so it doesn't need my laptop to be on. It can run on a schedule, send the new tables to my phone on Telegram, and tell me when something breaks.
- Python
- Docker
- Terraform
- AWS
- GitHub Actions
- Telegram bot
- 1I push code to GitHub
Tests run, then a Docker image is built - 2The image goes to AWS
Stored in ECR, rolled out to an EC2 server - 3The server runs the job
At 11:15, 12:30 and 23:30, Lagos time - 4Results land on my phone
New tables, failure alerts, and /run to rerun it
41 automated tests. No open ports on the server, and no AWS keys stored anywhere.