PyCon Balkan 2018 Talks
TLK PyCon Talk
Processing billions of events with Python in real time
In this session will present how we approach the hot/cold processing data pipeline design pattern here at Warply. Our infrastructure can scale into processing billions of events only with pythonic open source libraries without any cloud PaaS implementation.
Hot processing constitutes of a tornado, kafka, Dask/Celery microservices and cassandra/redis, stuck. While our cold storage performs the entire machine learning model training and validation with keras and tensorflow.