DevOpsDays Berlin 2013
It was the end of 2011 - I apply to a new company. System Administrator position. Inside of this company - I would stay for the next 4 years. The only reason why I leave this company, I leave Bosnia and Herzegovina. Like most of the youth people who don't want to live in Bosnia and Herz - simple to say there is no future with the current political system and process. I describe the situation in Corruption in RS and how IT was destroyed because of a lack liberal market.
So back to story. Inside of this company - I got 2 more co-worker system administrators. One of them was team-lead. Complete decision and the rest of the process were under one person - team leader. The company was doing a content delivery network. One of the first in the world. Ipercast. From there start Akamai CDN. Anyway - so the position of sysadmin simple described: just process(add user, troubleshooting, etc) at that time. So no coding (little shell script but not more than that).
So after 5 months, we got visited by the owner of the company. In front of the complete company, he tells us that our BU(business unit) in Bosnia and Herz would be closed in the next 3 months. One of the smartest people in the room asked: "what we could do to change the situation?"
He was a front-end developer. The answer is "little or nothing".
In that time - I did not know that most of the software project for back-end was rejected by our team leader. He did not like coding/programming. He was bad at that. Next 2-3 weeks after this meeting, one of my CTO try to ask me could I build some shell script that would make the database/list and send it to some HTTP API. He asks me because the rest of the team (other 2 guys) did not want to spend time on that.
After the script was done - he asks me to review some projects that the company has. I did not know what was talking about - so I ask him to show me. In a moment I got the idea that we have outside BU that bring us projects - but no one in the company (actually sysadmin team) did not want to deal with it. Reason: The team leader was not able to do programming.
My CTO picks up the most paid project from the list. It was a huge list of TODO. He explains to me that we already fail this part once. Ex-CTO (a co-worker of my CTO and my ex-CTO) tries to build one project in PHP and it fails. He has 12 years in this company and he could do many things that complete my sysadmin team. But also he did not find a proper way to build this project. It happens. People get tired of duty or get pissed off by owners, or just want to leave the company and don't care at all.
So, I review things: he uses PHP, Apache. Already it makes me cry. I move to Lighttpd. Try to build a different approach. Then I got an idea. Why not build the C plugin/module for Lighttpd? Why? Simple it is faster than PHP. It has better and faster work.
The task was like this: provide unique access to some file with timelimit and IP limit.
In 2 days, the demo was done.
Germans BU ask the company that orders this project to start testing. After testing they just ask me how much time I need? I was not sure so take 1-3 days to review complete TODOs and to get an idea about the project. Simple: Download files but each is encrypted with unique key as URL also that has limited values(time, IP, start limit), with token. Also, preview files as streaming services.
I told my CTO we need like 2 months. And he responds with taking 6 months. I was like ... ok.
German BU shows my results of testing:
- 100% unique token URLs
- 50 clients - 26 ms response
- 25 clients - 25 ms response.
and before this was:
- 60% unique token URLs
- 50 clients - 3-sec response
- 25 clients - 1 sec response.
This solution was made in PHP and speed as to how it was worked - was not so the best way to do.
On this demo - my company earns half a million euros for development and we stay in the game. Later it would be used in different companies (RAI IT, Sony Germany, etc) for streaming, live streaming - all well know the platform.
DevOps thinking and implementation
DevOps have 2 parts: tools and processes. Most of the process comes as logical, as must be like this. Some processes got improved later.
From the start - we implement the best way to move from test server to production. So we optimize all parts: building, testing, moving to production.
It was natural to be done like that way. I did not saw the issue. The only thing we did not do is rollback. But we did have commit inside of git so we could reverse to old code. Also moving from test server to production - after client review - was manual. Why? Simple on the start if goes wrong I want to know ASAP and stop updating. We use the load-balancing and 2 servers.
It was June 2013 - the project was finished and the company was saved. As also 25 working places.
In the first 24h, there were 150 million unique IPs.
One of the rare stories that I am proud of. I use all my skills to get this product to Vodafone Germany (yea, we did not know until last day). C/Java/Shell scripting, Cryptography, Linux engineering, network, NFS, rtmp/rtp/htps low-level manipulation, and others. It was a really good product that was in usage for a long time. One thing I did not mention: after I finish the complete project they told me this solution would be used for complete Germany. Then it was used for Vodaphone - and they show me all documents from other companies who apply: Akamai, Amazon, Leaseweb, MaxCDN, and 10 more.
My solution was 100% that they asked. The hardest part was encryption on the fly. Each time file is downloaded to the client - it has a unique key for decryption. So mp3 was not possible to share without a key. With all my Linux engineering tricks we got ~400 files per second. And it was not bad at all.
Later, thinking and approach would be part of DevOps. Many things - but still a good way of delivering code and services.