In the few years since its introduction, Netflix’s online video streaming service has grown to serve over 50 million subscribers in 40 countries. We’ve already shared some of the best practices that Netflix’s software development engineers adopted as they transitioned from a traditional monolithic development process to continuous delivery and microservices, in Adopting Microservices at Netflix: Lessons for Architectural Design and Adopting Microservices at Netflix: Lessons for Team and Process Design.
In this post, we’ll discuss another core contributor to Netflix’s success: its content delivery network (CDN), Open Connect. We’re proud that NGINX runs on every Open Connect delivery appliance, playing a key role in Netflix’s ability to keep pace with the explosive growth of the video service. NGINX’s Gleb Smirnoff has worked alongside the Open Connect team for over two years, and last October at our user conference, nginx.conf2014, he explained why Netflix chose NGINX (along with FreeBSD) to power this crucial part of its business.
Why Netflix Built Its Own CDN
Netflix initially outsourced streaming video delivery to three large CDN vendors (Akamai, Level3, and LimeLight). As the service became more popular, Netflix decided that building and managing its own CDN made sense, for several reasons:
- From a practical perspective, the CDN vendors were struggling to expand their infrastructure at a pace that matched the growth in customer demand for Netflix’s video streaming.
- From a financial perspective, the expense of outsourcing was quickly becoming prohibitive as the volume of streamed video increased (a challenge experienced by many popular applications and web properties).
- From a business perspective, it was clear that video streaming was replacing DVD lending as Netflix’s primary source of revenue, and it didn’t make sense to outsource a critical piece of the company’s main business.
Most importantly, Netflix built its own CDN in order to have greater control over application delivery and the user experience. To provide optimal streaming media delivery to customers, Netflix needed to maximize its control over the three basic components in the delivery chain:
- The user’s video player. Netflix already controlled this component, because its developers write all the device-specific apps used by customers to view Netflix content.
- The network between the user and the Netflix servers. There is no way to control this component directly, but Netflix minimizes the network distance to its customers by providing free video-streaming appliances to ISPs in exchange for rack space in the ISP’s data centers for housing the appliances. (Appliances are also placed at Internet exchange points [IXPs] to serve customers whose ISPs are not interested in housing third-party equipment.) Video streaming is particularly sensitive to the packet delay and loss, misordered arrival, and unpredictable (jittery) round-trip times inherent to TCP/IP, and minimizing the network distance reduces the potential exposure to these anomalies.
- The video server (Open Connect itself). Running its own CDN gives Netflix freedom to tune the CDN software to compensate for Internet anomalies as much as possible. It can run custom TCP connection-control algorithms and HTTP modules. It can also detect server and network problems very quickly and reroute clients to alternative servers, then log in to the server hardware and troubleshoot “from inside.”
Netflix was able to optimize Open Connect for video streaming in a way that’s not possible with a generic CDN provided by a vendor. Open Connect enables Netflix to offer a superior user experience at a lower cost, and with greater visibility into the performance of the application around the world.
Why Netflix Chose NGINX and FreeBSD
From the start, Netflix’s goal was, as Gleb puts it, to “get more and more gigabits per second from a single box.” Specifically, Netflix needed to maximize the number of subscribers each appliance could serve concurrently. The Open Connect engineers anticipated needing to fine-tune the software to achieve this goal, so they decided to go with open source software for its unlimited extensibility.
As mentioned previously, Netflix places its video-streaming appliances in the data centers of its customers’ ISPs when possible. Because the software running on the appliances would be in the hands of third parties, Netflix chose projects that use a BSD-style license rather than the GNU Public License (GPL).
The specific open source projects Netflix chose were:
- FreeBSD as the operating system, because it’s known to be fast and stable. The developer community is strong and willing to work with vendors.
-
NGINX as the streaming media server. Its proven speed and stability was important because Netflix wanted to launch Open Connect as quickly as possible, without the need to tweak it just to get going. Once the CDN was up and running, Netflix was able to examine traffic patterns and fine-tune the NGINX settings.
Another benefit of NGINX is that although the open source software is distributed under a BSD-style license, all of its core developers are full-time employees of NGINX, Inc., which provides enterprise-class support for its commercial product, NGINX Plus. In this regard, it combines the best features of OSS and commercial software.
NGINX’s flexible framework for running custom modules also appealed to Netflix, and the Open Connect team has created modules specific to its video streaming needs.
Combining FreeBSD and NGINX yields further benefits:
- NGINX’s event-driven design is one of the keys to its outstanding performance, and FreeBSD’s
kqueueevent notification system call is one of best APIs for multiplexed I/O. - Without any modification required, NGINX can use the
sendfilesystem call together with theaio_readsystem call. Together the calls avoid blocking on disk I/O, leading to outstanding performance.
NGINX Plus and NGINX Can Optimize Your Application Delivery, Too
From its inception, NGINX was designed to be adaptable and support every aspect of application delivery. To make applications similar to Netflix easier for our commercial customers to deploy, NGINX Plus combines web serving, load balancing, content caching, and media streaming in one easy to use package. Check out how our case studies to learn how other leading companies use NGINX Plus to deliver their applications with performance, security, and scale.
We enjoy working closely with customers, providing guidance on how to get the most out of NGINX Plus in their specific application delivery architectures. Our Support and Professional Services teams can help you with architectural guidance, installation, configuration, updates, and more. Contact us to learn more.
