Bloomberg Eschews Vendors For Direct Kubernetes Involvement

Financial information behemoth Bloomberg is a big fan of Kubernetes, and is using it for everything from serving up Bloomberg.com to complex data processing pipelines.

Rather than use a managed Kubernetes service or employ an outsourced provider, Bloomberg has chosen to invest in deep Kubernetes expertise and keep the skills in-house. Like many enterprise organizations, Bloomberg originally went looking for an off-the-shelf approach before settling on the decision to get involved more deeply with the open source project directly.

Lori Hoffman/Bloomberg

Steven Bower, Data and Infrastructure Lead, Bloomberg photographed at Bloomberg World Headquarters in New York on June 15, 2018.

“We started looking at Kubernetes a little over two years ago,” said Steven Bower, Data and Infrastructure Lead at Bloomberg. “We had built our own orchestration system that pre-dated Docker and Kubernetes, and we were using that to manage our search infrastructure in the team I originally lead.”

“We were looking at the future of our orchestration framework and trying to figure out if there was a way for us not to have all this custom code,” he said. Bower’s team looked at Mesos, Yarn, Rancher, and similar offerings but nothing seemed to fit.

Around the same time, other parts of Bloomberg started to adopt Kubernetes. “Things like Bloomberg.com are running it,” he said, “and some other non-infrastructure engineering teams were using it for things like stateless content processing pipelines.” Eighteen months ago Bower’s team started using Kubernetes, driven largely by data science workloads.

“It’s a great execution environment for data science,” says Bower. “The real Aha! moment for us was when we realized that not only does it have all these great base primitives like pods and replica sets, but you can also define your own primitives and custom controllers that use them.”

“I have always said that people contribute to open source in a selfish way,” says Dan Kohn, Executive Director of the CNCF, “but everyone in the ecosystem benefits from the collective selfishness.” This doesn’t happen automatically, and the role of the CNCF in shepherding the development of Kubernetes, and ensuring that individual selfishness doesn’t overwhelm the collective good, shouldn’t be overlooked.

“It’s been a complicated growing process,” says Kohn. “Nobody is born knowing how to run a TOC [Technical Oversight Committee] meeting.” The members of the Foundation and its ‘three ring circus’ governance structure have tried to learn from history and their personal experience with other governance efforts. “We’ve gotten clearer about a set of principles,” says Kohn. These principles are captured, in typical developer fashion, in a GitHub repo.

“The Cloud Native Computing Foundation (CNCF) has done a fantastic job with the Kubernetes ecosystem,” Bower said. “It has a well managed roadmap, so we can plan ahead.”

Bloomberg has moved away from relying on vendors to drive innovation and has become heavily involved in open-source software communities instead. “Culturally, Bloomberg is like lots of small startups attached to a central spine,” Bower says. “A core value is delivering capability quickly.”

Bower has advice for other organizations considering a similar approach. “You need to have your organization in a good place for managing software in this way already,” he says. “If people are just randomly building stuff in a disorganized way, you’re going to have the same thing with open source software.”

Organizations also need to carefully consider how involved they want to be with Kubernetes. “With a cloud-hosted service, a lot of the complexity of Kubernetes is hidden from you. On-site you have to have people who understand how it works at the core, not just interfacing with it.” This isn’t a decision to be taken lightly, because it becomes a critical operational constraint.

Leaving support up to an external vendor makes you beholden to the vendor’s responsiveness if there are issues. Bloomberg decided that a typical vendor response time wasn’t going to be enough. “No one ever talks about nines of uptime here,” Bower says. “This thing cannot go down.”

The traders relying on Bloomberg’s data to inform their trades stand to lose big money if they can’t access the information they need due to an outage, and Bloomberg is often on the hook to make them whole if they miss out on an opportunity. These financial incentives have reinforced Bloomberg’s decision to be in tight control of its data supply-chain.

Other organizations should look carefully at their own circumstances before following Bloomberg’s lead.

Facebook is Testing Subscriptions, but Only for Private Groups

Facebook CEO Mark Zuckerberg has shied away from offering a paid version of his social network that is ad-free, for users who worry about their online activities being tracked. But there’s at least one area where the company is willing to experiment with subscriptions: private Facebook Groups.

Facebook announced the subscriptions a blog post that said administrators of certain Facebook Groups could begin charging $4.99 to $29.99 a month for memberships. Administrators of groups that are currently free won’t be affected, although some will have the option of creating subscription-supported groups in the future.

“Group admins build safe and supportive communities that people come back to every day,” the blog post announcing the change said. “We know that admins invest their time and energy to maintain their groups, and some have told us that they would like tools to help them continue to invest in their community and offer more to members.”

Facebook doesn’t allow ads in its groups, and, for now, the company won’t take a cut of the subscription revenue that administrators choose to introduce. The social network’s sales depend almost entirely on ads, a model that has landed the company in a number of controversies, such as the Cambridge Analytica scandal that prompted Congress to summon Zuckerberg to give testimony in April.

When asked during that testimony whether Facebook would transition from ad revenues toward a subscription-based model, Zuckerberg replied,

A number of people suggest that we should offer a version where people cannot have ads if they pay a monthly subscription, and certainly we consider ideas like that. I think that they’re reasonable ideas to think through. But overall, I think that the ads experience is going to be the best one. I think in general, people like not having to pay for a service. A lot of people can’t afford to pay for a service around the world, and this aligns with our mission the best.

Facebook’s announcement today doesn’t mark an abrupt shift from that thinking, but it does suggest that the company is willing to experiment with subscriptions in places where it believes it makes sense.