Machine learning platform MLflow joins the Linux Foundation


Handing the platform run by Databricks to the vendor-neutral foundation will speed growth, the organizations say.


Image: boygovideo, Getty Images/iStockphoto

Databricks, the company behind open source end-to-end machine learning (ML) platform MLflow, announced Thursday that it is handing control of MLflow to the Linux Foundation.

“Our experience in working with the largest open source projects in the world shows that an open governance model allows for faster innovation and adoption through broad industry contribution and consensus building,” said VP of strategic programs at the Linux Foundation Michael Dolan.

Under the control of the foundation, MLflow will be managed using Apache License v.2, which Databricks CEO Ali Ghodsi said will easily allow businesses to use it without worry. 

“Handing MLflow over to the Linux Foundation makes it more independent, and will drive even more businesses to contribute to the growth of the platform,” Ghodsi said. 

SEE: Hiring Kit: Computer Research Scientist (TechRepublic Premium)

Databricks, which was co-founded by Apache Spark creator Matei Zaharia, released the alpha build of MLflow in 2018, and said it has seen explosive growth in interest and use since then. To contrast, Ghodsi said, it took three years to get the same amount of participation in Spark that MLflow garnered in three months.

MLflow has been adopted for ML data projects by numerous large organizations, such as Microsoft, Accenture, Zillow, Virgin, and Starbucks

MLflow was built with an open interface “designed to work with any ML library, algorithm, deployment tool or language,” Databricks said in its 2018 MLflow introductory post. Because it’s designed to be end-to-end, MLflow also incorporates every step in the machine learning process from data preparation to presentation of results. 

SEE: Robotic process automation: A cheat sheet (free PDF) (TechRepublic)

In the same introductory post, Ghodsi explained that MLflow was designed to address several problems in the machine learning process that Databricks had repeatedly heard mentioned: 

  • Too many ML products meant wasting time searching for the right combination of tools,
  • There are too many variables in each ML experiment to keep track of,

  • Reproducibility is difficult because of the above two reasons, and the problem of passing projects between teams working from different perspectives, and

  • Deployment of ML models is difficult due to a lack of standardization between tools.

“MLflow keeps this process from becoming overwhelming by providing a platform to manage the end-to-end ML development lifecycle from data preparation to production deployment, including experiment tracking, packaging code into reproducible runs, and model sharing and collaboration,” Databricks said in a press release.

Developers interested in experimenting with MLflow, which is designed to scale from small projects to enterprise-level initiatives, can find out how to install it and learn to use it at MLflow’s GitHub page.

Data, Analytics and AI Newsletter

Learn the latest news and best practices about data science, big data analytics, and artificial intelligence.
Delivered Mondays

Sign up today

Also see

Make online learning a priority to increase your company’s competitive edge


A formal eLearning policy can encourage employees to develop new skills and serve as a recruiting tool to attract new team members.

Training is a good way to keep good employees and attract new ones. Encouraging employees to develop new skills with an official eLearning policy can give individuals and companies a competitive edge over organizations that lack financial incentives and formal guidelines for ongoing professional education. 

A look at the data shows that people are turning to online platforms to improve their soft skills and their technical expertise more now than ever before. Online education site Udemy compared activity 21 days before coronavirus lockdowns started and 21 days after in the report “Online Education Steps Up: What the World is Learning (from Home).”

SEE: eLearning and continuing education policy (TechRepublic Premium)

The report demonstrated an increased demand around the world for digital instruction:

  • 425% increase in enrollments for consumers
  • 80% increase in usage from businesses and governments
  • 55% increase in course creation by instructors

Enjoying this article?

Download this article and thousands of whitepapers and ebooks from our Premium library. Enjoy expert IT analyst briefings and access to the top IT professionals, all in an ad-free experience.

Join Premium Today

People turned to Udemy for help with in-demandskills most needed during remote work. Udemy reported increased interest in these skills:

  • Telecommuting:      21,598% increase   
  • Virtual teams:           1,523% increase

  • Decision making:         277% increase

  • Self discipline:             237% increase

  • Stress management:    235% increase

In addition to learning these soft skills online, professionals have turned to online classrooms to improve and validate their technical skills. Kelly Ricker, executive vice president for events and education at CompTIA, said that in March the company saw a spike of interest in Live Online Training (LOT), instructor-led courses that simulate a traditional classroom experience with an instructor and hands-on eLearning tools. 

SEE: eLearning and continuing education policy (TechRepublic Premium)

“Earlier this year when many stay-at-home orders took effect, we saw enrollment numbers skyrocket, so much so that we added classes to accommodate demand,” she said. “We’ve taken location out of the equation, allowing individuals to train at home and test at home.” 

Lily Mok, research vice president on the leadership, culture, and people dynamics team within the Gartner CIO research group, added that having a highly trained team can be a competitive advantage.

“It’s even more important [in this economy] that you grow and build talent to support your needs, because otherwise you won’t be able to recover and scale based on where the business trajectory might be,” Mok said. 

One of the more effective ways to invest in your workforce is with a comprehensive eLearning and continuing education policy. The eLearning and continuing education policy template from TechRepublic Premium provides a customizable framework you can use to create a development incentive package for your organization.

SEE: eLearning and continuing education policy (TechRepublic Premium)

In April, CompTIA launched any time and anywhere testing for certifications after more than 25 years of working in brick- and mortar test centers. Randy Gross, CompTIA’s CIO, said that online testing has had a slow uptake due to the perception of security issues and technology challenges. COVID-19 erased that barrier.

“The pandemic forced us into creativity while ensuring equivalent and fair exams, and they’ve taken off like a rocket,” he said. We had tens of thousands sign up to be alerted to the launch or remote testing, so we know that many CIOs had staff in this situation.”

SEE: eLearning and continuing education policy (TechRepublic Premium)