Engineering Labs

Premises Since I joined the MLOps Community , many things have changed

March 1, 2021

Ivan Nardini

Premises

Since I joined the MLOps Community, many things have changed.

People like Demetrios Brinkmann, David Aponte and many others work hard together spreading knowledge about MLOps and ModelOps disciplines. They promote several initiatives like Slack channel, Coffee Sessions, Meetups, and Talks. With experts and practitioners who join from 32 different countries. All of that with the aim of creating a place where you can learn about experiences and best practices around ML and Model operations.

So when I start thinking about how I can contribute, I ask myself:

…What another way to support people…?
…For a person who joins the community, how can I turn this knowledge in something can help him/her solving most common ModelOps challenges in a concrete way?

These were the questions I asked myself at the very beginning.

And the unique answer I find to all them is…

Then, here I am so eager to talk about that. Indeed, this is the opening article of a Medium’s series the MLOps community is going to have about the initiative.

With it, I will introduce to the Engineering Labs and its guiding principles.

In particular, I will give you an overall picture about the typical lab journey and I will describe each phases in details. To make the telling point, I will use the 1st Lab as an example of what I’m referring at. Finally I provide contact information in case I success to convince you join us =)

Below you can see the Table of Content of the rest of the article:

The Initiative and its guiding principles

Originally we announce the initiative in August. And now, as then, Engineering Labs is a place where you as a person has the opportunity to join a team and work together on solving a particular Machine Learning Engineering challenge.

No matter how tough challenge will be, we firmly believe that collaboration is the key factor to take down it. Why? Because it is in the DNA of the MLOps. If you want to put machine learning systems in production you need to develop solutions through the collaborative effort of self-organising and cross-functional team.

In a team, each person has the chance to learn from other members. Indeed, it is one of our main concern combining heterogeneous skills. Both technical and soft ones. In this way, you improve at the same time you have fun.

Of course, each challenge has many possible ways to be solved.So we decidenot to impose strict constraints. Once we clearly define the content of the lab, you are free to experiment and test each combination of solution and technology. But because we want to create value for our end users, your solution needs to be as much replicable as possible. That’s why we create a GitHub repository as source of truth.

About the content of labs, depending on the participation and how the initiative will evolve as the community changes, we will ask you and anyone who wants to propose ML challenge to solve. We will collect them and we will let the community cast its vote on it.

And, in the end, once you and your team solved the challenge, we will ask you to share your experience and what you learn. Because we commonly agree that the true knowledge is the one people share with feelings and not only with words.

Now that you have an idea about the initiative and its guiding principles, let provide you an overall picture about the Engineering Lab’s journey.

The Engineering Lab’s journey

I always thought that a picture is worth a thousand words. So there you go.

As you can see, the Engineering Lab’s journey consists in 6 steps:

Step 1: Fill the form

We provide a subscription form to collect contact info, background and everything is useful to assign you to the right team. Also you can propose a challenge you would like to solve.

Then, we review the info and we announce the lab. In particular, we let you know the lab challenge (with a community vote as optional step), which is your team and all the information you need to success.

Step 2: Join the team

Once you know your team, we have an onboarding call where you present yourself to the entire team. And remember that it is the right time to ask all questions you have and solve all doubts about what’s next.

Step 3: Solve the challenge in 6 weeks

From now on, you have all you need to start solving the challenge. You and your team are free to plan your 6 weeks as you prefer. Just to be sure everything is going well, the entire team will have 30 min stand up meeting to check how lab is going on.

Step 4: Merge your project and provide feedback

At the end of the 6 weeks, you should be ready to merge your projectto the official Engineering Labs’ GitHub repounder the MLOps.community organisation. Then, we will ask you to fill a feedback form in order to tell us what you think about the experience.

For the rest of the journey, we you will present what you and your team have done.

Step 5: Coffee session

In the Coffee session, you attend a general discussion about your lab experience as a team. You talk about aspects like project management, the solution you propose, all technical issues with infrastructure and tools you use to solve the challenge. And because it is an informal session, whatever you like to share.

Step 6: Medium Article and the Final Technical Discussion

In this last step, you have the chance to go in depth of your solution describing it in an article series we will publish on the official MLOps Community Medium page. Then, because each team has its own experience, one representative for each team prepare a number of questions based on the article and we set up a final talk discussing about consensus.

Since this moment, the lab is considered as finished. All teams learned from each other, the challenge is fixed with a proper solution (given time limits), and we can move on to new lab.

That’s it. Probably, at that point, you’re wondering questions like

Is it too much? Will I able to go thought it?

Honestly, we were not sure about that as well at the beginning.

But you know what?

Two out of four teams who joined the 1st lab did it.

And actually they did it really well.

To give you an idea, I will use the 1st Engineering Lab as an example of what I’m referring at. But I let each teams present to you their solutions with articles are coming in the series.

The 1st Engineering Lab: PyTorch meets MLflow

This summer, PyTorch community announced a number of technical contributions to enable end-to-end support for MLflow usage with PyTorch promising to accelerate the workflow from research to production.

As MLOps Community, we know from discussions we had that operationalizing Pytorch models could be tough. That’s why we decide to test these new capabilities and provide a possible end-to-end model lifecycle of Pytorch model using Mlflow in the 1st official engineering labs.

Content

As I mentioned before, we decidenot to impose constraints. We just asked to use PyTorch and MLflow. We suggest to have a look at the full MLOps workflow for PyTorch models building and serving using MLflow in the original article. But then we let teams to figure it out.

Teams

After collecting the answer to the form, we decide to have 4×4 teams. In particular, community members involved in this scenario were:

Team 1 : Michel Vasconcelos, Alexey Naiden, Varuna Jayasiri, John Savage
Team 2 : Itirell, Mark Peters, Carlos Leyson, Semendiak
Team 3 : Laszlo Sragner, Artem Yushkovsky, Dimi, Paulo Maia
Team 4 : Shadab Hussain, Susana Brandao, Godwin Ekainu, Josh

The 1st Journey and its Results

We officially announce the lab in December. And teams start solving the challenge during the month of January.

It was not easy. In the first week, two teams gave up. Indeed, we realised later that set only one constraint (MLflow and Pytorch technologies) was not the best way of doing it because we didn’t ask them to solve a specific problem. That’s why 2 groups had a hard time deciding on a problem. And they folded.

But who remains collaborate a lot and worked hard on it. And at the end, they deliver two final projects:

Team 1, the News Classification project
Team 3, the Yelp Review project

In the meanwhile, I’d got contact with the AI&PIE community in Milan and together we agreed to have a meetup where teams were presented what they achieved. If you are interesting in how the meetup went, you can find the recording here. Also both teams shared their experience in dedicated Coffee sessions. Feel free to watch them. Both teams had frank talks about their overall experience. Finally, at the time I’m writing they are working on Medium articles and we are going to have the Final Technical Discussion soon.

By the way, doing the math I can count

1 lab about how put in production Pytorch model with MLflow
4×4 teams
2 final projects
1 meetup in the AI&PIE community
2 insightful coffee sessions
(coming) 2 medium articles and 1 final tech talk

Not so bad to be your 1st experience, isn’t it?

Of course, this would like to be a quick overview about what awaits you in case you decide to join the initiative. And with that in mind let me close this article telling you how you can reach us.

How you can join us

As lead of the initiative, I would recommend to

Join the slack channel: https://go.mlops.community/slack
Join the #engineeringlabs channel
Candidate yourself for the next lab with this form
Feel free to reach me out both in the community (@ilnardo92) and on Linkedin for clarifications

Then, STAY TUNE!

Summary

To sum up, this article is the official reference about everything you need to know about the MLOps Community’s Engineering Labs.

I push myself to provide a concise and clear picture about the journey describing each phases in details.

To go straight to the point, I used the 1st Engineering Lab as an example. And I did some math to show you how valuable is this initiative.

Finally, I provided contact information in case I convinced you to join us. Indeed, as Uncle Sam would say