Deployment Platform Updates

The Truefoundry team has been working really hard the last month adding features to our ML Deployment platform. Our goal here is to build a deployment platform that makes is absolutely easy to deploy ML models and services while enforcing the best engineering and security principles. To build a great ML platform, we need to have a solid engineering platform and that's why a lot of the initial focus has been on delivering a solid platform to deploy code.

Out of all the pieces of the Ml platform described above, we focus on the serving infrastructure, monitoring and all the automation around that.

A lot of work went in building our deployment platform on top of Kubernetes. The goal here has been to make it absolutely easy to deploy in under 5 minutes wherein the platform takes care of building the image from the source code, storing it in a docker registry and then finally deploying the application on Kubernetes. A few of the updates from our last month include the following:

Ability to choose instance family while deploying

Machine learning models can have very different inference latency or performance based on the instance type. For e.g, when testing the inference latency of a hugging face model on Intel vs AMD processors, we found Intel processors to be around 30% faster. That’s why we now have an option to allow users to choose the instance type while deploying their workloads. If the instance type is not selected, the workload can be deployed on any available instance type.

Logs and Metrics for Deployments

We earlier had a Grafana link for showing logs and metrics. While Grafana was highly customizable, permission and access control wasn’t really possible on Grafana. Also, it turned out to be a bit slow and difficult to understand for users who weren’t used to Grafana. That’s why we implemented our own UI for showing logs and metrics which should suffice in most cases. We still offer the Grafana integration in public cloud for more advanced users.

Permission Control On Secret Groups

We can now add users as editor, viewer or admin on secret groups.

Github and Bitbucket integration

We can now deploy directly to Truefoundry from any Github or bitbucket repositories. Users can integrate with their own private repositories using the Oauth Flow and select the appropriate parameters to deploy the application.

In the next month, we are working on a few exciting features like:

1. Making the platform more intuitive and easy to use.
2. Automated deployment of truefoundry stack on any Kubernetes cluster
3. Support for teams
4. Deployment rollback functionality

Stay tuned and let us know your feedback!