What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build, ...
My guess is the Data Science ones give more structure around the common ML operations and are better optimized for compute and memory needed to train, deploy, doing things in parallel and run inference on models?
TData science pipelines are designed to manage the end-to-end data lifecycle (e.g., clean data, fit model, and serve model).
CI/CD pipelines are more general purpose software engineering tools around the automation of common tasks (e.g., running a testing suite).
The advantage of using data science pipelines is that the pipelines have already primitives for common data science tasks. The disadvantages are that the tools are often immature and require a user to adhere to the tools established workflow.
In general, both systems do not optimize for compute or memory. They are just code runners. The code or platform would have to optimize for compute or memory.
These system can sometimes parallelize the parts of the code that is easy to parallelize. For example, serving models is often independent for different users.
