Allegro Trains is a full system open source ML / DL experiment manager and ML-Ops solution. It enables data scientists and data engineers to effortlessly track, manage, compare and collaborate on their experiments as well as easily manage their training workloads on remote machines.
Trains is a suite of open source Python packages and plugins, including:
- Trains Python Client package - Integrate Trains into your AutoKeras tasks with just two lines of code, and get all of Trains robust features.
- Trains Server - The Trains backend infrastructure and web UI. Use the public Trains demo server, or deploy your own.
- Trains Agent - The Trains DevOps component for experiment execution, resource control, and autoML..
- Additional integrations - Integrate Trains with PyCharm and Jupyter Notebook.
Setting up Trains
To integrate Trains into your AutoKeras project, do the following:
Install the Trains Python Client package.
pip install trains
Add the short Trains initialization code to your task.
from trains import Task task = Task.init(project_name="autokeras", task_name="autokeras imdb example with scalars")
Run your task. The console output will include the URL of the task's RESULTS page.
TRAINS Task: overwriting (reusing) task id=60763e04c0ba45ea9fe3cfe79f3f06a3 TRAINS results page: https://demoapp.trains.allegro.ai/projects/21643e0f1c4a4c99953302fc88a1a84c/experiments/60763e04c0ba45ea9fe3cfe79f3f06a3/output/log</code></pre>
See an example script here.
Tracking your AutoKeras tasks
Visualizing Task Results
Trains automatically logs comprehensive information about your AutoKeras task: code source control, execution environment, hyperparameters and more.
It also automatically records any scalars, histograms and images reported to Tensorboard/Matplotlib or Seaborn.
For example, making use of Tensorboard in your task will make all recorded information available in Trains as well:
from tensorflow import keras tensorboard_callback_train = keras.callbacks.TensorBoard(log_dir='log') tensorboard_callback_test = keras.callbacks.TensorBoard(log_dir='log') clf.fit(x_train, y_train, epochs=2, callbacks=[tensorboard_callback_train]) clf.fit(x_test, y_test, epochs=2, callbacks=[tensorboard_callback_test])
When your task runs, you can follow its results, including any collected metrics through the Trains web UI.
View your task results in the Trains web UI, by clicking on it in the EXPERIMENTS table.
Find the EXPERIMENT table under the specified project listed in the HOME or PROJECTS page:
Trains automatically tracks models produced by your AutoKeras tasks.
To upload models, specify the
output_uri parameter when calling
Task.init to provide the upload destination:
task = Task.init(project_name="autokeras", task_name="autokeras imdb example with scalars", output_uri="http://localhost:8081/")
View models information in the experiment details panel, ARTIFACTS tab:
Tracking Model Performance
Use the Trains web UI to easily create experiment leaderboards and quickly identify best performing models.
Customize your board adding any valuable metric or hyperparameter.
Additional information on customizing Trains experiment and model tables can be obtained in the relevant Trains Documentation
Model Development Insights
Use the Trains web UI to view side-by-side comparison of experiments: Easily locate the differences and impact of experiment configuration parameters, metrics, scalars etc.
Compare multiple experiments, by selecting two or more experiments in the EXPERIMENTS table, and clicking COMPARE.
The following image shows how two experiments compare in their epoch_accuracy and epoch_loss behaviour: