These instructions are a modified version of this guide, and the code is a modified version of the pytorch example in this repo.
- Start serving session, grab outputted service ID:
clearml-serving create --name "clearml triton serving"
- Make env file (e.g. example.env):
CLEARML_WEB_HOST="https://app.clear.ml"
CLEARML_API_HOST="https://api.clear.ml"
CLEARML_FILES_HOST="https://files.clear.ml"
CLEARML_API_ACCESS_KEY="<ACCESS_KEY>"
CLEARML_API_SECRET_KEY="<SECRET_KEY>"
CLEARML_SERVING_TASK_ID="<SERVICE_ID_FROM_ABOVE>"
- Create docker container (example docker yml file is in this repo, but can also get it from original clearml-serving repo):
sudo docker compose --env-file example.env -f docker-compose-triton.yml up
- Will auto-get logs in terminal, but can also get them back via:
sudo docker compose -f docker-compose-triton.yml logs -f --tail=200
- Run model training and saving code, copy hash from ClearML
- Add model to serving endpoint:
clearml-serving --id <SERVICE_ID> model add \
--engine triton \
--endpoint "<ENDPOINT_NAME>" \
--model-id <MODEL_ID_FROM_CLEARML> \
--input-size "[-1,5,384]" "[-1,384]" \
--input-name "INPUT__0" “INPUT__1” \
--input-type float32 float32 \
--output-size "[-1]" \
--output-type float32 \
--output-name "OUTPUT__0" \
--preprocess preprocess.py
- Check models directory (should see a folder for the new endpoint):
sudo docker exec -it clearml-serving-triton ls -la /models/
- Make requests to the endpoint at:
http://127.0.0.1:8080/serve/<ENDPOINT_NAME>
- View metrics: First create an ssh tunnel into the machine running the container:
ssh -L 3000:localhost:3000 user@HOST
- Then navigate to http://localhost:3000 and log in with admin/admin.
-
Triton serving expects list of numpy arrays (plain lists might also work?). Automatically converts them to tensor before passing to model’s forward() call. (I think)
-
(Shut down docker container):
sudo docker compose -f docker-compose-triton.yml down