Skip to content

api_README.md

Latest commit

 

History

History
500 lines (446 loc) · 24.8 KB

api_README.md

File metadata and controls

500 lines (446 loc) · 24.8 KB
Raw

Augmentoolkit API Documentation

This document describes the REST API endpoints for interacting with the Augmentoolkit server, allowing you to manage and run data generation pipelines, monitor their status, and manage associated files (inputs, outputs, logs, configs).

The API is built using FastAPI and utilizes Huey for background task queueing and Redis for state management (task status, PID mapping, output directory mapping).

Base URL: (Assuming the server runs locally on port 8000) http://127.0.0.1:8000

Core Concepts

  • Pipelines: Defined sequences of data processing steps, typically corresponding to Python functions within the codebase (e.g., in a pipelines directory).
  • Tasks: An instance of a pipeline run, managed by the Huey background task queue. Each task has a unique task_id.
  • Nodes: Refers to the importable path of the Python function that executes a pipeline (e.g., pipelines.my_pipeline.run).
  • Configs: YAML configuration files (stored in ./external_configs/) that provide parameters for pipeline runs. Can be overridden by parameters in the API request.
  • Super Config (super_config.yaml): Defines path aliases used for resolving node and config paths.
  • Directories:
    • ./inputs/: Stores input data files used by pipelines.
    • ./outputs/: Stores output files generated by pipelines. Output directories are often named using the task_id.
    • ./external_configs/: Stores reusable pipeline configuration files.
    • ./logs/: Stores log files (<task_id>.log) for each pipeline task run.

Endpoints

Health Check

  • GET /
    • Summary: Health Check
    • Description: Basic health check endpoint. Returns a simple message indicating the API is running.
    • Response: {"message": "Augmentoolkit API is running."}

Pipeline Execution

  • POST /pipelines/run

    • Summary: Queue a dataset generation pipeline for execution.
    • Status Code: 202 Accepted
    • Description: Resolves node and config paths using aliases from super_config.yaml and enqueues a pipeline task using Huey (tasks.run_pipeline_task). The task executes the pipeline defined by node_path potentially using configuration from config_path and parameters via run_augmentoolkit.py.
    • Request Body: 
      {
  "node_path": "string (required, can use aliases)",
  "config_path": "string (optional, relative to ./external_configs/ or absolute, can use aliases)",
  "parameters": { /* JSON object of key-value pairs (optional) */ }
}
    • Response: 
      {
  "pipeline_id": "string (Huey Task ID)",
  "message": "Pipeline run queued successfully."
}

  • GET /pipelines/available

    • Summary: Get available pipeline aliases from super_config.yaml.
    • Description: Reads the super_config.yaml file and returns a sorted list of path aliases (keys) whose corresponding values do not end with .yaml. These aliases typically represent the runnable entry points for pipelines that can be passed to the node_path field in the /pipelines/run endpoint.
    • Path Parameters: None
    • Query Parameters: None
    • Response: A JSON list of strings. 
      [
  "alias1",
  "alias2",
  "another-pipeline-alias",
  ...
]
    • Error Responses:
      • 500 Internal Server Error: If super_config.yaml cannot be found or parsed, or an unexpected error occurs. (Note: If the file is found but empty or path_aliases is missing/invalid, it currently returns an empty list [] gracefully).

Task Status & Management

  • GET /tasks/{task_id}/status

    • Summary: Get the status of a pipeline run.
    • Description: Retrieves the current status (PENDING, RUNNING, COMPLETED, FAILED, REVOKED) and result (if finished) of a specific pipeline task using its task_id. Checks Huey's state and potentially Redis for progress information.
    • Path Parameters:
      • task_id: The Huey task ID obtained from /pipelines/run.
    • Response: 
      {
  "task_id": "string",
  "status": "string (PENDING | RUNNING | COMPLETED | FAILED | REVOKED)",
  "message": "string (optional)",
  "progress": "float (optional, 0.0-1.0)",
  "details": { /* Optional details, e.g., error info or result */ }
}
    • Error Responses:
      • 404 Not Found: If task_id does not exist.
      • 500 Internal Server Error: If status retrieval fails.

  • GET /tasks/{task_id}/parameters

    • Summary: Get the parameters a task was executed with.
    • Description: Retrieves the parameters (combined from config file and request overrides) that were used to start a specific task run. Parameters are stored in Redis when the task begins execution and persist based on a configured timeout (typically several days).
    • Path Parameters:
      • task_id: The Huey task ID obtained from /pipelines/run.
    • Response: 
      {
  "task_id": "string",
  "parameters": { /* JSON object of key-value pairs used for the task */ }
}
    • Error Responses:
      • 404 Not Found: If the task_id does not exist or if the parameters were not stored/have expired.
      • 500 Internal Server Error: If retrieving or parsing the parameters from Redis fails (e.g., data corruption).

  • POST /tasks/{task_id}/interrupt

    • Summary: Interrupt a running pipeline task subprocess or revoke a pending task.
    • Description: Attempts to interrupt a running task or revoke a pending one.
      • If the task is RUNNING: Looks up the subprocess PID associated with the task_id in Redis (set by tasks.py) and sends a SIGINT signal to it using os.kill.
      • If the task is PENDING: Revokes the task using huey.revoke.
      • Reports an error if the task is already finished or revoked.
    • Path Parameters:
      • task_id: The Huey task ID.
    • Response:
      • {"message": "Sent SIGINT signal..."} (for running tasks)
      • {"message": "Task ... was pending and has been successfully revoked."} (for pending tasks)
    • Error Responses:
      • 404 Not Found: If task_id does not exist or the subprocess PID is gone.
      • 409 Conflict: If the task is already finished/revoked or fails to revoke.
      • 500 Internal Server Error: If signaling fails or other errors occur.

  • GET /tasks/queue

    • Summary: Get lists of pending and scheduled tasks.
    • Description: Retrieves the IDs of tasks currently pending or scheduled in the Huey queue using huey.pending() and huey.scheduled(). Note: This does not reliably show tasks that are actively running.
    • Path Parameters: None
    • Query Parameters: None
    • Response: 
      {
  "pending_tasks": ["string (task_id)", ...],
  "scheduled_tasks": ["string (task_id)", ...],
  "message": "string (e.g., Found X pending and Y scheduled tasks.)"
}
    • Error Responses:
      • 500 Internal Server Error: If retrieving queue status fails.

Log Management

  • GET /tasks/{task_id}/logs

    • Summary: Get logs for a specific task.
    • Description: Retrieves the content of the log file (./logs/<task_id>.log) generated by a specific task run.
    • Path Parameters:
      • task_id: The Huey task ID.
    • Query Parameters:
      • tail (int, optional): Return only the last N lines of the log.
    • Response: 
      {
  "task_id": "string",
  "message": "string",
  "logs": "string (log content)"
}
    • Error Responses:
      • 404 Not Found: If the log file doesn't exist.
      • 500 Internal Server Error: If reading the file fails.

  • DELETE /tasks/{task_id}/logs

    • Summary: Delete log file for a specific task.
    • Description: Deletes the specific log file (./logs/<task_id>.log).
    • Path Parameters:
      • task_id: The Huey task ID.
    • Response: {"message": "Successfully deleted log file..."}
    • Error Responses:
      • 404 Not Found: If the log file doesn't exist.
      • 500 Internal Server Error: If deletion fails.

  • GET /logs

    • Summary: List all available log files.
    • Description: Returns a list of all .log filenames currently in the ./logs/ directory.
    • Response: {"log_files": ["task_id1.log", "task_id2.log", ...]}
    • Error Responses:
      • 500 Internal Server Error: If listing the directory fails.

  • DELETE /logs

    • Summary: Clear all task log files.
    • Description: Deletes all files ending with .log within the ./logs/ directory.
    • Response: Reports the number of deleted files and any errors encountered. 
      {
    "message": "Successfully deleted X log file(s)...",
    "errors": [ /* list of error strings, if any */ ]
}
    • Error Responses:
      • 500 Internal Server Error: If accessing or clearing the directory fails.

Output File Management

  • GET /tasks/{task_id}/outputs/download

    • Summary: Download output directory for a task.
    • Description: Retrieves the path of the output directory associated with the task_id from Redis (set by tasks.py), zips the entire directory content, and returns it as a downloadable file (task_<task_id>_output.zip).
    • Path Parameters:
      • task_id: The Huey task ID.
    • Response: A zip file (application/zip).
    • Error Responses:
      • 404 Not Found: If the task ID mapping or the directory itself doesn't exist.
      • 500 Internal Server Error: If zipping or file access fails.

  • GET /tasks/{task_id}/outputs/structure

    • Summary: Get structure of a task's output directory.
    • Description: Retrieves the path of the output directory associated with the task_id from Redis and returns its file/folder structure as a JSON list. Paths are relative to the specific task's output directory root.
    • Path Parameters:
      • task_id: The Huey task ID.
    • Response: List of FileStructure objects (see below).
    • Error Responses:
      • 404 Not Found: If the task ID mapping or the directory itself doesn't exist.
      • 500 Internal Server Error: If reading the directory fails.

  • GET /outputs/structure/{relative_path:path}

    • Summary: Get structure of a path within the main outputs directory.
    • Description: Retrieves the file/folder structure for a given path relative to the root ./outputs/ directory. Uses the shared handle_get_structure helper.
    • Path Parameters:
      • relative_path: The path relative to ./outputs/ (e.g., my_folder/sub_folder or leave empty/. for the root).
    • Response: List of FileStructure objects: 
      [
  {
    "path": "string (relative path)",
    "is_dir": true,
    "children": [ /* nested FileStructure objects */ ]
  },
  {
    "path": "string (relative path)",
    "is_dir": false,
    "children": null
  }
]
    • Error Responses:
      • 404 Not Found: If the path doesn't exist.
      • 400 Bad Request: If the path is not a directory.
      • 500 Internal Server Error: If listing fails.

  • GET /outputs/download/{relative_path:path}

    • Summary: Download a specific file or folder from outputs.
    • Description: Downloads a specific file or zips and downloads a specific folder relative to the root ./outputs/ directory. Uses the shared handle_download_item helper.
    • Path Parameters:
      • relative_path: The path relative to ./outputs/.
    • Response: A file (application/octet-stream or detected type) or a zip archive (application/zip).
    • Error Responses:
      • 404 Not Found: If the path doesn't exist.
      • 500 Internal Server Error: If reading/zipping fails.

  • DELETE /outputs/{relative_path:path}

    • Summary: Delete a file or folder from outputs.
    • Description: Deletes a specific file or folder relative to the root ./outputs/ directory. Uses the shared handle_delete_item helper. Warning: This is permanent.
    • Path Parameters:
      • relative_path: The path relative to ./outputs/.
    • Response: {"message": "Successfully deleted..."}
    • Error Responses:
      • 404 Not Found: If the path doesn't exist.
      • 400 Bad Request: If attempting to delete the root ./outputs directory.
      • 500 Internal Server Error: If deletion fails.

  • POST /outputs/move

    • Summary: Move/Rename a file or folder within outputs.
    • Description: Moves or renames a file or folder within the ./outputs/ directory using shutil.move.
      • The source path must exist.
      • The destination path must not already exist (to prevent overwrites).
      • The parent directory of the destination path must exist.
      • Cannot be used to move the root ./outputs directory itself.
    • Request Body: 
      {
  "source_relative_path": "string (required, current relative path)",
  "destination_relative_path": "string (required, new relative path)"
}
    • Response (Status 200): 
      {
  "message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
    • Error Responses:
      • 400 Bad Request: If source/destination paths are invalid, if the destination parent doesn't exist, or if attempting to move the root directory.
      • 404 Not Found: If the source path does not exist.
      • 409 Conflict: If the destination path already exists.
      • 500 Internal Server Error: If the move operation fails for other reasons (e.g., permissions).

Input File Management

Input file management endpoints mirror the Output File Management endpoints, but operate on the ./inputs/ directory. Also, it has a route to create a directory, too.

  • GET /inputs/structure/{relative_path:path}

    • Summary: Get structure of a path within the inputs directory.
    • Description: Retrieves structure relative to ./inputs/.
    • Response: List of FileStructure objects.

  • GET /inputs/download/{relative_path:path}

    • Summary: Download a specific file or folder from inputs.
    • Description: Downloads file/folder relative to ./inputs/.
    • Response: File or Zip archive.

  • DELETE /inputs/{relative_path:path}

    • Summary: Delete a file or folder from inputs.
    • Description: Deletes file/folder relative to ./inputs/. Warning: Permanent.
    • Response: {"message": "Successfully deleted..."}

  • POST /inputs/move

    • Summary: Move/Rename a file or folder within inputs.
    • Description: Moves or renames a file or folder within the ./inputs/ directory using shutil.move. See /outputs/move for detailed behavior and error conditions.
    • Request Body: 
      {
  "source_relative_path": "string (required, current relative path)",
  "destination_relative_path": "string (required, new relative path)"
}
    • Response (Status 200): 
      {
  "message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
    • Error Responses: 400, 404, 409, 500 (see /outputs/move)

  • POST /inputs/directory

    • Summary: Create a new directory within the inputs folder.
    • Description: Creates a new directory at the specified relative_path within the ./inputs/ directory. The path can include parent directories (e.g., existing_folder/new_subfolder).
    • Request Body: 
      {
  "relative_path": "string (required, relative path)"
}
    • Response (Status 201): 
      {
  "message": "Successfully created directory '{relative_path}' in inputs."
}
    • Error Responses:
      • 400 Bad Request: If relative_path is invalid (empty, '.', targets root) or if the parent directory specified in the path does not exist.
      • 409 Conflict: If a file or directory already exists at the target relative_path.
      • 500 Internal Server Error: If directory creation fails for other reasons (e.g., permissions).

  • POST /inputs/upload/{relative_path:path}

    • Summary: Upload files to a specific path within inputs, automatically extracting zip files.
    • Description: Uploads one or more files to a specified path within ./inputs/. The path is created if it doesn't exist.
      • If a file ends with .zip (case-insensitive), it is automatically extracted into the target relative_path.
      • Upon successful extraction, the original .zip file is deleted.
      • If extraction fails, the original .zip file is kept, and an error is reported.
      • Non-zip files are saved directly to the relative_path.
      • Allows uploading directly to the root ./inputs/ directory (e.g., by specifying . as the relative_path).
    • Path Parameters:
      • relative_path: Target path relative to ./inputs/. Files (including extracted zip contents) will be placed here.
    • Request Body: multipart/form-data containing one or more files.
    • Response (Status 201 or 207): 
      {
  "message": "Upload process completed...",
  "uploaded_files": [
    {"filename": "string", "size": int, "path": "string (relative path)", "error": "string (optional, only on failed extraction)"}
  ],
  "extracted_zips": [
    {"filename": "string (original zip)", "extracted_to": "string (relative path)"}
  ],
  "errors": [ /* list of general error strings, if any */ ]
}
    • Error Responses:
      • 500 Internal Server Error: If directory creation, file writing, or extraction fails unexpectedly.

Config File Management

Configuration file management endpoints operate on the ./external_configs/ directory.

  • GET /configs/aliases

    • Summary: Get available config file aliases from super_config.yaml.
    • Description: Reads the super_config.yaml file and returns a sorted list of path aliases (keys) whose corresponding values do end with .yaml (case-insensitive). These aliases can be used to identify existing configuration files for duplication or other purposes.
    • Path Parameters: None
    • Query Parameters: None
    • Response: A JSON list of strings. 
      [
  "config-alias1",
  "another_config.yaml",
  "path/to/config_alias_3",
  ...
]
    • Error Responses:
      • 500 Internal Server Error: If super_config.yaml cannot be found or parsed, or an unexpected error occurs. (Note: If the file is found but empty or path_aliases is missing/invalid, it returns an empty list [] gracefully).

  • GET /configs/structure/{relative_path:path}

    • Summary: Get structure of a path within the configs directory.
    • Description: Retrieves structure relative to ./external_configs/.
    • Response: List of FileStructure objects.

  • GET /configs/content/{relative_path:path}

    • Summary: Get content of a specific config file.
    • Description: Retrieves the plain text content of a specific file relative to ./external_configs/.
    • Path Parameters:
      • relative_path: The path relative to ./external_configs/.
    • Response: Plain text file content (text/plain).
    • Error Responses:
      • 404 Not Found: If the file doesn't exist.
      • 400 Bad Request: If the path is not a file.

  • POST /configs/content/{relative_path:path}

    • Summary: Create or update a config file.
    • Description: Creates a new file or overwrites an existing file at the specified path relative to ./external_configs/. Parent directories are created if they don't exist.
    • Path Parameters:
      • relative_path: The path relative to ./external_configs/.
    • Request Body: Raw text content (text/plain).
    • Response: {"message": "Successfully saved config..."}
    • Error Responses:
      • 400 Bad Request: If attempting to write to the root.
      • 500 Internal Server Error: If writing fails.

  • DELETE /configs/{relative_path:path}

    • Summary: Delete a specific config file or directory.
    • Description: Deletes a specific file or directory (recursively) relative to ./external_configs/. Warning: Permanent.
    • Path Parameters:
      • relative_path: The path relative to ./external_configs/.
    • Response: {"message": "Successfully deleted config {file|directory}..."}
    • Error Responses:
      • 404 Not Found: If the file or directory doesn't exist.
      • 400 Bad Request: If attempting to delete the root ./external_configs directory.
      • 500 Internal Server Error: If deletion fails.

  • POST /configs/move

    • Summary: Move/Rename a file or folder within configs.
    • Description: Moves or renames a file or folder within the ./external_configs/ directory using shutil.move. See /outputs/move for detailed behavior and error conditions.
    • Request Body: 
      {
  "source_relative_path": "string (required, current relative path)",
  "destination_relative_path": "string (required, new relative path)"
}
    • Response (Status 200): 
      {
  "message": "Successfully moved {file/directory} from '{source}' to '{destination}'."
}
    • Error Responses: 400, 404, 409, 500 (see /outputs/move)

  • POST /configs/directory

    • Summary: Create a new directory within the configs folder.
    • Description: Creates a new directory at the specified relative_path within the ./external_configs/ directory. Uses the shared handle_create_directory helper.
    • Request Body: 
      {
  "relative_path": "string (required, relative path)"
}
    • Response (Status 201): 
      {
  "message": "Successfully created directory '{relative_path}' in external_configs."
}
    • Error Responses:
      • 400 Bad Request: If relative_path is invalid (empty, '.', targets root) or if the parent directory specified in the path does not exist.
      • 409 Conflict: If a file or directory already exists at the target relative_path.
      • 500 Internal Server Error: If directory creation fails for other reasons (e.g., permissions).

  • POST /configs/duplicate

    • Summary: Duplicate a pipeline config into external_configs.
    • Description: Duplicates a configuration file identified by a source_alias (from super_config.yaml) into the ./external_configs/ directory at the specified destination_relative_path. The source alias must resolve to an existing .yaml file. The destination path must not already exist, and its parent directory must exist.
    • Request Body: 
      {
  "source_alias": "string (required, alias from super_config.yaml)",
  "destination_relative_path": "string (required, new relative path in external_configs)"
}
    • Response (Status 201): 
      {
  "message": "Successfully duplicated config from alias '{source_alias}' to '{destination_relative_path}' in external_configs."
}
    • Error Responses:
      • 400 Bad Request: If the source alias doesn't point to a .yaml file, if destination path is invalid, or if destination parent directory doesn't exist.
      • 404 Not Found: If the source alias is not found in super_config.yaml or the resolved source file doesn't exist.
      • 409 Conflict: If the destination path already exists.
      • 500 Internal Server Error: If resolving paths, reading the source, or writing the destination file fails.