Python Cloud Inventory

A small Python command-line application that creates sanitized S3 and EC2 inventory reports from deterministic sample data or explicit read-only AWS API calls.

Purpose

This foundation project demonstrates practical Python skills used in cloud engineering:

• variables, conditions, loops, functions, lists, and dictionaries
• modules, imports, type hints, and data classes
• JSON configuration and environment variables
• reading and writing files safely
• exception handling and documented exit codes
• command-line arguments with argparse
• dependency isolation with a virtual environment
• read-only AWS calls with Boto3
• automated tests with Python's standard unittest module
• secure Git preparation and generated-report handling

The default sample mode requires no AWS credentials and makes no network calls. Live AWS collection happens only when --source aws is provided.

Project Structure

python-cloud-inventory/
+-- README.md
+-- .gitignore
+-- requirements.txt
+-- inventory.py
+-- cloud_inventory/
|   +-- __init__.py
|   +-- cli.py
|   +-- collectors.py
|   +-- config.py
|   +-- reporting.py
+-- config/
|   +-- inventory.example.json
+-- docs/
|   +-- aws-read-only-policy.md
+-- examples/
|   +-- sample-inventory.json
+-- reports/
|   +-- .gitkeep
+-- tests/
|   +-- __init__.py
|   +-- test_cli.py
|   +-- test_collectors.py
|   +-- test_config.py
|   +-- test_reporting.py
+-- steps-by-steps-process.md  # local guide, intentionally ignored

Requirements

• Python 3.10 or later
• Boto3 only for live AWS mode
• AWS credentials with read-only permissions only for live AWS mode

The sample workflow and test suite use the Python standard library and can run before Boto3 is installed.

Quick Start

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install the live-mode dependency:

python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt

Run the safe offline sample:

python3 inventory.py --source sample

The command creates:

reports/inventory-report.json
reports/inventory-report.txt

Both files are ignored by Git.

Command Reference

Display all options:

python3 inventory.py --help

Print sample JSON without writing files:

python3 inventory.py --source sample --dry-run

Use the example configuration:

python3 inventory.py --config config/inventory.example.json

Override the Region and output directory:

python3 inventory.py --region eu-west-2 --output-dir reports

Configuration precedence, from highest to lowest, is:

1. Command-line option
2. Environment variable
3. JSON configuration
4. Built-in default

Supported environment variables:

Variable	Default	Purpose
AWS_REGION	us-east-1	Region used for EC2 and displayed in reports.
INVENTORY_OUTPUT_DIR	reports	Directory for generated reports.

Example command-scoped values:

AWS_REGION=eu-west-2 INVENTORY_OUTPUT_DIR=/tmp/python-inventory \
  python3 inventory.py --source sample

Live AWS Mode

Live mode is opt-in:

python3 inventory.py --source aws

It calls only:

• s3:ListAllMyBuckets
• ec2:DescribeInstances

Read docs/aws-read-only-policy.md before enabling live access. Use a restricted AWS profile, IAM Identity Center session, or workload role. Never place access keys in Python files, JSON configuration, shell history, or committed environment files.

The EC2 report intentionally excludes instance IDs, tags, IP addresses, account IDs, and user data. S3 bucket names are included because listing buckets is the project requirement; therefore, live reports can still be sensitive and must remain local.

The project does not create, update, or delete AWS resources.

Reports

Every run builds one in-memory dictionary and writes two representations:

• JSON for machines and later automation
• plain text for quick human review

Reports include:

• UTC generation timestamp
• selected data source
• configured Region
• resource counts
• S3 bucket names from the chosen source
• EC2 instance type, state, and Availability Zone

Writes use a temporary file in the destination directory followed by os.replace. This prevents a failed write from leaving a partially written final file.

Exit Codes

Code	Meaning
0	Inventory completed successfully.
2	argparse rejected command-line usage.
3	Configuration was missing or invalid.
4	Boto3 was unavailable for live AWS mode.
5	Sample or AWS inventory collection failed.
6	Report files could not be written.

Use the status in automation:

python3 inventory.py --source sample
printf 'exit status: %s\n' "$?"

Testing

Compile all Python files without running live AWS collection:

python3 -m compileall -q inventory.py cloud_inventory tests

Run the complete standard-library test suite:

python3 -m unittest discover -s tests -v

The verified suite contains 32 passing tests.

The tests use temporary directories, deterministic JSON, mock objects, and patched functions. They do not need AWS credentials and do not call AWS APIs.

The suite covers:

• defaults and configuration precedence
• invalid, missing, and malformed configuration
• service allow-list validation
• sample-data validation and sorting
• simulated S3 and paginated EC2 collection
• exclusion of EC2 instance IDs
• API-error wrapping
• report metadata, formatting, and replacement
• sample dry runs and report creation
• dependency, collection, configuration, and report failures

Security

• No AWS credentials are stored in the project.
• Sample data contains invented values only.
• Live AWS mode must be requested explicitly.
• Live APIs are read-only.
• Generated reports are ignored because resource names can be sensitive.
• EC2 IDs, IP addresses, tags, user data, and AWS account IDs are not collected.
• .venv/, .env*, .aws/, private keys, caches, and generated reports are ignored.
• JSON input is validated before use.
• Error messages do not print credentials or raw AWS responses.
• Temporary files are replaced atomically and cleaned after write failures.

.gitignore cannot remove a file that Git already tracks and cannot erase secrets from history. Rotate an exposed credential immediately and follow an approved history-remediation process.

Troubleshooting

ModuleNotFoundError: No module named 'boto3'

Activate the virtual environment and install the requirements:

source .venv/bin/activate
python3 -m pip install -r requirements.txt

Sample mode and tests do not require Boto3.

Unable to locate credentials

Live mode needs a configured AWS identity. Use an approved profile, IAM Identity Center session, or workload role. Do not hardcode credentials.

Access denied

Confirm the active identity has only the documented list and describe actions. Also confirm the intended Region and any organization policies or permission boundaries.

Configuration error

Validate the JSON syntax, supported service names, Region, and output directory. The only supported services are s3 and ec2.

Report error

Confirm the destination parent is writable and is not an existing regular file. Try an explicit temporary directory while diagnosing the problem.

Unexpected files in Git

git status --short
git check-ignore -v reports/inventory-report.json
git ls-files

Review every staged file before committing.

What I Learned

This project connects core Python syntax to a realistic cloud workflow. It shows how lists and dictionaries represent API data, how functions divide responsibilities, how modules organize code, how exceptions turn failures into clear command outcomes, and how JSON supports both configuration and reports.

It also reinforces that cloud automation should be safe by default. Offline sample mode is deterministic, live access is explicit and read-only, credentials stay outside the codebase, reports remain local, and tests exercise failure paths without contacting AWS.

Limitations and Future Improvements

This is a learning project, not a complete cloud asset-management system. Future improvements could include:

• CI testing on supported Python versions and operating systems
• Ruff, Black, and static type checking
• pagination and location enrichment for S3 at larger scale
• additional read-only AWS services
• CSV output
• structured logging
• packaging as an installable command

These are not part of the current implementation.

Official References

• Python tutorial
• Python virtual environments
• Python unittest framework
• Boto3 documentation
• Boto3 credentials guide
• Amazon S3 ListBuckets API
• Amazon EC2 DescribeInstances API