Added anomaly detection script and requirements file - #140
Added anomaly detection script and requirements file#140Swastik (swastik-21) wants to merge 1 commit into
Conversation
… requirements.txt
|
@microsoft-github-policy-service agree |
Alif Sathar (QuantumAlchemist03)
left a comment
There was a problem hiding this comment.
Added a few suggestions for improvements. Overall looks good!
There was a problem hiding this comment.
The anomaly detection function is clear and well-documented. A few suggestions:
- Consider adding a check for std == 0 to avoid division by zero errors.
- In the example usage, it might help to explain why z_thresh=2.0 is chosen (e.g., more sensitive to outliers).
- Optionally, you could mention that this method assumes a roughly normal distribution for data.
There was a problem hiding this comment.
This requirements.txt is clear and version-pinned, which is great for reproducibility. A few suggestions:
- Double-check if numpy==2.3.2 and pandas==2.3.2 are compatible; sometimes older versions of one library may conflict with the other.
- Consider whether all dependencies (like
sixortzdata) are actually used in the project — removing unused packages can simplify the environment. - Optionally, you could mention Python version compatibility, e.g., if this works for Python 3.11 or higher.
Manya Sharma (ManyaS-Git)
left a comment
There was a problem hiding this comment.
Nice, small and readable sample — a good first contribution. The Z-score approach is a clean choice for univariate water-usage data. A few issues to consider before this is merge-ready:
1. Division by zero on constant input (detect_anomalies).
std = data.std() is 0.0 whenever all values are identical, and (data - mean) / std then produces NaN (plus a NumPy RuntimeWarning). Since abs(NaN) > z_thresh is False, the function silently returns an empty result for a legitimately flat series. The same happens for single-element input (std is NaN with the default ddof=1). Recommend guarding, e.g.:
if std == 0:
return data[data != mean] # or pd.Series([], dtype=data.dtype)or delegating to scipy.stats.zscore, which returns 0.0 for constant data.
2. ddof=1 vs population Z-score.
Series.std() defaults to sample standard deviation (ddof=1), while the classic Z-score for a full dataset uses population std (ddof=0). With small samples this inflates the Z-scores and can push legitimately normal points past the threshold. Either is defensible, but it should be a deliberate choice — consider passing ddof=0 or noting the convention in the docstring.
3. NaN in the input silently disables detection.
If the series contains any NaN, mean/std become NaN and all comparisons evaluate False, so the function reports no anomalies without warning. Suggest data = data.dropna() at the top (documented) so missing readings don't silently swallow the check.
4. Root-level requirements.txt with exact pins of transitive deps.
The repo root currently has no requirements.txt; this adds one that pins numpy, python-dateutil, pytz, six, and tzdata to exact versions — all transitive dependencies of pandas that the sample never imports directly. Exact == pins of transitive packages are fragile and could conflict with other samples/modules later added to the repo. Prefer pinning only what's actually used with a floor bound, e.g. pandas>=2.0, and drop the transitive entries (or generate a dedicated requirements.txt inside the sample folder instead of at the repo root).
5. New top-level ai_modules/ folder.
The repo doesn't have an ai_modules/ directory today; introducing a generic top-level folder with a broad name doesn't match the existing layout (ai100-samples, ai200-architectures, ai300-practices, utilities, ...). Consider placing this under an existing area or a self-describing directory (e.g. ai300-practices/anomaly-detection-zscore/) so the sample is discoverable alongside the other examples.
6. Minor: the function returns the anomalous values rather than their indices/positions; the docstring says "points", which is ambiguous. Returning the boolean mask or the indices is often more useful for plotting/flagging, so clarify the contract in the docstring. The __main__ demo works correctly with z_thresh=2.0 (the 500 spike is flagged).
Overall a solid, focused PR — addressing items 1, 3, and 4 would be the most impactful.
Hi,
I’ve added a simple anomaly detection script under ai_modules/anomaly_detection.py. Right now it uses a z-score approach to flag unusual values in water usage data. The threshold can be adjusted so it works with different datasets.
I also created a requirements.txt so dependencies can be installed more easily.
Tested locally with sample data, and it correctly flagged extreme outliers. This is an initial version, so open to feedback or suggestions on improvements.
Thanks