Data Analysis with NumPy and pandas

Task 1: NumPy Array Math and pandas Data Cleanup Task

Create a script named numpy_pandas_task.py inside your lesson_59 folder:

• Import numpy and pandas.
• Create a 2D NumPy array of random float values ($4 \times 4$) and compute row-wise sums and column-wise means.
• Construct a pandas DataFrame containing missing `NaN` values in a numerical column.
• Fill missing `NaN` values using the column median and drop duplicate rows.
• Print formatted summary outputs using f-strings.

Task 2: Pattern-Matched DataFrame Analytical Dispatcher

Create a script named df_analytics_task.py:

• Build function analyze_metric_row(row: dict) -> str parsing row metric dictionaries.
• Write a match-case dispatcher:
  • case {"cpu_usage": float(cpu)} if cpu >= 90.0 → Return "CRITICAL_CPU_SPIKE_ALERT".
  • case {"ram_usage": float(ram)} if ram >= 85.0 → Return "HIGH_RAM_WARNING".
  • case {"status": "OFFLINE"} → Return "SERVER_OFFLINE".
  • case _ → Return "METRIC_NOMINAL".
• Create a sample DataFrame with server metrics, iterate through row dicts, and print classification results.

Task 3: Master Review Capstone Project — Enterprise High-Performance Analytics & Data Science OS (`data_analytics_os.py`)

Create a script named data_analytics_os.py inside your lesson_59 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 59**.

Project Architectural Requirements Specification:

1. NumPy & pandas Analytics Architecture (Lesson 59):
   • Perform vectorized array operations via NumPy and construct analytical pandas DataFrames.
   • Execute data cleaning (`fillna`, `dropna`), advanced indexing (`loc`), and category aggregations (`groupby()`).
   • Export processed summary tables to CSV using df.to_csv() and read inputs via pd.read_csv().
2. FastAPI & Web Framework Integration (Lesson 58):
   • Expose analytical query endpoints using asynchronous FastAPI path operation functions (`async def`) and Pydantic validation models (`BaseModel`).
3. Flask & Web Development Integration (Lesson 57):
   • Incorporate route parameter patterns and view handlers.
4. RPA & Office Automation Integration (Lesson 56):
   • Generate Excel summary reports via openpyxl and export email alerts using email.message.EmailMessage when analytics thresholds breach limits.
5. Web Scraping Integration (Lesson 55):
   • Extract HTML elements using BeautifulSoup4 (`bs4`) to populate DataFrame source tables.
6. HTTP APIs & requests Integration (Lesson 54):
   • Fetch remote operational datasets via requests.Session().
7. SQLite Persistence Integration (Lesson 53):
   • Persist DataFrame analytical records into an SQLite database table using df.to_sql() or parameterized SQL statements.
8. Security, Performance & Memory Hardening (Lesson 52):
   • Use __slots__ across core domain models for memory optimization.
   • Read secrets and data paths from environment variables using os.environ.
9. Packaging & Distribution Integration (Lesson 51):
   • Include a programmatically generated pyproject.toml package specification string and an argparse CLI entry point structure.
10. Clean Code, Documentation & Refactoring (Lesson 50):
    • Apply SOLID principles, DRY, and Google Style Docstrings across all functions and classes.
11. Testing & Quality Assurance (Lessons 47-48):
    • Include unit tests verified via pytest fixtures testing DataFrame transformations.
12. Asyncio & Concurrency Integration (Lessons 45-46):
    • Incorporate thread pool workers or async event loops for parallel multi-dataset analysis.
13. Temporal & RegEx Text Processing (Lessons 43-44):
    • Parse UTC-aware timestamps using datetime and zoneinfo.ZoneInfo.
    • Pre-compile RegEx patterns with Named Groups for parsing dataset string column entries.
14. Type Hints, Generics & Static Safety (Lesson 42):
    • Apply explicit modern type annotations (X | Y, list[str], dict[str, Any]) throughout all functions and classes.
15. Metaprogramming, Closures & Decorators (Lessons 40-41):
    • Build function factories and 3-level configurable decorators with @functools.wraps.
16. Iterators & Generators (Lesson 39):
    • Implement generator functions streaming DataFrame row chunks lazily with $O(1)$ memory.
17. Full OOP Architecture (Lessons 32-38):
    • Define abstract base class BaseAnalyticsEngine(ABC) and concrete Data Classes DataMetricRecord and AnalyticsAudit.
    • Build composite manager DataAnalyticsOS composing database persistence drivers and analytics handles.
18. Observability & Fault Tolerance (Lessons 29-31):
    • Set up multi-handler logging to console and data_analytics.log file.
    • Incorporate developer assertions (assert) verifying DataFrame length bounds.
    • Define custom exception hierarchy (DataAnalyticsOSError, DataTransformationError).
19. Context Managers & Persistence Layer (Lessons 27-30):
    • Use custom class-based context managers to manage temp file cleanup.
    • Persist JSON payload backups to analytics_db.json and export CSV reports to analytics_audit.csv using pathlib.Path.
20. Nested Collections & Comprehensions (Lessons 21-26):
    • Maintain in-memory registry dictionaries mapping IDs to Data Class Instances.
    • Use List and Dictionary Comprehensions to clean and transform records.
21. Advanced Function Parameters & Scope (Lessons 15-17):
    • Structure logic into pure modular functions with type hints, docstrings, early return Guard Clauses, and *args / **kwargs logging.
22. Indefinite & Definite Loops (Lessons 13-14):
    • Run the interactive CLI interface inside a continuous while True menu loop.
    • Iterate through output reports using for loops with enumerate() and .items().
23. Pattern Matching CLI Dispatcher (Lesson 12):
    • Process user CLI command tokens inside a match-case block with data analytics match guards:
      • case ["LOAD", "CSV", filepath] → Read CSV into pandas DataFrame and display summary info.
      • case ["CLEAN", "MISSING"] → Execute fillna() data cleaning sequence.
      • case ["GROUPBY", col_name] → Run category aggregation using pandas groupby().
      • case ["AUDIT", "ROWS"] → Route DataFrame row dicts using match-case pattern dispatcher.
      • case ["EXPORT", "CSV"] → Export analytical DataFrame results to CSV.
      • case ["EXIT" | "QUIT"] → Terminate session safely using a sentinel flag.
      • case _ → Output command error message.
24. Conditionals, Formatting & Foundations (Lessons 1-11):
    • Sanitize all inputs using .strip() and .upper().
    • Format tabular reports using f-strings with field width alignment specifiers (:<15, :>10) and clear visual borders.