Task 1: Ethical HTML Table Extractor Task

Create a script named ethical_scraper_task.py inside your lesson_55 folder:

• Import requests, BeautifulSoup from bs4, and time.
• Define function scrape_html_table(html_content: str) -> list[dict] parsing table rows () and table data cells ().
• Enforce rate limiting using time.sleep(1.0) and attach a custom descriptive User-Agent header.
• Extract text from table rows into a list of dictionaries and print formatted outputs using f-strings.

Task 2: Pattern-Matched DOM Element Dispatcher

Create a script named dom_dispatcher_task.py:

• Build function classify_scraped_node(node_dict: dict) -> str parsing DOM nodes.
• Write a match-case dispatcher:
  • case {"tag": "a", "href": str(link)} if "download" in link → Return "ACTION_FILE_DOWNLOAD".
  • case {"tag": "img", "src": str(img_url)} → Return "ACTION_IMAGE_INGEST".
  • case {"tag": "h1" | "h2", "text": str(heading)} → Return "ACTION_INDEX_HEADING".
  • case _ → Return "ACTION_IGNORE".
• Execute test calls across 4 sample node dictionaries and print result strings.

Task 3: Master Review Capstone Project — Production Enterprise Web Ingestion & Data Collection OS (`web_ingestion_os.py`)

Create a script named web_ingestion_os.py inside your lesson_55 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 55**.

Project Architectural Requirements Specification:

1. Web Scraping & Responsible Data Collection Architecture (Lesson 55):
   • Incorporate programmatically validated robots.txt permission checks via urllib.robotparser.
   • Extract DOM nodes using BeautifulSoup4 (`bs4`) CSS selectors (`.select()`) with rate limiting (`time.sleep`) and descriptive User-Agent headers.
2. HTTP APIs & requests Integration (Lesson 54):
   • Manage web scraping requests via requests.Session() with connection pooling, network timeouts, and raise_for_status().
3. SQLite Persistence Integration (Lesson 53):
   • Persist scraped HTML market records into an SQLite database table using parameterized SQL queries and sqlite3.Row.
4. Security, Performance & Memory Hardening (Lesson 52):
   • Use __slots__ across internal scraped data model classes for memory optimization.
   • Read scraping system configuration keys dynamically from environment variables using os.environ.
5. Packaging & Distribution Integration (Lesson 51):
   • Include a programmatically generated pyproject.toml package specification string and an argparse CLI entry point structure.
6. Clean Code, Documentation & Refactoring (Lesson 50):
   • Apply SOLID principles, DRY, and Google Style Docstrings across all functions and classes.
7. Testing & Quality Assurance (Lessons 47-48):
   • Include unit tests verified via pytest fixtures and unittest.mock (mocking requests.get and HTML responses).
8. Asyncio & Concurrency Integration (Lessons 45-46):
   • Incorporate thread pool workers or async event loops for parallel multi-page crawling tasks.
9. Temporal & RegEx Text Processing (Lessons 43-44):
   • Parse UTC-aware ingestion timestamps using datetime and zoneinfo.ZoneInfo.
   • Pre-compile RegEx patterns with Named Groups for parsing scraped text elements.
10. Type Hints, Generics & Static Safety (Lesson 42):
    • Apply explicit modern type annotations (X | Y, list[str], dict[str, Any]) throughout all functions and classes.
11. Metaprogramming, Closures & Decorators (Lessons 40-41):
    • Build function factories and 3-level configurable decorators with @functools.wraps.
12. Iterators & Generators (Lesson 39):
    • Implement generator functions streaming scraped record batches lazily with $O(1)$ memory.
13. Full OOP Architecture (Lessons 32-38):
    • Define abstract base class BaseScraperEngine(ABC) and concrete Data Classes ScrapedArticleRecord and HarvestAudit.
    • Build composite manager WebIngestionOS composing database persistence drivers and web parsing handles.
14. Observability & Fault Tolerance (Lessons 29-31):
    • Set up multi-handler logging to console and web_ingestion.log file.
    • Incorporate developer assertions (assert) verifying DOM element count bounds.
    • Define custom exception hierarchy (WebIngestionOSError, EthicalScrapingViolationError).
15. Context Managers & Persistence Layer (Lessons 27-30):
    • Use custom class-based context managers to manage network session handles.
    • Persist JSON payload backups to scraped_db.json and export CSV reports to harvest_audit.csv using pathlib.Path.
16. Nested Collections & Comprehensions (Lessons 21-26):
    • Maintain in-memory registry dictionaries mapping IDs to Data Class Instances.
    • Use List and Dictionary Comprehensions to clean and transform records.
17. Advanced Function Parameters & Scope (Lessons 15-17):
    • Structure logic into pure modular functions with type hints, docstrings, early return Guard Clauses, and *args / **kwargs logging.
18. Indefinite & Definite Loops (Lessons 13-14):
    • Run the interactive CLI interface inside a continuous while True menu loop.
    • Iterate through output reports using for loops with enumerate() and .items().
19. Pattern Matching CLI Dispatcher (Lesson 12):
    • Process user CLI command tokens inside a match-case block with web harvesting match guards:
      • case ["CHECK", "ROBOTS", target_url] → Verify crawling permissions programmatically via urllib.robotparser.
      • case ["SCRAPE", "PAGE", target_url] → Download HTML, parse DOM nodes via bs4, and persist.
      • case ["AUDIT", "ELEMENTS"] → Route DOM element dictionaries using match-case pattern dispatcher.
      • case ["EXPORT", "CSV"] → Export completed harvest history to CSV.
      • case ["EXIT" | "QUIT"] → Terminate session safely using a sentinel flag.
      • case _ → Output command error message.
20. Conditionals, Formatting & Foundations (Lessons 1-11):
    • Sanitize all inputs using .strip() and .upper().
    • Format tabular reports using f-strings with field width alignment specifiers (:<15, :>10) and clear visual borders.