refactor and add tests, v0.2.0

2025-08-13 20:07:40 -07:00
parent 76be59254c
commit 809dbeb783
32 changed files with 1401 additions and 32 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -5,10 +5,11 @@ code in this repository.

 ## Project Overview

-EmbeddingBuddy is a Python Dash web application for interactive exploration and
+EmbeddingBuddy is a modular Python Dash web application for interactive exploration and
 visualization of embedding vectors through dimensionality reduction techniques
 (PCA, t-SNE, UMAP). The app provides a drag-and-drop interface for uploading
-NDJSON files containing embeddings and visualizes them in 2D/3D plots.
+NDJSON files containing embeddings and visualizes them in 2D/3D plots. The codebase
+follows a clean, modular architecture that prioritizes testability and maintainability.

 ## Development Commands

@@ -21,32 +22,79 @@ uv sync
 **Run the application:**

 ```bash
-uv run python app.py
+uv run python main.py
 ```

 The app will be available at http://127.0.0.1:8050

+**Run tests:**
+
+```bash
+uv run pytest tests/ -v
+```
+
 **Test with sample data:**
-Use the included `sample_data.ndjson` file for testing the application functionality.
+Use the included `sample_data.ndjson` and `sample_prompts.ndjson` files for testing the application functionality.

 ## Architecture

-### Core Files
+### Project Structure

- `app.py` - Main Dash application with complete web interface, data processing,
-   and visualization logic
- `main.py` - Simple entry point (currently minimal)
- `pyproject.toml` - Project configuration and dependencies using uv package manager
+The application follows a modular architecture with clear separation of concerns:
+
+```
+src/embeddingbuddy/
+├── app.py              # Main application entry point and factory
+├── main.py             # Application runner
+├── config/
+│   └── settings.py     # Centralized configuration management
+├── data/
+│   ├── parser.py       # NDJSON parsing logic
+│   └── processor.py    # Data transformation and processing
+├── models/
+│   ├── schemas.py      # Data models and validation schemas
+│   └── reducers.py     # Dimensionality reduction algorithms
+├── visualization/
+│   ├── plots.py        # Plot creation and factory classes
+│   └── colors.py       # Color mapping and management
+├── ui/
+│   ├── layout.py       # Main application layout
+│   ├── components/     # Reusable UI components
+│   │   ├── sidebar.py  # Sidebar component
+│   │   └── upload.py   # Upload components
+│   └── callbacks/      # Organized callback functions
+│       ├── data_processing.py  # Data upload/processing callbacks
+│       ├── visualization.py    # Plot update callbacks
+│       └── interactions.py     # User interaction callbacks
+└── utils/              # Utility functions and helpers
+```

 ### Key Components

- **Data Processing**: NDJSON parser that handles embedding documents with
-  required fields (`embedding`, `text`) and optional metadata (`id`, `category`, `subcategory`, `tags`)
- **Dimensionality Reduction**: Supports PCA, t-SNE (openTSNE), and UMAP algorithms
- **Visualization**: Plotly-based 2D/3D scatter plots with interactive features
- **UI Layout**: Bootstrap-styled sidebar with controls and large visualization area
- **State Management**: Dash callbacks for reactive updates between upload,
-  method selection, and plot rendering
+**Data Layer:**
+- `data/parser.py` - NDJSON parsing with error handling
+- `data/processor.py` - Data transformation and combination logic
+- `models/schemas.py` - Dataclasses for type safety and validation
+
+**Algorithm Layer:**
+- `models/reducers.py` - Modular dimensionality reduction with factory pattern
+- Supports PCA, t-SNE (openTSNE), and UMAP algorithms
+- Abstract base class for easy extension
+
+**Visualization Layer:**
+- `visualization/plots.py` - Plot factory with single and dual plot support
+- `visualization/colors.py` - Color mapping and grayscale conversion utilities
+- Plotly-based 2D/3D scatter plots with interactive features
+
+**UI Layer:**
+- `ui/layout.py` - Main application layout composition
+- `ui/components/` - Reusable, testable UI components
+- `ui/callbacks/` - Organized callbacks grouped by functionality
+- Bootstrap-styled sidebar with controls and large visualization area
+
+**Configuration:**
+- `config/settings.py` - Centralized settings with environment variable support
+- Plot styling, marker configurations, and app-wide constants

 ### Data Format

@@ -56,17 +104,77 @@ The application expects NDJSON files where each line contains:
 {"id": "doc_001", "embedding": [0.1, -0.3, 0.7, ...], "text": "Sample text", "category": "news", "subcategory": "politics", "tags": ["election"]}
 ```

+Required fields: `embedding` (array), `text` (string)
+Optional fields: `id`, `category`, `subcategory`, `tags`
+
 ### Callback Architecture

- File upload → Data processing and storage in dcc.Store
- Method/parameter changes → Dimensionality reduction and plot update
- Point clicks → Detail display in sidebar
+The refactored callback system is organized by functionality:
+
+**Data Processing (`ui/callbacks/data_processing.py`):**
+- File upload handling
+- NDJSON parsing and validation
+- Data storage in dcc.Store components
+
+**Visualization (`ui/callbacks/visualization.py`):**
+- Dimensionality reduction pipeline
+- Plot generation and updates
+- Method/parameter change handling
+
+**Interactions (`ui/callbacks/interactions.py`):**
+- Point click handling and detail display
+- Reset functionality
+- User interaction management
+
+### Testing Architecture
+
+The modular design enables comprehensive testing:
+
+**Unit Tests:**
+- `tests/test_data_processing.py` - Parser and processor logic
+- `tests/test_reducers.py` - Dimensionality reduction algorithms
+- `tests/test_visualization.py` - Plot creation and color mapping
+
+**Integration Tests:**
+- End-to-end data pipeline testing
+- Component integration verification
+
+**Key Testing Benefits:**
+- Fast test execution (milliseconds vs seconds)
+- Isolated component testing
+- Easy mocking and fixture creation
+- High code coverage achievable

 ## Dependencies

 Uses modern Python stack with uv for dependency management:

- Dash + Plotly for web interface and visualization
- scikit-learn (PCA), openTSNE, umap-learn for dimensionality reduction
- pandas/numpy for data manipulation
- dash-bootstrap-components for styling
+- **Core Framework:** Dash + Plotly for web interface and visualization
+- **Algorithms:** scikit-learn (PCA), openTSNE, umap-learn for dimensionality reduction
+- **Data:** pandas/numpy for data manipulation
+- **UI:** dash-bootstrap-components for styling
+- **Testing:** pytest for test framework
+- **Dev Tools:** uv for package management
+
+## Development Guidelines
+
+**When adding new features:**
+
+1. **Data Models** - Add/update schemas in `models/schemas.py`
+2. **Algorithms** - Extend `models/reducers.py` using the abstract base class
+3. **UI Components** - Create reusable components in `ui/components/`
+4. **Configuration** - Add settings to `config/settings.py`
+5. **Tests** - Write tests for all new functionality
+
+**Code Organization Principles:**
+- Single responsibility principle
+- Clear module boundaries  
+- Testable, isolated components
+- Configuration over hardcoding
+- Error handling at appropriate layers
+
+**Testing Requirements:**
+- Unit tests for all core logic
+- Integration tests for data flow
+- Component tests for UI elements
+- Maintain high test coverage