file-transformer-s3/README.md

# File Transformer S3

A comprehensive file transformation system with a React dashboard, Knative functions, PostgreSQL database, and MinIO S3-compatible storage.

## 🚀 Features

- **Modern React Dashboard**: Beautiful, responsive UI for managing files and transformations
- **Knative Functions**: Serverless Python functions for file processing
- **PostgreSQL Database**: Robust data storage with comprehensive schema
- **MinIO Storage**: S3-compatible object storage
- **Environment-Driven**: Fully configurable via environment variables
- **Docker & Kubernetes**: Complete containerization and orchestration
- **Automated Setup**: Makefile for easy deployment and management

## 🏗️ Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   React         │    │   Knative       │    │   PostgreSQL    │
│   Dashboard     │◄──►│   Functions     │◄──►│   Database      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   MinIO         │    │   API Gateway   │    │   File Storage  │
│   Console       │    │   (Optional)    │    │   & Processing  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
```

## 📋 Prerequisites

- Docker and Docker Compose
- Kubernetes cluster with Knative installed
- kubectl configured
- Node.js 18+ (for development)
- Python 3.11+ (for development)

## 🛠️ Quick Start

### 1. Clone and Setup

```bash
git clone <repository-url>
cd file-transformer-s3
make setup
```

### 2. Configure Environment

Edit the `.env` file with your configuration:

```bash
cp env.example .env
# Edit .env with your settings
```

### 3. Local Development

```bash
# Start all services locally
make deploy-local

# Or start individual components
make dev-dashboard
make dev-functions
```

### 4. Kubernetes Deployment

```bash
# Deploy to Knative cluster
make deploy-knative

# Check status
make status
```

## 📁 Project Structure

```
file-transformer-s3/
├── dashboard/                 # React frontend
│   ├── src/
│   │   ├── components/       # React components
│   │   ├── pages/           # Page components
│   │   ├── services/        # API services
│   │   └── contexts/        # React contexts
│   ├── Dockerfile
│   └── package.json
├── functions/                # Knative functions
│   ├── upload/              # File upload function
│   ├── transform/           # File transformation function
│   ├── download/            # File download function
│   ├── metadata/            # File metadata function
│   └── requirements.txt
├── k8s/                     # Kubernetes manifests
│   ├── namespace.yaml
│   ├── postgres.yaml
│   ├── minio.yaml
│   ├── dashboard.yaml
│   └── functions/
├── database/                # Database scripts
│   └── init.sql
├── docker-compose.yml       # Local development
├── Makefile                 # Automation scripts
├── env.example              # Environment template
└── README.md
```

## 🔧 Configuration

### Environment Variables

Key configuration options in `.env`:

```bash
# Application
APP_NAME=file-transformer-s3
APP_ENV=development

# Dashboard
REACT_APP_PORT=3000
REACT_APP_API_BASE_URL=http://localhost:8080

# PostgreSQL
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=file_transformer
POSTGRES_USER=file_user
POSTGRES_PASSWORD=secure_password_123

# MinIO
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin123
MINIO_BUCKET_NAME=file-transformer-bucket

# Knative Functions
KNATIVE_NAMESPACE=file-transformer
FUNCTION_UPLOAD_ENDPOINT=http://file-upload-service.file-transformer.svc.cluster.local
FUNCTION_TRANSFORM_ENDPOINT=http://file-transform-service.file-transformer.svc.cluster.local
```

## 🎯 Available Commands

### Setup & Installation
```bash
make setup              # Initial setup
make install-deps       # Install dependencies
make build-dashboard    # Build React dashboard
make build-functions    # Build Knative functions
```

### Deployment
```bash
make deploy-local       # Deploy locally with Docker Compose
make deploy-knative     # Deploy to Knative cluster
make deploy-all         # Deploy everything
```

### Management
```bash
make logs               # View logs from all services
make status             # Check status of all services
make clean              # Clean up all resources
make reset-db           # Reset PostgreSQL database
```

### Development
```bash
make dev-dashboard      # Start dashboard in development mode
make dev-functions      # Start functions in development mode
```

## 📊 Dashboard Features

- **File Management**: Upload, download, delete, and view files
- **Transformation Pipeline**: Convert files between formats
- **Real-time Monitoring**: Live status updates and progress tracking
- **Analytics**: File type distribution and storage usage
- **Bucket Management**: MinIO bucket operations
- **User Management**: Authentication and authorization

## 🔄 File Transformations

Supported transformations:

- **Text Extraction**: Extract text from PDF and DOCX files
- **Format Conversion**: CSV/Excel to JSON, image format conversion
- **Image Processing**: Resize, compress, and convert images
- **Document Processing**: PDF manipulation and text extraction

## 🗄️ Database Schema

Key tables:

- `files`: File metadata and storage information
- `transformations`: Transformation job tracking
- `buckets`: MinIO bucket management
- `users`: User authentication and authorization
- `sessions`: User session management
- `file_access_logs`: Audit trail for file operations

## 🔒 Security

- JWT-based authentication
- Role-based access control
- Secure file upload validation
- Audit logging for all operations
- Environment variable configuration
- Non-root container execution

## 📈 Monitoring & Logging

- Structured logging with structlog
- Health check endpoints
- Prometheus metrics (planned)
- Real-time dashboard updates
- Error tracking and reporting

## 🚀 Production Deployment

### Prerequisites
- Kubernetes cluster with Knative
- Ingress controller (nginx-ingress)
- Persistent volume provisioner
- Container registry access

### Deployment Steps
1. Build and push container images
2. Apply Kubernetes manifests
3. Configure ingress and DNS
4. Set up monitoring and logging
5. Configure backups and disaster recovery

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

## 📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🆘 Support

For support and questions:
- Create an issue in the repository
- Check the documentation
- Review the troubleshooting guide

## 🔮 Roadmap

- [ ] Advanced file transformations
- [ ] Batch processing capabilities
- [ ] Webhook integrations
- [ ] Advanced analytics
- [ ] Multi-tenant support
- [ ] API rate limiting
- [ ] Advanced security features