File Transformer S3

A comprehensive file transformation system with a React dashboard, Knative functions, PostgreSQL database, and MinIO S3-compatible storage.

🚀 Features

• Modern React Dashboard: Beautiful, responsive UI for managing files and transformations
• Knative Functions: Serverless Python functions for file processing
• PostgreSQL Database: Robust data storage with comprehensive schema
• MinIO Storage: S3-compatible object storage
• Environment-Driven: Fully configurable via environment variables
• Docker & Kubernetes: Complete containerization and orchestration
• Automated Setup: Makefile for easy deployment and management

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   React         │    │   Knative       │    │   PostgreSQL    │
│   Dashboard     │◄──►│   Functions     │◄──►│   Database      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   MinIO         │    │   API Gateway   │    │   File Storage  │
│   Console       │    │   (Optional)    │    │   & Processing  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

📋 Prerequisites

• Docker and Docker Compose
• Kubernetes cluster with Knative installed
• kubectl configured
• Node.js 18+ (for development)
• Python 3.11+ (for development)

🛠️ Quick Start

1. Clone and Setup

git clone <repository-url>
cd file-transformer-s3
make setup

2. Configure Environment

Edit the .env file with your configuration:

cp env.example .env
# Edit .env with your settings

3. Local Development

# Start all services locally
make deploy-local

# Or start individual components
make dev-dashboard
make dev-functions

4. Kubernetes Deployment

# Deploy to Knative cluster
make deploy-knative

# Check status
make status

📁 Project Structure

file-transformer-s3/
├── dashboard/                 # React frontend
│   ├── src/
│   │   ├── components/       # React components
│   │   ├── pages/           # Page components
│   │   ├── services/        # API services
│   │   └── contexts/        # React contexts
│   ├── Dockerfile
│   └── package.json
├── functions/                # Knative functions
│   ├── upload/              # File upload function
│   ├── transform/           # File transformation function
│   ├── download/            # File download function
│   ├── metadata/            # File metadata function
│   └── requirements.txt
├── k8s/                     # Kubernetes manifests
│   ├── namespace.yaml
│   ├── postgres.yaml
│   ├── minio.yaml
│   ├── dashboard.yaml
│   └── functions/
├── database/                # Database scripts
│   └── init.sql
├── docker-compose.yml       # Local development
├── Makefile                 # Automation scripts
├── env.example              # Environment template
└── README.md

🔧 Configuration

Environment Variables

Key configuration options in .env:

# Application
APP_NAME=file-transformer-s3
APP_ENV=development

# Dashboard
REACT_APP_PORT=3000
REACT_APP_API_BASE_URL=http://localhost:8080

# PostgreSQL
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=file_transformer
POSTGRES_USER=file_user
POSTGRES_PASSWORD=secure_password_123

# MinIO
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin123
MINIO_BUCKET_NAME=file-transformer-bucket

# Knative Functions
KNATIVE_NAMESPACE=file-transformer
FUNCTION_UPLOAD_ENDPOINT=http://file-upload-service.file-transformer.svc.cluster.local
FUNCTION_TRANSFORM_ENDPOINT=http://file-transform-service.file-transformer.svc.cluster.local

🎯 Available Commands

Setup & Installation

make setup              # Initial setup
make install-deps       # Install dependencies
make build-dashboard    # Build React dashboard
make build-functions    # Build Knative functions

Deployment

make deploy-local       # Deploy locally with Docker Compose
make deploy-knative     # Deploy to Knative cluster
make deploy-all         # Deploy everything

Management

make logs               # View logs from all services
make status             # Check status of all services
make clean              # Clean up all resources
make reset-db           # Reset PostgreSQL database

Development

make dev-dashboard      # Start dashboard in development mode
make dev-functions      # Start functions in development mode

📊 Dashboard Features

• File Management: Upload, download, delete, and view files
• Transformation Pipeline: Convert files between formats
• Real-time Monitoring: Live status updates and progress tracking
• Analytics: File type distribution and storage usage
• Bucket Management: MinIO bucket operations
• User Management: Authentication and authorization

🔄 File Transformations

Supported transformations:

• Text Extraction: Extract text from PDF and DOCX files
• Format Conversion: CSV/Excel to JSON, image format conversion
• Image Processing: Resize, compress, and convert images
• Document Processing: PDF manipulation and text extraction

🗄️ Database Schema

Key tables:

• files: File metadata and storage information
• transformations: Transformation job tracking
• buckets: MinIO bucket management
• users: User authentication and authorization
• sessions: User session management
• file_access_logs: Audit trail for file operations

🔒 Security

• JWT-based authentication
• Role-based access control
• Secure file upload validation
• Audit logging for all operations
• Environment variable configuration
• Non-root container execution

📈 Monitoring & Logging

• Structured logging with structlog
• Health check endpoints
• Prometheus metrics (planned)
• Real-time dashboard updates
• Error tracking and reporting

🚀 Production Deployment

Prerequisites

• Kubernetes cluster with Knative
• Ingress controller (nginx-ingress)
• Persistent volume provisioner
• Container registry access

Deployment Steps

1. Build and push container images
2. Apply Kubernetes manifests
3. Configure ingress and DNS
4. Set up monitoring and logging
5. Configure backups and disaster recovery

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For support and questions:

• Create an issue in the repository
• Check the documentation
• Review the troubleshooting guide

🔮 Roadmap

• Advanced file transformations
• Batch processing capabilities
• Webhook integrations
• Advanced analytics
• Multi-tenant support
• API rate limiting
• Advanced security features