mirror of
https://github.com/ghndrx/file-transformer-s3.git
synced 2026-02-10 06:45:05 +00:00
Initial commit: File Transformer S3 project with React dashboard and Knative functions
This commit is contained in:
greg
264
README.md
Normal file
264
README.md
Normal file
@@ -0,0 +1,264 @@
|
||||
# File Transformer S3
|
||||
|
||||
A comprehensive file transformation system with a React dashboard, Knative functions, PostgreSQL database, and MinIO S3-compatible storage.
|
||||
|
||||
## 🚀 Features
|
||||
|
||||
- **Modern React Dashboard**: Beautiful, responsive UI for managing files and transformations
|
||||
- **Knative Functions**: Serverless Python functions for file processing
|
||||
- **PostgreSQL Database**: Robust data storage with comprehensive schema
|
||||
- **MinIO Storage**: S3-compatible object storage
|
||||
- **Environment-Driven**: Fully configurable via environment variables
|
||||
- **Docker & Kubernetes**: Complete containerization and orchestration
|
||||
- **Automated Setup**: Makefile for easy deployment and management
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ React │ │ Knative │ │ PostgreSQL │
|
||||
│ Dashboard │◄──►│ Functions │◄──►│ Database │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│ │ │
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ MinIO │ │ API Gateway │ │ File Storage │
|
||||
│ Console │ │ (Optional) │ │ & Processing │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
- Docker and Docker Compose
|
||||
- Kubernetes cluster with Knative installed
|
||||
- kubectl configured
|
||||
- Node.js 18+ (for development)
|
||||
- Python 3.11+ (for development)
|
||||
|
||||
## 🛠️ Quick Start
|
||||
|
||||
### 1. Clone and Setup
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd file-transformer-s3
|
||||
make setup
|
||||
```
|
||||
|
||||
### 2. Configure Environment
|
||||
|
||||
Edit the `.env` file with your configuration:
|
||||
|
||||
```bash
|
||||
cp env.example .env
|
||||
# Edit .env with your settings
|
||||
```
|
||||
|
||||
### 3. Local Development
|
||||
|
||||
```bash
|
||||
# Start all services locally
|
||||
make deploy-local
|
||||
|
||||
# Or start individual components
|
||||
make dev-dashboard
|
||||
make dev-functions
|
||||
```
|
||||
|
||||
### 4. Kubernetes Deployment
|
||||
|
||||
```bash
|
||||
# Deploy to Knative cluster
|
||||
make deploy-knative
|
||||
|
||||
# Check status
|
||||
make status
|
||||
```
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
file-transformer-s3/
|
||||
├── dashboard/ # React frontend
|
||||
│ ├── src/
|
||||
│ │ ├── components/ # React components
|
||||
│ │ ├── pages/ # Page components
|
||||
│ │ ├── services/ # API services
|
||||
│ │ └── contexts/ # React contexts
|
||||
│ ├── Dockerfile
|
||||
│ └── package.json
|
||||
├── functions/ # Knative functions
|
||||
│ ├── upload/ # File upload function
|
||||
│ ├── transform/ # File transformation function
|
||||
│ ├── download/ # File download function
|
||||
│ ├── metadata/ # File metadata function
|
||||
│ └── requirements.txt
|
||||
├── k8s/ # Kubernetes manifests
|
||||
│ ├── namespace.yaml
|
||||
│ ├── postgres.yaml
|
||||
│ ├── minio.yaml
|
||||
│ ├── dashboard.yaml
|
||||
│ └── functions/
|
||||
├── database/ # Database scripts
|
||||
│ └── init.sql
|
||||
├── docker-compose.yml # Local development
|
||||
├── Makefile # Automation scripts
|
||||
├── env.example # Environment template
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Key configuration options in `.env`:
|
||||
|
||||
```bash
|
||||
# Application
|
||||
APP_NAME=file-transformer-s3
|
||||
APP_ENV=development
|
||||
|
||||
# Dashboard
|
||||
REACT_APP_PORT=3000
|
||||
REACT_APP_API_BASE_URL=http://localhost:8080
|
||||
|
||||
# PostgreSQL
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_DB=file_transformer
|
||||
POSTGRES_USER=file_user
|
||||
POSTGRES_PASSWORD=secure_password_123
|
||||
|
||||
# MinIO
|
||||
MINIO_ENDPOINT=localhost:9000
|
||||
MINIO_ACCESS_KEY=minioadmin
|
||||
MINIO_SECRET_KEY=minioadmin123
|
||||
MINIO_BUCKET_NAME=file-transformer-bucket
|
||||
|
||||
# Knative Functions
|
||||
KNATIVE_NAMESPACE=file-transformer
|
||||
FUNCTION_UPLOAD_ENDPOINT=http://file-upload-service.file-transformer.svc.cluster.local
|
||||
FUNCTION_TRANSFORM_ENDPOINT=http://file-transform-service.file-transformer.svc.cluster.local
|
||||
```
|
||||
|
||||
## 🎯 Available Commands
|
||||
|
||||
### Setup & Installation
|
||||
```bash
|
||||
make setup # Initial setup
|
||||
make install-deps # Install dependencies
|
||||
make build-dashboard # Build React dashboard
|
||||
make build-functions # Build Knative functions
|
||||
```
|
||||
|
||||
### Deployment
|
||||
```bash
|
||||
make deploy-local # Deploy locally with Docker Compose
|
||||
make deploy-knative # Deploy to Knative cluster
|
||||
make deploy-all # Deploy everything
|
||||
```
|
||||
|
||||
### Management
|
||||
```bash
|
||||
make logs # View logs from all services
|
||||
make status # Check status of all services
|
||||
make clean # Clean up all resources
|
||||
make reset-db # Reset PostgreSQL database
|
||||
```
|
||||
|
||||
### Development
|
||||
```bash
|
||||
make dev-dashboard # Start dashboard in development mode
|
||||
make dev-functions # Start functions in development mode
|
||||
```
|
||||
|
||||
## 📊 Dashboard Features
|
||||
|
||||
- **File Management**: Upload, download, delete, and view files
|
||||
- **Transformation Pipeline**: Convert files between formats
|
||||
- **Real-time Monitoring**: Live status updates and progress tracking
|
||||
- **Analytics**: File type distribution and storage usage
|
||||
- **Bucket Management**: MinIO bucket operations
|
||||
- **User Management**: Authentication and authorization
|
||||
|
||||
## 🔄 File Transformations
|
||||
|
||||
Supported transformations:
|
||||
|
||||
- **Text Extraction**: Extract text from PDF and DOCX files
|
||||
- **Format Conversion**: CSV/Excel to JSON, image format conversion
|
||||
- **Image Processing**: Resize, compress, and convert images
|
||||
- **Document Processing**: PDF manipulation and text extraction
|
||||
|
||||
## 🗄️ Database Schema
|
||||
|
||||
Key tables:
|
||||
|
||||
- `files`: File metadata and storage information
|
||||
- `transformations`: Transformation job tracking
|
||||
- `buckets`: MinIO bucket management
|
||||
- `users`: User authentication and authorization
|
||||
- `sessions`: User session management
|
||||
- `file_access_logs`: Audit trail for file operations
|
||||
|
||||
## 🔒 Security
|
||||
|
||||
- JWT-based authentication
|
||||
- Role-based access control
|
||||
- Secure file upload validation
|
||||
- Audit logging for all operations
|
||||
- Environment variable configuration
|
||||
- Non-root container execution
|
||||
|
||||
## 📈 Monitoring & Logging
|
||||
|
||||
- Structured logging with structlog
|
||||
- Health check endpoints
|
||||
- Prometheus metrics (planned)
|
||||
- Real-time dashboard updates
|
||||
- Error tracking and reporting
|
||||
|
||||
## 🚀 Production Deployment
|
||||
|
||||
### Prerequisites
|
||||
- Kubernetes cluster with Knative
|
||||
- Ingress controller (nginx-ingress)
|
||||
- Persistent volume provisioner
|
||||
- Container registry access
|
||||
|
||||
### Deployment Steps
|
||||
1. Build and push container images
|
||||
2. Apply Kubernetes manifests
|
||||
3. Configure ingress and DNS
|
||||
4. Set up monitoring and logging
|
||||
5. Configure backups and disaster recovery
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Add tests
|
||||
5. Submit a pull request
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
For support and questions:
|
||||
- Create an issue in the repository
|
||||
- Check the documentation
|
||||
- Review the troubleshooting guide
|
||||
|
||||
## 🔮 Roadmap
|
||||
|
||||
- [ ] Advanced file transformations
|
||||
- [ ] Batch processing capabilities
|
||||
- [ ] Webhook integrations
|
||||
- [ ] Advanced analytics
|
||||
- [ ] Multi-tenant support
|
||||
- [ ] API rate limiting
|
||||
- [ ] Advanced security features
|
||||
Reference in New Issue
Block a user