🎯 ISSUE RESOLVED: Profit calculations were different between pages due to inconsistent date filtering
📊 ROOT CAUSE:
- Profit Analysis Dashboard: "2026" = calendar year 2026 (Jan-Dec 2026)
- Analytics Page: "This Year" = rolling 365 days (May 2025-May 2026)
🔧 FIXES APPLIED:
- Unified date filtering logic using consistent calendar year approach
- Added "2026 Full Year" option to Analytics page for exact comparison
- Changed default to "2026" to match Profit Analysis dashboard
- Updated "This Year" to mean current calendar year (2026), not rolling 365 days
✅ NOW BOTH PAGES SHOW IDENTICAL DATA when comparing same date ranges!
The profit calculations will now match exactly when comparing equivalent time periods.
🚀 PERFORMANCE IMPROVEMENTS:
- Add bulk expenses API endpoint (/expenses/bulk) for fast batch processing
- Replace slow sequential processing (3 seconds per expense) with instant bulk operations
- Use MongoDB insertMany with ordered:false for optimal bulk inserts
- Handle duplicates, validation errors, and partial failures gracefully
🧹 CODE CLEANUP:
- Remove excessive date parsing test logs that were spamming console
- Clean up unused imports and test code
- Improve error handling with detailed bulk operation results
⚡ SPEED IMPROVEMENT:
- Before: 60+ seconds for 20 expenses (3 seconds each)
- After: <2 seconds for any number of expenses (bulk operation)
This eliminates the painfully slow background processing while maintaining duplicate prevention and error handling.
- Add proper array validation for existingExpenses API response
- Handle different API response structures (data.expenses vs data)
- Add try-catch blocks around individual expense processing loops
- Improve error messages to show specific error details
- Prevent TypeError when API response format is unexpected
Resolves 'gt.some is not a function' error in background processing.
- Process expenses immediately when CSV files are uploaded, not when user clicks analyze
- Use much more conservative rate limiting (3 seconds between expenses) to prevent 429 errors
- Remove bulk batch processing that was overwhelming the rate limiter
- Add background processing status indicator with spinner animation
- Clean up unused state variables and functions
- Improve user experience by eliminating long wait times during analysis
This prevents HTTP 429 rate limiting errors by spreading expense creation over time naturally as files are uploaded, rather than trying to process 60+ expenses rapidly in batches.
- Reduce batch size from 3 to 2 expenses per batch for more conservative processing
- Increase delay between batches from 1.5s to 2s to prevent HTTP 429 errors
- Add isProcessingExpenses state to prevent multiple simultaneous imports
- Disable import buttons during expense processing for better UX
- Ensure processing state is always cleared in finally block
Resolves HTTP 429 rate limiting errors during bulk expense creation from CSV imports.
Frontend improvements:
- Add batch processing for expense creation (3 expenses per batch)
- Implement 1.5 second delays between batches to avoid overwhelming server
- Better progress logging and user feedback during batch processing
- Handle rate limit errors gracefully with proper error categorization
Backend improvements:
- Add specific rate limiter for expense creation endpoint (50 per minute)
- More informative error messages for rate limit violations
- Separate rate limiting for expense creation vs general API usage
This prevents the HTTP 429 'Too Many Requests' errors when importing large CSV files with many individual expense records (listing fees, ads, GST entries, etc.).
- Remove unused orderCosts parameter from createExpensesFromCsvData function
- Update function call to match new signature
- Upgrade GitHub Actions to use Node.js 24 instead of deprecated Node.js 20
- Add FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 environment variable
This should resolve the build failure and deprecation warnings in CI/CD.
Major improvements to expense categorization and profit calculations:
NEW: Granular Etsy Fee Parsing
- Parse individual Etsy fees with proper categorization:
* Listing Fees (should be included in expenses)
* Marketing & Advertising (Etsy Ads - should be included)
* Transaction Fees (tied to orders - excluded to avoid double-counting)
* Processing Fees (tied to orders - excluded to avoid double-counting)
* Taxes & GST (should be included)
* Shipping & Postage (should be included)
FIXED: Profit Calculation Logic
- Only exclude sale transaction fees that have order references
- Include all other Etsy business expenses (ads, listing fees, GST)
- More accurate profit margins that account for all business costs
ENHANCED: CSV Import
- Creates specific expense categories instead of lumping as 'Transaction Fees'
- Better duplicate detection based on description, amount, date, and category
- Improved user feedback showing specific fee types created
This fixes the issue where listing fees and advertising costs were incorrectly excluded from profit calculations, resulting in unrealistically high profit margins.
- Added type=raw,value=main to both API and Nginx image metadata
- This will create both SHA tags and :main tags for the default branch
- Fixes deployment timeout issues when pulling from GHCR
- Also added temporary deployment file using specific SHA
Multi-layer duplicate prevention system:
- Enhanced frontend duplicate detection with tracking number, amount, and date comparison
- Added MongoDB compound index to prevent database-level duplicates
- Improved backend error handling for duplicate key violations
- Added cleanup endpoint to remove existing duplicates
- Enhanced user feedback for import operations
Frontend changes:
- Stricter duplicate detection comparing tracking number, vendor, amount, and date
- Better error handling and user feedback for duplicate scenarios
- Added 'Clean Duplicates' button to remove existing duplicates
Backend changes:
- Database compound index on reference, vendor, userId, amount, date
- Enhanced error responses with duplicate detection flags
- New POST /expenses/cleanup-duplicates endpoint
- Improved duplicate key error handling
This should eliminate the double Australia Post expense entries.
- Updated ProfitAnalysisService to accept and integrate expense data
- Modified calculateProfitMetrics to include expenses in profit calculations
- Added totalExpenses field to ProfitMetrics interface
- Updated ProfitAnalysis component to pass filtered expenses to calculations
- Enhanced UI to show breakdown of expenses vs printing costs
- Fixed inconsistency between Analytics and ProfitAnalysis profit calculations
- Now shipping expenses and Etsy fees are properly deducted from profits
- Excluded transaction fees to avoid double-counting
✅ FIXES MAJOR WORKFLOW ISSUE:
- Data Import CSV processing now automatically creates expenses
- Etsy fees → 'Transaction Fees' category expenses
- Australia Post shipping → 'Shipping & Postage' expenses
- Prevents duplicate expense creation with reference checking
- Added user notices in both Data Import and Expenses tabs
🔄 INTEGRATION FEATURES:
- Automatic expense creation from CSV order costs
- Duplicate prevention by order number/tracking number
- Proper categorization and tax-deductible flagging
- Clear user notifications and workflow guidance
📊 USER EXPERIENCE:
- Expenses tab now shows integration notice
- Data Import shows automatic expense creation info
- Seamless workflow between sales data and expense tracking
- Add duplicate prevention with isProcessing state
- Implement loading spinner and disabled states
- Enhance modal closure logic in DataImport.tsx
- Add proper state reset on modal close
- Prevent multiple button clicks during processing
- Improve user feedback with console logging
Enhanced product matching algorithm to properly handle:
- Large vs Small size variants for shelf decor products
- Improved size extraction from product titles
- Higher confidence thresholds for shelf decor products to prevent wrong matches
- Better handling of 3D-Printed product variations
- Enhanced logging for debugging matching issues
This fixes the issue where packing slips would create duplicate products
instead of matching to existing Large/Small variants of the same product.
Features Added:
- Enhanced CSV import service with smart parsing for product data
- Two CSV templates: simple and complete with example data
- Advanced product import UI with template downloads
- Comprehensive CSV import documentation and guide
- Support for all product fields including costs, dimensions, materials
- Error handling and validation for bulk imports
- Integration with profit analysis system for accurate cost tracking
UI Improvements:
- Template download buttons (Simple Template, Full Template)
- Enhanced import workflow with progress feedback
- Better error messaging and validation
- Updated Products page with CSV functionality
Documentation:
- Complete CSV_IMPORT_GUIDE.md with examples and best practices
- Updated DEPLOYMENT_STATUS.md reflecting production readiness
- Enhanced README.md with current feature status
- CHANGELOG.md with comprehensive development history
Production Ready:
- All functionality tested and integrated with existing profit analysis
- Bulk import capability for efficient product catalog setup
- Supports accurate cost tracking for comprehensive profit margins
Missing target: production caused the API image to build the last
Dockerfile stage (nginx-frontend) instead of the Node.js server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The default :root styles (dark background, white text, centered #root)
were overriding Tailwind and making all pages appear blank/dark.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove localStorage from all 4 Redux slices (products, orders, expenses, customers)
- Layout fetches all data from API on mount; adds logout button with active nav highlighting
- Wire API calls in Products, Orders, Expenses pages for all CRUD operations
- DataImport uses POST /orders/bulk for CSV upserts and API for PDF slip orders
- MissingProductsModal creates products via API
- Relax Order model: optional customerId, embedded customer, fees, printingCost on items, default paymentStatus=paid
- Relax Expense model: free-string category, add taxDeductible/vendor/reference fields
- Add printingCost to Product model
- Add POST /orders/bulk endpoint for upsert-by-orderNumber
- Raise rate limit to 1000 req/15min for bulk imports
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frontend:
- Login and Register pages wired up to API
- PrivateRoute redirects unauthenticated users to /login
- Token persisted in localStorage, restored on page load
- Axios instance automatically attaches Bearer token, redirects on 401
Backend:
- userId field added to all models (Product, Order, Customer, Expense)
- All queries scoped to authenticated user's userId
- Register/login return JWT token
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- User model with bcrypt password hashing
- Register, login, logout, and /me endpoints
- authenticate middleware applied to all API routes
- JWT_SECRET configurable via environment variable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Enable MongoDB connection in server
- Add Customer and Expense models
- Implement full CRUD for products, orders, customers, expenses
- Implement analytics dashboard with revenue, sales chart, top products/customers
- Add MongoDB service to docker-compose.deploy.yml with persistent volume
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The runtime cp from etsy-tracker to a shared volume silently failed
because the nodejs user lacked write permissions on the root-owned volume.
Now a dedicated nginx-frontend Docker stage is built and pushed to GHCR
with the React client baked in, eliminating the volume-sharing approach.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fixed nginx 404 errors with enhanced file copying and fallback configuration
- Added docker-compose.simple.yml for streamlined first-time deployment
- Enhanced docker-compose.deploy-local.yml with detailed debugging and health checks
- Improved nginx.deploy.conf with fallback pages and auto-refresh
- Added comprehensive DEPLOYMENT_GUIDE.md with multiple deployment options
- Created validate-deployment.sh script for environment validation
- Updated container interface deployment documentation
- Added DEPLOYMENT_STATUS.md summary of ready features
Deployment improvements:
- Enhanced container startup sequence with health checks
- Detailed logging for troubleshooting file copying issues
- Multiple deployment strategies for different use cases
- Fallback nginx configuration prevents 404 errors during startup
- Auto-refresh functionality for seamless user experience
Ready for production deployment via container interfaces or command line.
🔧 Deployment Fixes:
- Change default compose file to docker-compose.deploy-local.yml
- Remove obsolete 'version' attribute from compose files
- Ensures deployment works immediately without GitHub Container Registry
✅ Ready to Deploy:
- Use docker-compose.deploy-local.yml (builds locally)
- No more 'version is obsolete' warnings
- No dependency on GitHub Container Registry access
This makes deployment work out-of-the-box while GHCR image access is resolved.
🔧 Fix Docker Compose YAML Issue:
- Remove duplicate 'command' key that was causing parse error
- Keep the cleaner command with better logging
- Ensures proper YAML structure for container interfaces
✅ Now Ready for Deployment:
- Valid YAML syntax for Portainer/Docker Desktop
- Single command block for container startup
- Clear logging for troubleshooting
This fixes the 'mapping key command already defined' error.
🔧 Container Interface Deployment Fix:
- Use nginx.deploy.conf for deployment-specific nginx config
- Add command to copy client files to shared volume
- Improve startup logging and error handling
- Ensure client files are available to nginx container
✅ Deployment Ready:
- Works with Portainer, Docker Desktop, and similar interfaces
- Copies React build files from API container to nginx volume
- Proper container startup sequence and health checks
- Clear logging for troubleshooting startup issues
This fixes the missing client files issue when deploying from container management interfaces.
🔧 Critical Fix:
- Remove --only=production from client build stage (needs devDependencies)
- Remove --only=production from server build stage (needs TypeScript, etc.)
- Add production dependencies install in final stage only
- Remove duplicate npm ci commands
🏗️ Build Process Fixed:
- Client: npm ci (all deps) → build with TypeScript/Vite → copy dist/
- Server: npm ci (all deps) → build with TypeScript → copy dist/
- Production: Install only runtime dependencies for final image
⚡ Why This Fixes It:
- Build tools (TypeScript, Vite) are devDependencies
- --only=production excludes build tools, causing 'npm run build' to fail
- Final production stage still gets minimal runtime dependencies
This should resolve the 'npm run build exit code 127' Docker build error.
Architecture Updates:
- Nginx serves static React files for optimal performance
- Nginx proxies API requests to Node.js backend (port 8080)
- Separation of concerns: static files vs API handling
- Professional production setup with proper caching
Features Added:
- nginx.conf with optimized configuration:
- Static file serving with long-term caching
- API reverse proxy with rate limiting
- Security headers and GZIP compression
- Health check proxying and SPA routing support
- Updated docker-compose.yml for multi-container setup
- build-deploy.sh script for automated deployment
- Updated environment configuration for container networking
Security & Performance:
- Rate limiting on API and auth endpoints
- Security headers (XSS, CSRF, clickjacking protection)
- GZIP compression for static assets
- Proper cache control headers
- Container-to-container communication
Deployment:
- Single command deployment with ./build-deploy.sh
- Nginx on port 80 (exposed as 3000) serving React app
- API server on internal port 8080 (not exposed)
- Persistent data volume mounting for business files
Features:
- Multi-stage Dockerfile for optimized production builds
- Docker Compose configuration with health checks
- Health check endpoint for container monitoring
- Production environment configuration template
- Comprehensive deployment guide for Mac Mini
- Docker ignore file for efficient build context
- Security: Non-root user, proper signal handling
- Persistence: Data directory volume mounting
- Performance: Alpine Linux base, optimized layers
- Future-ready: MongoDB service configuration (commented)
Deployment:
- Simple 'docker-compose up' deployment
- Automatic health monitoring and restart policies
- Persistent data storage with volume mounts
- Port configuration and environment customization
- Complete troubleshooting and management guide
- Create secure data/ directory structure for local business files
- Add comprehensive data organization guide with security notes
- Move all sensitive files to protected data/ directory
- Enhanced .gitignore to exclude entire data/ directory (except README)
- Provide clear instructions for local data file management
- Ensure no business data can accidentally be committed to public repo
- Remove Excel business tracker spreadsheet from git tracking
- Enhanced .gitignore to exclude all business documents:
- CSV files, Excel files, PDF documents
- Transaction statements and business trackers
- Preserve essential config files (package.json, tsconfig, etc.)
- Protect sensitive financial and customer data from public exposure
Features:
- React + TypeScript frontend with Tailwind CSS
- Node.js + Express backend with TypeScript
- Comprehensive order tracking and management
- Product catalog with inventory tracking
- Customer data management
- Expense tracking and categorization
- Advanced Profit Analysis Dashboard with:
- Real-time profit metrics and KPI visualization
- Detailed order-level profit breakdown
- Product performance analysis
- Enhanced time range filtering (monthly, quarterly, yearly)
- Interactive expandable order analysis
- Performance categorization and color coding
- CSV import functionality for Etsy statements
- PDF parsing capabilities
- Redux state management with persistence
- Responsive design with mobile support
- Australian date formatting and currency display
