- syncListings pages active listings, fetches each listing's inventory,
and upserts Products (one per size variation, collapsing colour).
Title/description/price/tags/SKU/quantity and etsyListingId are
updated; printingCost/costOfGoods are preserved (user-owned). Old
titles are kept as aliases so historical order matching still works.
- POST /api/etsy/sync-catalog endpoint; Settings gains a 'Sync Catalog'
button that refreshes products after running.
- Uses existing listings_r scope (no reconnect needed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unique expense index can silently fail to build over pre-existing
duplicate data, so re-syncs were re-adding ledger fees every run.
- syncLedgerEntries now explicitly checks existing references (and
de-dupes within the batch) instead of trusting the unique index
- dedupeLedgerExpenses keeps one row per etsy-ledger-<entry_id> and
deletes the rest; runs automatically at the start of each sync so
existing duplicates self-heal. Distinct entries sharing a date/amount
are untouched (each has its own reference).
- Sync response/toast report deduped and already-imported counts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Order model gains refundTotal; sync sums receipt.refunds (Money
objects) onto each order and marks fully-refunded orders 'refunded'
- orderNetRevenue() helper (total - refundTotal); profit metrics,
monthly trends, and per-order analysis now use net revenue so
refunded sales no longer overstate profit
- ProfitMetrics adds totalRefunds; Analytics revenue card shows net
revenue with the refunded amount, and monthly revenue is net
Note: product-level profitability stays gross since receipt refunds are
order-level amounts with no line-item breakdown.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- 'prolist' (Etsy Promoted Listings / onsite ads) now maps to
Marketing & Advertising instead of being skipped
- classifyLedgerEntry distinguishes 'skip' (credits, disbursements,
refunds — deliberately not expenses) from null (unrecognised debit).
Only null entries are reported, so disbursements/refunds no longer
appear in the 'unknown charge types' list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ledger-entries endpoint rejects ranges over 2678400s (31 days) with
a 400. Reduce the chunk window from 90 to 30 days.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- syncLedgerEntries pulls payment-account ledger entries (chunked by 90d
over required min_created/max_created), classifies debits into fee
categories, and inserts them as idempotent expenses (reference
etsy-ledger-<entry_id>). Amounts are integer minor units -> /100.
- Conservative classifier: only genuine costs become expenses; sale
credits, disbursements, and refunds are skipped. Unclassified debits
are reported back so rules can be refined.
- Folded into POST /api/etsy/sync alongside orders; response includes
ledger result and a legacyEtsyExpenses count (pre-ledger CSV fees).
- DELETE /api/etsy/legacy-fees removes CSV-imported Etsy fee expenses to
avoid double counting; Settings surfaces the count with a one-click
remove, plus a list of skipped/unknown ledger charge types.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One listing can sell multiple sizes that map to different catalog
products with different costs. Match 'title + variant' before bare
title so size-specific products win, and report unmatched items with
their variant suffix so each size resolves to its own product.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etsy v3 expects both credentials joined by a colon in the x-api-key
header; sending the shared secret alone returns 403 'incorrect shared
secret for API key'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etsy returns 403 'Shared secret is required in x-api-key header' when the
keystring is used on resource endpoints: the keystring is only the OAuth
client id. Store the shared secret alongside it (Settings form + env
fallback) and send it as x-api-key on users/me and receipt requests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Callback failures now redirect with the underlying error message, shown
in the Settings toast, instead of a generic failure
- Token endpoint requests use application/x-www-form-urlencoded per
RFC 6749 instead of JSON
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New EtsySettings model holds the per-user API keystring and callback URL,
managed via GET/PUT /api/etsy/config; env vars remain as optional fallback
- Settings UI gains an API Configuration form (masked saved key, callback URL
prefilled with this origin's /api/etsy/callback); Connect is enabled once
configuration is saved
- OAuth and sync resolve the key per user; post-callback redirect derives
from the stored callback URL origin instead of CLIENT_URL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- OAuth 2.0 authorization code flow with PKCE; /api/etsy/connect returns the
consent URL, /api/etsy/callback exchanges the code (validated via one-time
state, no JWT) and stores tokens per user with automatic refresh
- /api/etsy/sync pulls all shop receipts and upserts orders by receipt id:
items with SKU/variations, totals, shipping address, tracking, and status,
with catalog costs snapshotted at sync time
- Product matching by exact title/alias first, then etsyListingId
(size-disambiguated); listing ids are learned onto products on first match
- Packing-slip items with cost data are preserved when synced items can't all
be matched
- Settings page: Connect Etsy Shop, Sync Orders Now, Disconnect, and a list
of unmatched item titles after each sync
- Requires ETSY_API_KEY and ETSY_REDIRECT_URI env vars (see .env.example)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Products store packing-slip title aliases; matcher checks exact title/alias
first so previously-matched items skip fuzzy matching entirely
- Packing-slip imports snapshot printingCost/costOfGoods/productId onto order
items; profit analysis reads stored costs so catalog edits don't rewrite history
- Etsy statement fees allocate to orders via Order # references in Title/Info
instead of date proximity; shop-level fees (listings, ads) no longer leak
into order fees
- Remove broken transaction-fee exclusion guard (order totals are gross, so
all expenses count once)
- Remove debug buttons, date-fix banner, test files, and console.log noise;
Clear All Orders now uses a bulk DELETE /orders endpoint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 PERFORMANCE IMPROVEMENTS:
- Add bulk expenses API endpoint (/expenses/bulk) for fast batch processing
- Replace slow sequential processing (3 seconds per expense) with instant bulk operations
- Use MongoDB insertMany with ordered:false for optimal bulk inserts
- Handle duplicates, validation errors, and partial failures gracefully
🧹 CODE CLEANUP:
- Remove excessive date parsing test logs that were spamming console
- Clean up unused imports and test code
- Improve error handling with detailed bulk operation results
⚡ SPEED IMPROVEMENT:
- Before: 60+ seconds for 20 expenses (3 seconds each)
- After: <2 seconds for any number of expenses (bulk operation)
This eliminates the painfully slow background processing while maintaining duplicate prevention and error handling.
Frontend improvements:
- Add batch processing for expense creation (3 expenses per batch)
- Implement 1.5 second delays between batches to avoid overwhelming server
- Better progress logging and user feedback during batch processing
- Handle rate limit errors gracefully with proper error categorization
Backend improvements:
- Add specific rate limiter for expense creation endpoint (50 per minute)
- More informative error messages for rate limit violations
- Separate rate limiting for expense creation vs general API usage
This prevents the HTTP 429 'Too Many Requests' errors when importing large CSV files with many individual expense records (listing fees, ads, GST entries, etc.).
Multi-layer duplicate prevention system:
- Enhanced frontend duplicate detection with tracking number, amount, and date comparison
- Added MongoDB compound index to prevent database-level duplicates
- Improved backend error handling for duplicate key violations
- Added cleanup endpoint to remove existing duplicates
- Enhanced user feedback for import operations
Frontend changes:
- Stricter duplicate detection comparing tracking number, vendor, amount, and date
- Better error handling and user feedback for duplicate scenarios
- Added 'Clean Duplicates' button to remove existing duplicates
Backend changes:
- Database compound index on reference, vendor, userId, amount, date
- Enhanced error responses with duplicate detection flags
- New POST /expenses/cleanup-duplicates endpoint
- Improved duplicate key error handling
This should eliminate the double Australia Post expense entries.
- Remove localStorage from all 4 Redux slices (products, orders, expenses, customers)
- Layout fetches all data from API on mount; adds logout button with active nav highlighting
- Wire API calls in Products, Orders, Expenses pages for all CRUD operations
- DataImport uses POST /orders/bulk for CSV upserts and API for PDF slip orders
- MissingProductsModal creates products via API
- Relax Order model: optional customerId, embedded customer, fees, printingCost on items, default paymentStatus=paid
- Relax Expense model: free-string category, add taxDeductible/vendor/reference fields
- Add printingCost to Product model
- Add POST /orders/bulk endpoint for upsert-by-orderNumber
- Raise rate limit to 1000 req/15min for bulk imports
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frontend:
- Login and Register pages wired up to API
- PrivateRoute redirects unauthenticated users to /login
- Token persisted in localStorage, restored on page load
- Axios instance automatically attaches Bearer token, redirects on 401
Backend:
- userId field added to all models (Product, Order, Customer, Expense)
- All queries scoped to authenticated user's userId
- Register/login return JWT token
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- User model with bcrypt password hashing
- Register, login, logout, and /me endpoints
- authenticate middleware applied to all API routes
- JWT_SECRET configurable via environment variable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Enable MongoDB connection in server
- Add Customer and Expense models
- Implement full CRUD for products, orders, customers, expenses
- Implement analytics dashboard with revenue, sales chart, top products/customers
- Add MongoDB service to docker-compose.deploy.yml with persistent volume
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Architecture Updates:
- Nginx serves static React files for optimal performance
- Nginx proxies API requests to Node.js backend (port 8080)
- Separation of concerns: static files vs API handling
- Professional production setup with proper caching
Features Added:
- nginx.conf with optimized configuration:
- Static file serving with long-term caching
- API reverse proxy with rate limiting
- Security headers and GZIP compression
- Health check proxying and SPA routing support
- Updated docker-compose.yml for multi-container setup
- build-deploy.sh script for automated deployment
- Updated environment configuration for container networking
Security & Performance:
- Rate limiting on API and auth endpoints
- Security headers (XSS, CSRF, clickjacking protection)
- GZIP compression for static assets
- Proper cache control headers
- Container-to-container communication
Deployment:
- Single command deployment with ./build-deploy.sh
- Nginx on port 80 (exposed as 3000) serving React app
- API server on internal port 8080 (not exposed)
- Persistent data volume mounting for business files
Features:
- Multi-stage Dockerfile for optimized production builds
- Docker Compose configuration with health checks
- Health check endpoint for container monitoring
- Production environment configuration template
- Comprehensive deployment guide for Mac Mini
- Docker ignore file for efficient build context
- Security: Non-root user, proper signal handling
- Persistence: Data directory volume mounting
- Performance: Alpine Linux base, optimized layers
- Future-ready: MongoDB service configuration (commented)
Deployment:
- Simple 'docker-compose up' deployment
- Automatic health monitoring and restart policies
- Persistent data storage with volume mounts
- Port configuration and environment customization
- Complete troubleshooting and management guide
Features:
- React + TypeScript frontend with Tailwind CSS
- Node.js + Express backend with TypeScript
- Comprehensive order tracking and management
- Product catalog with inventory tracking
- Customer data management
- Expense tracking and categorization
- Advanced Profit Analysis Dashboard with:
- Real-time profit metrics and KPI visualization
- Detailed order-level profit breakdown
- Product performance analysis
- Enhanced time range filtering (monthly, quarterly, yearly)
- Interactive expandable order analysis
- Performance categorization and color coding
- CSV import functionality for Etsy statements
- PDF parsing capabilities
- Redux state management with persistence
- Responsive design with mobile support
- Australian date formatting and currency display
