The unique expense index can silently fail to build over pre-existing
duplicate data, so re-syncs were re-adding ledger fees every run.
- syncLedgerEntries now explicitly checks existing references (and
de-dupes within the batch) instead of trusting the unique index
- dedupeLedgerExpenses keeps one row per etsy-ledger-<entry_id> and
deletes the rest; runs automatically at the start of each sync so
existing duplicates self-heal. Distinct entries sharing a date/amount
are untouched (each has its own reference).
- Sync response/toast report deduped and already-imported counts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Order model gains refundTotal; sync sums receipt.refunds (Money
objects) onto each order and marks fully-refunded orders 'refunded'
- orderNetRevenue() helper (total - refundTotal); profit metrics,
monthly trends, and per-order analysis now use net revenue so
refunded sales no longer overstate profit
- ProfitMetrics adds totalRefunds; Analytics revenue card shows net
revenue with the refunded amount, and monthly revenue is net
Note: product-level profitability stays gross since receipt refunds are
order-level amounts with no line-item breakdown.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- syncLedgerEntries pulls payment-account ledger entries (chunked by 90d
over required min_created/max_created), classifies debits into fee
categories, and inserts them as idempotent expenses (reference
etsy-ledger-<entry_id>). Amounts are integer minor units -> /100.
- Conservative classifier: only genuine costs become expenses; sale
credits, disbursements, and refunds are skipped. Unclassified debits
are reported back so rules can be refined.
- Folded into POST /api/etsy/sync alongside orders; response includes
ledger result and a legacyEtsyExpenses count (pre-ledger CSV fees).
- DELETE /api/etsy/legacy-fees removes CSV-imported Etsy fee expenses to
avoid double counting; Settings surfaces the count with a one-click
remove, plus a list of skipped/unknown ledger charge types.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Previously a different-size variant was boosted to 0.95 confidence
('same product, different size'), so a Small order would be confidently
suggested the Large product. Now suggestions require the same size; a
known different size is suppressed entirely (you create the right one
instead). Also handle 'Size: X for ...' phrasing, medium sizes, and
American 'Color:' spelling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Previously submitting created a £0 product for every unmatched row the
user didn't touch, silently hiding it from the unmatched list. Now rows
with no chosen product and no printing cost are skipped and stay
unmatched, so items can be resolved a few at a time across syncs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reuses the missing-products modal on the Settings page: each unmatched
listing title can be aliased to an existing product or created with a
printing cost, then orders re-sync automatically to fill in costs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etsy returns 403 'Shared secret is required in x-api-key header' when the
keystring is used on resource endpoints: the keystring is only the OAuth
client id. Store the shared secret alongside it (Settings form + env
fallback) and send it as x-api-key on users/me and receipt requests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Callback failures now redirect with the underlying error message, shown
in the Settings toast, instead of a generic failure
- Token endpoint requests use application/x-www-form-urlencoded per
RFC 6749 instead of JSON
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The /settings route existed but had no nav entry, leaving the Etsy
integration and data management UI unreachable by clicking.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New EtsySettings model holds the per-user API keystring and callback URL,
managed via GET/PUT /api/etsy/config; env vars remain as optional fallback
- Settings UI gains an API Configuration form (masked saved key, callback URL
prefilled with this origin's /api/etsy/callback); Connect is enabled once
configuration is saved
- OAuth and sync resolve the key per user; post-callback redirect derives
from the stored callback URL origin instead of CLIENT_URL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- OAuth 2.0 authorization code flow with PKCE; /api/etsy/connect returns the
consent URL, /api/etsy/callback exchanges the code (validated via one-time
state, no JWT) and stores tokens per user with automatic refresh
- /api/etsy/sync pulls all shop receipts and upserts orders by receipt id:
items with SKU/variations, totals, shipping address, tracking, and status,
with catalog costs snapshotted at sync time
- Product matching by exact title/alias first, then etsyListingId
(size-disambiguated); listing ids are learned onto products on first match
- Packing-slip items with cost data are preserved when synced items can't all
be matched
- Settings page: Connect Etsy Shop, Sync Orders Now, Disconnect, and a list
of unmatched item titles after each sync
- Requires ETSY_API_KEY and ETSY_REDIRECT_URI env vars (see .env.example)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Products store packing-slip title aliases; matcher checks exact title/alias
first so previously-matched items skip fuzzy matching entirely
- Packing-slip imports snapshot printingCost/costOfGoods/productId onto order
items; profit analysis reads stored costs so catalog edits don't rewrite history
- Etsy statement fees allocate to orders via Order # references in Title/Info
instead of date proximity; shop-level fees (listings, ads) no longer leak
into order fees
- Remove broken transaction-fee exclusion guard (order totals are gross, so
all expenses count once)
- Remove debug buttons, date-fix banner, test files, and console.log noise;
Clear All Orders now uses a bulk DELETE /orders endpoint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add value labels on revenue chart bars with smart formatting (1.2k for >)
- Implement proper bar scaling with 75% max height for better proportions
- Add enhanced hover tooltips with revenue and profit breakdown
- Fix top profitable products to show meaningful names instead of generic titles
- Improve product identification for orders with generic 'Product from Etsy' names
- Add comprehensive debugging for product sales tracking
- Enhanced visual design with better gradients and transitions
- Show units sold and total revenue in product details
- Improve month labels with cleaner formatting (no rotation)
- Better empty state handling and responsive layout
- Fix revenue trend chart to display month-year format for multi-year data
- Implement proper date sorting (newest first) for monthly revenue data
- Enhance chart with gradients, hover tooltips, and better visual design
- Fix top profitable products to use actual sales data instead of catalog data
- Calculate profits based on real order performance within date range
- Add comprehensive debugging and empty state handling
- Change default date range to 'All Time' for better initial experience
- Improve month-year parsing and data validation
- Add dynamic date range generation based on actual order data
- Implement custom date range picker with start/end date inputs
- Unify date filtering logic between Analytics and ProfitAnalysis pages
- Support multiple date formats (YYYY, YYYY-MM, YYYY-QX, custom ranges)
- Remove hardcoded years for future-proof date selection
- Enhance ProfitAnalysisService with custom date range support
🎯 ISSUE RESOLVED: Profit calculations were different between pages due to inconsistent date filtering
📊 ROOT CAUSE:
- Profit Analysis Dashboard: "2026" = calendar year 2026 (Jan-Dec 2026)
- Analytics Page: "This Year" = rolling 365 days (May 2025-May 2026)
🔧 FIXES APPLIED:
- Unified date filtering logic using consistent calendar year approach
- Added "2026 Full Year" option to Analytics page for exact comparison
- Changed default to "2026" to match Profit Analysis dashboard
- Updated "This Year" to mean current calendar year (2026), not rolling 365 days
✅ NOW BOTH PAGES SHOW IDENTICAL DATA when comparing same date ranges!
The profit calculations will now match exactly when comparing equivalent time periods.
🚀 PERFORMANCE IMPROVEMENTS:
- Add bulk expenses API endpoint (/expenses/bulk) for fast batch processing
- Replace slow sequential processing (3 seconds per expense) with instant bulk operations
- Use MongoDB insertMany with ordered:false for optimal bulk inserts
- Handle duplicates, validation errors, and partial failures gracefully
🧹 CODE CLEANUP:
- Remove excessive date parsing test logs that were spamming console
- Clean up unused imports and test code
- Improve error handling with detailed bulk operation results
⚡ SPEED IMPROVEMENT:
- Before: 60+ seconds for 20 expenses (3 seconds each)
- After: <2 seconds for any number of expenses (bulk operation)
This eliminates the painfully slow background processing while maintaining duplicate prevention and error handling.
- Add proper array validation for existingExpenses API response
- Handle different API response structures (data.expenses vs data)
- Add try-catch blocks around individual expense processing loops
- Improve error messages to show specific error details
- Prevent TypeError when API response format is unexpected
Resolves 'gt.some is not a function' error in background processing.
- Process expenses immediately when CSV files are uploaded, not when user clicks analyze
- Use much more conservative rate limiting (3 seconds between expenses) to prevent 429 errors
- Remove bulk batch processing that was overwhelming the rate limiter
- Add background processing status indicator with spinner animation
- Clean up unused state variables and functions
- Improve user experience by eliminating long wait times during analysis
This prevents HTTP 429 rate limiting errors by spreading expense creation over time naturally as files are uploaded, rather than trying to process 60+ expenses rapidly in batches.
- Reduce batch size from 3 to 2 expenses per batch for more conservative processing
- Increase delay between batches from 1.5s to 2s to prevent HTTP 429 errors
- Add isProcessingExpenses state to prevent multiple simultaneous imports
- Disable import buttons during expense processing for better UX
- Ensure processing state is always cleared in finally block
Resolves HTTP 429 rate limiting errors during bulk expense creation from CSV imports.
Frontend improvements:
- Add batch processing for expense creation (3 expenses per batch)
- Implement 1.5 second delays between batches to avoid overwhelming server
- Better progress logging and user feedback during batch processing
- Handle rate limit errors gracefully with proper error categorization
Backend improvements:
- Add specific rate limiter for expense creation endpoint (50 per minute)
- More informative error messages for rate limit violations
- Separate rate limiting for expense creation vs general API usage
This prevents the HTTP 429 'Too Many Requests' errors when importing large CSV files with many individual expense records (listing fees, ads, GST entries, etc.).
- Remove unused orderCosts parameter from createExpensesFromCsvData function
- Update function call to match new signature
- Upgrade GitHub Actions to use Node.js 24 instead of deprecated Node.js 20
- Add FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 environment variable
This should resolve the build failure and deprecation warnings in CI/CD.
Major improvements to expense categorization and profit calculations:
NEW: Granular Etsy Fee Parsing
- Parse individual Etsy fees with proper categorization:
* Listing Fees (should be included in expenses)
* Marketing & Advertising (Etsy Ads - should be included)
* Transaction Fees (tied to orders - excluded to avoid double-counting)
* Processing Fees (tied to orders - excluded to avoid double-counting)
* Taxes & GST (should be included)
* Shipping & Postage (should be included)
FIXED: Profit Calculation Logic
- Only exclude sale transaction fees that have order references
- Include all other Etsy business expenses (ads, listing fees, GST)
- More accurate profit margins that account for all business costs
ENHANCED: CSV Import
- Creates specific expense categories instead of lumping as 'Transaction Fees'
- Better duplicate detection based on description, amount, date, and category
- Improved user feedback showing specific fee types created
This fixes the issue where listing fees and advertising costs were incorrectly excluded from profit calculations, resulting in unrealistically high profit margins.
Multi-layer duplicate prevention system:
- Enhanced frontend duplicate detection with tracking number, amount, and date comparison
- Added MongoDB compound index to prevent database-level duplicates
- Improved backend error handling for duplicate key violations
- Added cleanup endpoint to remove existing duplicates
- Enhanced user feedback for import operations
Frontend changes:
- Stricter duplicate detection comparing tracking number, vendor, amount, and date
- Better error handling and user feedback for duplicate scenarios
- Added 'Clean Duplicates' button to remove existing duplicates
Backend changes:
- Database compound index on reference, vendor, userId, amount, date
- Enhanced error responses with duplicate detection flags
- New POST /expenses/cleanup-duplicates endpoint
- Improved duplicate key error handling
This should eliminate the double Australia Post expense entries.
- Updated ProfitAnalysisService to accept and integrate expense data
- Modified calculateProfitMetrics to include expenses in profit calculations
- Added totalExpenses field to ProfitMetrics interface
- Updated ProfitAnalysis component to pass filtered expenses to calculations
- Enhanced UI to show breakdown of expenses vs printing costs
- Fixed inconsistency between Analytics and ProfitAnalysis profit calculations
- Now shipping expenses and Etsy fees are properly deducted from profits
- Excluded transaction fees to avoid double-counting
✅ FIXES MAJOR WORKFLOW ISSUE:
- Data Import CSV processing now automatically creates expenses
- Etsy fees → 'Transaction Fees' category expenses
- Australia Post shipping → 'Shipping & Postage' expenses
- Prevents duplicate expense creation with reference checking
- Added user notices in both Data Import and Expenses tabs
🔄 INTEGRATION FEATURES:
- Automatic expense creation from CSV order costs
- Duplicate prevention by order number/tracking number
- Proper categorization and tax-deductible flagging
- Clear user notifications and workflow guidance
📊 USER EXPERIENCE:
- Expenses tab now shows integration notice
- Data Import shows automatic expense creation info
- Seamless workflow between sales data and expense tracking
- Add duplicate prevention with isProcessing state
- Implement loading spinner and disabled states
- Enhance modal closure logic in DataImport.tsx
- Add proper state reset on modal close
- Prevent multiple button clicks during processing
- Improve user feedback with console logging
Enhanced product matching algorithm to properly handle:
- Large vs Small size variants for shelf decor products
- Improved size extraction from product titles
- Higher confidence thresholds for shelf decor products to prevent wrong matches
- Better handling of 3D-Printed product variations
- Enhanced logging for debugging matching issues
This fixes the issue where packing slips would create duplicate products
instead of matching to existing Large/Small variants of the same product.
Features Added:
- Enhanced CSV import service with smart parsing for product data
- Two CSV templates: simple and complete with example data
- Advanced product import UI with template downloads
- Comprehensive CSV import documentation and guide
- Support for all product fields including costs, dimensions, materials
- Error handling and validation for bulk imports
- Integration with profit analysis system for accurate cost tracking
UI Improvements:
- Template download buttons (Simple Template, Full Template)
- Enhanced import workflow with progress feedback
- Better error messaging and validation
- Updated Products page with CSV functionality
Documentation:
- Complete CSV_IMPORT_GUIDE.md with examples and best practices
- Updated DEPLOYMENT_STATUS.md reflecting production readiness
- Enhanced README.md with current feature status
- CHANGELOG.md with comprehensive development history
Production Ready:
- All functionality tested and integrated with existing profit analysis
- Bulk import capability for efficient product catalog setup
- Supports accurate cost tracking for comprehensive profit margins
The default :root styles (dark background, white text, centered #root)
were overriding Tailwind and making all pages appear blank/dark.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove localStorage from all 4 Redux slices (products, orders, expenses, customers)
- Layout fetches all data from API on mount; adds logout button with active nav highlighting
- Wire API calls in Products, Orders, Expenses pages for all CRUD operations
- DataImport uses POST /orders/bulk for CSV upserts and API for PDF slip orders
- MissingProductsModal creates products via API
- Relax Order model: optional customerId, embedded customer, fees, printingCost on items, default paymentStatus=paid
- Relax Expense model: free-string category, add taxDeductible/vendor/reference fields
- Add printingCost to Product model
- Add POST /orders/bulk endpoint for upsert-by-orderNumber
- Raise rate limit to 1000 req/15min for bulk imports
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frontend:
- Login and Register pages wired up to API
- PrivateRoute redirects unauthenticated users to /login
- Token persisted in localStorage, restored on page load
- Axios instance automatically attaches Bearer token, redirects on 401
Backend:
- userId field added to all models (Product, Order, Customer, Expense)
- All queries scoped to authenticated user's userId
- Register/login return JWT token
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Features:
- React + TypeScript frontend with Tailwind CSS
- Node.js + Express backend with TypeScript
- Comprehensive order tracking and management
- Product catalog with inventory tracking
- Customer data management
- Expense tracking and categorization
- Advanced Profit Analysis Dashboard with:
- Real-time profit metrics and KPI visualization
- Detailed order-level profit breakdown
- Product performance analysis
- Enhanced time range filtering (monthly, quarterly, yearly)
- Interactive expandable order analysis
- Performance categorization and color coding
- CSV import functionality for Etsy statements
- PDF parsing capabilities
- Redux state management with persistence
- Responsive design with mobile support
- Australian date formatting and currency display
