From f1d1d261e959034062d25d259e0ac233eb420c92 Mon Sep 17 00:00:00 2001 From: dlawler489 <104159223@student.swin.edu.au> Date: Sat, 20 Jun 2026 07:29:33 +1000 Subject: [PATCH] Add project wiki: overview, architecture, Etsy integration, data import, reports, deployment/CI, troubleshooting Co-Authored-By: Claude Opus 4.8 --- Architecture.md | 33 ++++++++++++++++++++++++++++++++ Data-Import.md | 22 +++++++++++++++++++++ Deployment-&-CI.md | 38 +++++++++++++++++++++++++++++++++++++ Etsy-Integration.md | 40 +++++++++++++++++++++++++++++++++++++++ Home.md | 33 ++++++++++++++++++++++++++++++++ Reports-&-Analytics.md | 43 ++++++++++++++++++++++++++++++++++++++++++ Troubleshooting.md | 27 ++++++++++++++++++++++++++ 7 files changed, 236 insertions(+) create mode 100644 Architecture.md create mode 100644 Data-Import.md create mode 100644 Deployment-&-CI.md create mode 100644 Etsy-Integration.md create mode 100644 Home.md create mode 100644 Reports-&-Analytics.md create mode 100644 Troubleshooting.md diff --git a/Architecture.md b/Architecture.md new file mode 100644 index 0000000..4e17fad --- /dev/null +++ b/Architecture.md @@ -0,0 +1,33 @@ +# Architecture + +## Stack + +- **Frontend:** React + Redux Toolkit + Tailwind, built with Vite, served by nginx +- **Backend:** Node.js / Express + Mongoose, port 8080 +- **Database:** MongoDB, per-user isolation (`userId` on every document) +- **Auth:** JWT (7-day tokens) in localStorage +- **Hosting:** Docker on the home server, behind Traefik (see [[Deployment & CI]]) + +## Repo layout + +``` +App/ + client/ React app (pages/, components/, store/ slices, utils/) + server/ Express API (routes/, models/, services/, middleware/) + Dockerfile multi-stage: client-build, server-build, production (API), nginx-frontend + docker-compose.deploy.yml the file Dockhand deploys + .forgejo/workflows/ CI +``` + +## Data model (MongoDB) + +- **Order** — `orderNumber` (= Etsy receipt id), items[{title, quantity, price, sku, variant, productId, printingCost, costOfGoods}], `total`, `refundTotal`, `tax`, status, customer, shippingAddress, tracking. Costs are **snapshotted on the order at sync time** so later catalogue edits don't rewrite history. +- **Product** — title, price, `costOfGoods`, `printingCost`, sku, category, tags, `aliases[]` (past order titles that map here), `etsyListingId`, inventory. +- **Expense** — category, description, amount, date, vendor, `reference`, taxDeductible. Etsy fees use `reference: etsy-ledger-` for idempotency. +- **Customer** — name, email, address, etsyUserId, totalOrders, totalSpent. +- **EtsyConnection** — OAuth tokens, shopId, lastSyncedAt. +- **EtsySettings** — per-user API keystring, shared secret, redirect URI. + +## Matching logic + +Order items are matched to catalogue products by, in order: exact title/alias (variant-aware, so `…Size: Large` beats the bare title) → `etsyListingId` (size-disambiguated). Matched titles are saved as aliases and listing IDs are learned, so matching gets more deterministic over time. Unmatched items surface in Settings for resolution. diff --git a/Data-Import.md b/Data-Import.md new file mode 100644 index 0000000..94b4f42 --- /dev/null +++ b/Data-Import.md @@ -0,0 +1,22 @@ +# Data Import + +Most data now comes from the [[Etsy Integration]]. The Data Import page remains for the one source the Etsy API can't provide and for legacy/manual imports. + +## Australia Post (CSV) — the one remaining manual import + +Postage bought **directly** from Australia Post (MyPost Business) is only in your AusPost account, not Etsy — so it's imported via CSV on the Data Import page. It creates Shipping & Postage expenses (vendor "Australia Post", keyed by tracking number). + +There is **no usable API** for this: MyPost Business has no public transaction-history API, and Etsy's API only knows about labels bought *through Etsy* (those early ones are in the ledger as Shipping & Postage). So the CSV stays. + +## Etsy statement CSV (legacy — superseded) + +Before the API, fees came from the Etsy statement CSV. That's now replaced by the ledger sync. **Stop importing the statement CSV** to avoid double-counting; use **Settings → Remove Old Fees** once to clear any previously imported ones. + +## PDF packing slips (legacy) + +The original cost path parsed packing-slip PDFs to match items and capture printing costs. Superseded by API order sync + catalogue matching. Still present but not needed. + +## Notes + +- Expense imports dedupe on `{reference, vendor, amount, date}`. +- Orders import via bulk upsert by order number, so re-imports update rather than duplicate. diff --git a/Deployment-&-CI.md b/Deployment-&-CI.md new file mode 100644 index 0000000..925674b --- /dev/null +++ b/Deployment-&-CI.md @@ -0,0 +1,38 @@ +# Deployment & CI + +Self-hosted end to end on the home server (Mac Mini, arm64, OrbStack). Code, CI, and the container registry all live on Forgejo at **git.plexultra.com** (migrated off GitHub/GHCR in June 2026). + +## Pipeline + +1. Commit and push to Forgejo (`origin` is the `forgejo:` SSH alias → 192.168.1.124:2222, LAN only). +2. **Forgejo Actions** (`.forgejo/workflows/docker-build.yml`) runs on `mac-mini-runner` and builds + pushes two images, natively arm64 (no QEMU): + - `git.plexultra.com/dlawler489/etsy-finance-tracker:main` (API, `production` stage) + - `git.plexultra.com/dlawler489/etsy-finance-tracker-nginx:main` (nginx, `nginx-frontend` stage) + - Both also tagged with the commit SHA (rollback targets). +3. **Redeploy in Dockhand**, which pulls `docker-compose.deploy.yml` and recreates the stack. + +The deploy host must be `docker login git.plexultra.com` (or the packages public) to pull. + +## Runner setup (gitea/act_runner in docker-compose) + +The runner runs each job in a container and has `/var/run/docker.sock` mounted. Key facts that make the workflow work: + +- **No Node in job containers** → can't use JS actions like `actions/checkout`. The workflow is node-free: manual `git init/fetch/checkout` over HTTPS with a token. +- Job runs in **`container: docker:cli`** (Alpine) → `shell: sh`, and `apk add git` for checkout. +- act_runner **auto-mounts the Docker socket** into job containers — do **not** add it in the workflow's `volumes:` (causes `Duplicate mount point`). +- **`runs-on: ubuntu-latest`** — the label only selects the runner; the `docker:cli` container override does the work. (The runner re-registered with default labels once a `config.yaml`/`CONFIG_FILE` was added, dropping the original `docker` label.) +- Runner `config.yaml` (loaded via `CONFIG_FILE=/data/config.yaml`) sets `container.valid_volumes`. + +## Registry auth + +Pushing packages needs a **Forgejo token with `write:package`**, stored as the Actions secret **`REGISTRY_TOKEN`** and used for `docker login`. The automatic `GITHUB_TOKEN` cannot push packages (`unauthorized: reqPackageAccess`); it's still used for the git checkout. + +## Traefik + +Label discovery was unreliable; routing uses the file provider: `/Users/davidlawler/docker/traefik/dynamic/etsy.yml` routes `Host(etsy.plexultra.com)` → `etsy-nginx:80` (websecure, `le` certresolver). + +## Persistence & secrets + +- MongoDB on a named volume (`etsy_mongodb`) — survives container recreation. Clearing orders/expenses in-app does not touch the volume. +- `JWT_SECRET` set in Dockhand env (don't leave the default). +- Etsy keystring/secret and OAuth tokens are stored **unencrypted** in MongoDB — fine for single-user self-host, but a DB backup contains them. diff --git a/Etsy-Integration.md b/Etsy-Integration.md new file mode 100644 index 0000000..6e30926 --- /dev/null +++ b/Etsy-Integration.md @@ -0,0 +1,40 @@ +# Etsy Integration + +The Etsy API is the primary data source. Setup and credentials live entirely in the app (Settings → Etsy Integration) and are stored in the database per user — no env vars or redeploy needed. + +## One-time setup + +1. **Create an Etsy app** at [etsy.com/developers](https://www.etsy.com/developers) (personal-use key is sufficient — it only touches your own shop). +2. **Register the callback URL** in the Etsy app settings, exactly: `https://etsy.plexultra.com/api/etsy/callback` +3. In the tracker: **Settings → Etsy Integration → API Configuration** + - Paste the **keystring** (OAuth client id) + - Paste the **shared secret** + - Confirm the **callback URL** (pre-filled) + - **Save Configuration** +4. **Connect Etsy Shop** → approve on Etsy's consent screen → you land back on Settings showing the shop name. + +### Credential quirks (learned the hard way) + +- The `x-api-key` header for API calls must be **`keystring:sharedsecret`** (both, colon-separated) — not the keystring alone. +- OAuth token requests are **form-encoded**, not JSON. +- Scopes requested: `transactions_r listings_r shops_r` (read-only). Listing *creation* would need `listings_w` and a reconnect (not implemented). + +## Syncing + +**Settings → Sync Orders Now** pulls, idempotently: + +- **Orders** — upserted by receipt id (= the packing-slip "Order #"), so they merge with anything already present. Items get costs snapshotted from the catalogue. +- **Fees** — from the payment-account ledger (chunked in 30-day windows; Etsy caps the query at 31 days). Amounts are integer minor units (÷100). Classified into Marketing & Advertising (`prolist`/offsite ads), Transaction Fees, Payment Processing Fees, Listing Fees, Taxes & GST. Credits, disbursements, and refunds are skipped. Unrecognised debits are reported back, not guessed. +- **Refunds** — summed from each receipt and stored as `refundTotal`; revenue is reported net. +- **Customers** — accumulated per buyer. + +**Settings → Sync Catalog** pulls active listings into Products — **one product per size variation** (colour collapsed). Updates title/price/tags/sku/quantity/listing-id but **preserves your costs**. + +### Resolving unmatched items + +After a sync, any item title that didn't match a product appears under a yellow banner with **Resolve Items** — map each to an existing product (saved as an alias) or create it with a cost. You can do a few at a time; the rest stay listed. + +## Idempotency / safety + +- Re-syncing never duplicates: orders upsert by number, fees by `etsy-ledger-`, and the sync auto-dedupes any historical duplicate fees. +- The **Remove Old Fees** button clears legacy CSV-imported Etsy fees so they don't double-count with ledger fees. diff --git a/Home.md b/Home.md new file mode 100644 index 0000000..dd8f8c3 --- /dev/null +++ b/Home.md @@ -0,0 +1,33 @@ +# Etsy Finance Tracker + +Self-hosted finance and profit tracker for an Etsy shop. Pulls orders, fees, refunds, and listings directly from the Etsy API, matches them against a product catalogue with per-item costs, and reports profit, margins, expenses, and an AU financial-year P&L. + +Live at **[etsy.plexultra.com](https://etsy.plexultra.com)**. + +## Pages + +- [[Architecture]] — stack, structure, data model +- [[Etsy Integration]] — connecting your shop, syncing orders/fees/listings +- [[Data Import]] — CSV/PDF imports (Australia Post + legacy) +- [[Reports & Analytics]] — dashboards, P&L, GST, ad ROI +- [[Deployment & CI]] — Forgejo Actions, registry, Dockhand, Traefik +- [[Troubleshooting]] — common issues and fixes + +## What it does + +- **Order sync** from Etsy — line items, variations/sizes, tracking, customer, with per-item costs snapshotted at sync time +- **Fee sync** from the Etsy payment ledger — transaction, processing, listing, ads (`prolist`/offsite), and GST, categorised automatically +- **Refunds** netted from revenue so profit isn't overstated +- **Catalogue sync** from active listings — one product per size, costs preserved +- **Customers** built from buyers, with repeat-rate +- **Reports** — profit/margin dashboards, monthly trends, ad ROAS, AU financial-year P&L, and an indicative GST summary + +## What you maintain + +Etsy provides the sales/catalogue side. You provide the **cost side** — set printing/material cost once per product (Products → "Missing costs" filter finds the ones needing it). Australia Post postage is imported via CSV (no usable API for MyPost Business history). + +## Quick links + +- Settings → **Etsy Integration**: connect shop, Sync Orders, Sync Catalog +- Products → **Missing costs**: fill in per-product printing/material costs +- Profit Analysis → **P&L / GST**: financial-year reporting diff --git a/Reports-&-Analytics.md b/Reports-&-Analytics.md new file mode 100644 index 0000000..6bc1303 --- /dev/null +++ b/Reports-&-Analytics.md @@ -0,0 +1,43 @@ +# Reports & Analytics + +## Analytics page + +Headline metrics for the selected period (date selector includes months, quarters, calendar years, financial years, and custom ranges): + +- **Total Revenue** — net of refunds (shows the refunded amount) +- **Net Profit** and margin — revenue − printing costs − expenses +- **Orders**, AOV, **Customers**, repeat rate +- **Printing Costs** +- **Revenue Trend** — monthly bar chart (scaled to values) +- **Expense Categories** — breakdown with amounts and % +- **Advertising Performance** — ad spend, % of revenue, and **ROAS** (revenue per $1 of ad spend). Ads are typically the largest expense, so this is the key lever to watch. + +## Profit Analysis page + +Tabs: + +- **Overview** — profit metrics and cards +- **Trends** — revenue vs profit by month (bar chart) + a per-month table (orders, revenue, printing, expenses, profit, margin) +- **Products** — per-product profitability ranking +- **Orders** — per-order profit with item-level breakdown +- **P&L / GST** — see below + +### P&L / GST tab + +A proper profit-and-loss statement for the selected period: + +``` +Revenue (net of refunds) + − Cost of goods sold (printing + materials) + = Gross profit + − Operating expenses (by category) + = Net profit +``` + +Plus an **indicative GST summary** (GST on sales vs GST paid on Etsy fees, and net position). Marked indicative — confirm against your BAS; Etsy generally collects/remits GST on AU sales on your behalf. + +**Financial-year reporting:** the date selector offers AU financial years (Jul–Jun), e.g. "FY 2025–26", so the P&L and all reports can be run per financial year. + +## Accuracy depends on costs + +COGS, margins, and net profit are only right once each product has its **printing/material cost** set. Use **Products → "Missing costs"** to find and fill products imported at $0. (Note: the P&L includes materials in COGS, so its net profit can read slightly lower than the Overview card, which historically counted printing only.) diff --git a/Troubleshooting.md b/Troubleshooting.md new file mode 100644 index 0000000..09412ba --- /dev/null +++ b/Troubleshooting.md @@ -0,0 +1,27 @@ +# Troubleshooting + +## Etsy connection / sync + +- **"Etsy connection failed"** on connect — check the callback URL registered in your Etsy app exactly matches `https://etsy.plexultra.com/api/etsy/callback`, and that both the keystring and shared secret are saved in Settings. Error detail is shown in the toast. +- **403 "Shared secret is required" / "incorrect shared secret"** — the `x-api-key` must be `keystring:sharedsecret`; make sure the shared secret is saved (Settings shows a masked value when set). +- **Ledger "time window … no more than 31 days"** — handled (the sync chunks 30 days at a time); if you see it, the build is stale. +- **Unmatched items after sync** — use Settings → **Resolve Items** to map each to a product (saved as alias) or create it; then re-sync. + +## Numbers look wrong + +- **Profit overstated / costs missing** — products imported from the catalogue start at $0 cost. Products → **Missing costs** filter → set printing/material costs. +- **Two pages disagree** — usually a stale cached bundle; hard-refresh (Cmd+Shift+R). Revenue is net of refunds everywhere; the revenue card shows "net of refunds". +- **Duplicate fees** — the sync auto-dedupes ledger fees; for legacy CSV fees use **Remove Old Fees** once. + +## Deployment / CI + +- **Workflow "waiting" forever** — no runner matches `runs-on`, or the runner is offline. Check Settings → Actions → Runners is Idle/online and has the `ubuntu-latest` label; check `docker compose logs forgejo-runner`. +- **`docker: command not found`** — job ran in a container without docker; ensure the workflow uses `container: docker:cli`. +- **`Duplicate mount point: /var/run/docker.sock`** — remove any explicit socket mount from the workflow; act_runner mounts it automatically. +- **`unauthorized: reqPackageAccess` on push** — the `REGISTRY_TOKEN` secret is missing or lacks `write:package`. Recreate the Forgejo token and secret. +- **Deploy can't pull image** — `docker login git.plexultra.com` on the host, or make the package public in Forgejo. +- **Stale UI after deploy** — hard-refresh; nginx serves `index.html` no-cache and hashed assets immutable, so a refresh picks up the new bundle. + +## Data resets + +- **Clear All Orders** (Data Import) bulk-deletes orders only; products, expenses, and customers are untouched. Re-sync from Etsy to rebuild. Product cost mappings (aliases, listing IDs, costs) live on products and survive an order clear.