# D1 Migration TODO — files.txt to Cloudflare D1 daily_images

## Phase 1: Infrastructure (COMPLETE)
- [x] Create tlc-daily-db D1 database (production + staging)
- [x] Apply daily_images schema (migration 0007)
- [x] Add DAILY_DB binding to wrangler.toml
- [x] Add DailyImageRow type to types.ts
- [x] Add D1 functions in db.ts (append, upsert, query, batch-query, counts, delete)
- [x] Create dailyImages.ts handler with 6 API endpoints
- [x] Wire routes in index.ts
- [x] Deploy to production

## Phase 1b: Optimised Storage Format (COMPLETE)
- [x] Design optimised format: filename_prefix + timestamps only (68% storage savings)
- [x] Migration 0008: Drop+recreate daily_images with filename_prefix column
- [x] Update types.ts: add filename_prefix to DailyImageRow
- [x] Update db.ts: parseFilename() extracts prefix/timestamp, all CRUD updated
- [x] Update dailyImages.ts: batch endpoint accepts filename_prefix
- [x] Update PHP daily-images-helper.php: consumer reconstructs full filenames
- [x] Fix delete endpoint: CAST json_each key to INTEGER for json_remove path
- [x] Deploy and verify: append, query, delete all working
- [x] Test camera backfill: 272 rows, exact match vs files.txt (0 differences)

## Phase 2: Writers (COMPLETE)
- [x] Add appendToDailyImages to upload.ts (DSLR handler, uses ctx.waitUntil)
- [x] Add appendToDailyImages to microProcess.ts (micro handler, uses await)
- [x] Create PHP helper: php/api/worker-config.php
- [x] Create PHP helper: php/api/daily-images-helper.php
- [x] Update download.and.unzip.php (main DSLR writer)
- [x] Update download.and.unzip.fullsize.php
- [x] Update download.and.unzip.security.php
- [x] Update delete.photo.php (delete-file call)
- [x] Update microcontrollers/process-8B3-HF2-4TZ.php
- [x] Update microcontrollers-v2/microControllerUploader.php
- [x] Make cam_id optional in append endpoint
- [x] Set API_SECRET in Worker
- [x] Create .writers-deployed feature flag
- [x] Fix Worker URL to workers.dev (DNS not resolving upload.livetimelapse.com from server)

## Phase 2b: Backfill (COMPLETE)
- [x] Create backfill script (scripts/backfill-daily-images.php)
- [x] Rewrite backfill to use HTTP API (SQL files too large for D1 execute)
- [x] Rewrite backfill for optimised format (group by prefix, send timestamps only)
- [x] Test single-camera backfill (272 rows, 0 errors, exact match)
- [x] Full backfill: 629,871+ rows, 1,018 cameras, dates 2016-01-01 to 2026-03-10
- [x] Validate: 100% match across 20 random days, 99.96% across last 30 days (±1 timing only)
- [x] Last-7-days sync: 2,494 rows, 0 errors
- [x] Dual-write active: both files.txt and D1 updated live by camera uploads

## Phase 3: Consumers (SINGLE CAMERA TRIAL)
- [x] Update tlc.include.b2native.php with _getFileListForDay() wrapper
- [x] Trial camera: tlc_fultonhogan_luggagepoint_jn1152
- [ ] Update other tlc.include.*.php templates
- [ ] Update camera wall consumers
- [ ] Update timelapse player files
- [ ] Create .d1-enabled feature flag (switches consumers to D1)

## Phase 4: Monitoring & Docs (NOT STARTED)
- [ ] Reconciliation cron (compare D1 vs files.txt)
- [ ] Documentation