# Microcontroller Image Upload System Documentation

## Overview

This system processes images uploaded by microcontroller-based cameras. Microcontrollers capture photos and upload them to an AWS S3 passthrough bucket. This script picks up those images, stores the full-size original in Backblaze B2, creates a resized web version with text overlay, uploads that to B2 as well, updates local files for the website, and deletes the original from S3.

## Architecture

```
Microcontroller Camera
        |
        | (uploads image)
        v
AWS S3 Bucket: "tlc-passthrough-new"
  (folder per camera serial, e.g. X2X-9DP-MEE/)
        |
        | (polled every minute by cron)
        v
microControllerUploader.php (6 parallel instances)
        |
        |-- 1. Downloads image from S3
        |-- 2. Looks up camera in MySQL database
        |-- 3. Uploads full-size original to Backblaze B2
        |-- 4. Resizes to 980x653, adds text overlay bar
        |-- 5. Uploads web version to Backblaze B2
        |-- 6. Updates local files (stills, timestamps, status)
        |-- 7. Deletes original from S3 passthrough bucket
        v
Backblaze B2 (10 accounts, ~100 buckets each)
  + Local filesystem for website stills/status
```

## File Location

- **Script:** `/home/subdomains/livetimelapse/public_html/php/microcontrollers/microControllerUploader.php`
- **Logs:** `/home/subdomains/livetimelapse/public_html/php/microcontrollers/logs/server-upload.log`
- **Vendor dependencies:** `vendor/` (AWS SDK via Composer)
- **Backups of older per-camera scripts:** `backup/`

## Scheduling (Cron)

Defined in `/etc/crontab`, running as `root`, every minute:

```
* * * * * root php .../microControllerUploader.php -i=instance1 > /dev/null 2>&1
* * * * * root php .../microControllerUploader.php -i=instance2 > /dev/null 2>&1
* * * * * root php .../microControllerUploader.php -i=instance3 > /dev/null 2>&1
* * * * * root php .../microControllerUploader.php -i=instance4 > /dev/null 2>&1
* * * * * root php .../microControllerUploader.php -i=instance5 > /dev/null 2>&1
* * * * * root php .../microControllerUploader.php -i=instance6 > /dev/null 2>&1
```

Each instance has a built-in duplicate check using `ps` -- if the same instance is already running, the new invocation exits immediately.

## Instance-to-Camera Mapping

Each instance handles a specific set of cameras identified by their 11-character serial numbers. This mapping is hardcoded in the `$threadList` array in the script:

| Instance | Camera Serials |
|----------|---------------|
| instance1 | X2X-9DP-MEE, NXY-FGZ-N58, LFY-MK6-GSK, LNP-65R-R6W |
| instance2 | YE9-ATY-4NM, MB5-ZWK-F7L, YM6-A92-87, W62-AMV-S94 |
| instance3 | 2HJ-ZYX-G5L, 6R4-YN5-RJT, ASL-GWS-NSM, YWC-ULT-UW4 |
| instance4 | DUF-CRR-DWP, WZM-FX3-WEU, 236-GHT-V7X, 46T-9YF-VGC |
| instance5 | 7XG-4WP-ZL9, 8B3-HF2-4TZ, KDC-C89-WBP, HU8-B9K-P63 |
| instance6 | TPR-KV3-SNB, MPP-GT8-9M4 |

**IMPORTANT:** The serial numbers in the `$threadList` array must exactly match the `cameraSerialNum` column in the database (all 11 characters). A mismatch will prevent the camera from being processed because the script validates `strlen($s3FilePath['dirname']) == 11`.

## Database

- **Host:** `127.0.0.1` (localhost)
- **User:** `uploadprocessor`
- **Database:** `timelapse`

### Table: cameraSettings

Key fields used by this script:

| Column | Type | Description |
|--------|------|-------------|
| `camId` | int (PK) | Auto-increment camera ID |
| `cameraSerialNum` | varchar(255) | 11-char serial (e.g., `YM6-A92-87Z`). Must match `$threadList` |
| `cameraName` | varchar(255) | System name used as B2 bucket name and directory name (e.g., `tlc-formplus-pinkenba2-jn1168`). Convention uses hyphens. |
| `cameraLongName` | varchar(255) | Display name shown on image overlay (e.g., `Form Plus - Pinkenba`) |
| `cloudStorageAccount` | varchar(255) | B2 account ID linking to `backBlazeKeys.accId` |
| `cloudStorageBucketCreated` | tinyint(1) | `0` = script will attempt to create the B2 bucket; `1` = bucket exists, skip creation |
| `cloudStorage` | varchar(10) | Storage type for web images (default: `b2`) |
| `cloudStorageFullsize` | varchar(10) | Storage type for full-size images (default: `b2`) |
| `archived` | tinyint(1) | `0` = active, `1` = archived. Only active cameras are processed. |
| `startDate` | datetime | Camera start date |
| `stopDate` | datetime | Camera stop date |
| `createDate` | datetime | Record creation date |
| `timezone` | varchar(100) | Camera timezone (default: `Australia/Brisbane`) |
| `dayStart` | time | Daily start time for captures |
| `dayStop` | time | Daily stop time for captures |
| `powerType` | varchar(20) | Power source (default: `solar`) |

Current stats: ~1109 total cameras, ~998 active, ~23 active microcontroller cameras (those with serial numbers).

### Table: backBlazeKeys

| Column | Type | Description |
|--------|------|-------------|
| `b2KeyId` | int (PK) | Auto-increment key ID |
| `accId` | varchar(255) | Backblaze account ID (e.g., `4235aee3081e`) |
| `keyName` | varchar(255) | Key name (e.g., `PHP-Scripting-rw` or `PHP-Scripting-RO`) |
| `appKeyId` | varchar(255) | Application key ID for API auth |
| `appKey` | varchar(255) | Application key secret for API auth |
| `region` | varchar(255) | B2 region (e.g., `us-west-004`, `us-west-002`, `eu-central-003`) |
| `newBucketsHere` | tinyint(1) | Flag for whether new buckets should be created in this account |
| `archiveAcct` | tinyint(1) | Flag for archive accounts |
| `emailAddress` | varchar(255) | Account email |
| `masterKey` | varchar(255) | Account master key |

There are **10 unique B2 accounts** with 21 total keys (each account typically has a read-write and a read-only key).

### SQL Query Used by the Script

The script looks up camera details by joining both tables:

```sql
SELECT bb.accId, bb.appKey, bb.appKeyId, bb.region,
       cs.camId, cs.cloudStorageBucketCreated, cs.cloudStorage,
       cs.cloudStorageFullsize, cs.cameraName, cs.cameraLongName
FROM backBlazeKeys AS bb
LEFT JOIN cameraSettings AS cs ON bb.accId = cs.cloudStorageAccount
WHERE cs.cameraSerialNum = '{serial}'
  AND cs.archived = 0
  AND bb.keyName = 'php-scripting-rw'
ORDER BY cs.createDate DESC
LIMIT 1
```

## Image Processing Pipeline

### 1. S3 Passthrough Bucket

- **Bucket:** `tlc-passthrough-new` (AWS S3, `us-east-1`)
- Cameras upload images as: `{SERIAL}/{filename}.jpg`
- Filename format: `CAM_{SERIAL_NO_DASHES}_{YYYYMMDD}_{HH}_{MM}_{SS}.jpg`
  - Example: `CAM_YM6A9287Z_20260216_12_45_48.jpg`

### 2. Full-Size Upload to B2

The original image is uploaded to the B2 bucket (named after `cameraName`) at:
```
fullsize/{YYYY}/{MM}/{DD}/{YYYYMMDD}_{HHMMSS}.jpg
```
Example: `fullsize/2026/02/16/20260216_124548.jpg`

### 3. Image Resize and Overlay

Using ImageMagick (PHP `Imagick`):
- Resized to **980x653** using Lanczos filter
- A semi-transparent dark blue bar (`#00008B60`) is drawn at the bottom (y: 633-653)
- Text overlay using `Gotham-Book.otf` font at 13px:
  - **Left side:** `{cameraLongName} - {date/time}` (e.g., `Form Plus - Pinkenba - 16/02/26 12:45PM`)
  - **Right side:** `Time Lapse Pty Ltd`
  - Black shadow text offset by -1px for readability
- Compressed at **85% JPEG quality**

### 4. Web Version Upload to B2

The resized image is uploaded to the same B2 bucket at:
```
{cameraName_underscored}/{YYYY}/{MM}/{DD}/980x655_{HHMMSS}.jpg
```
Note: The directory name uses underscores (legacy format via `str_replace("-", "_", cameraName)`).

Example: `tlc_formplus_pinkenba2_jn1168/2026/02/16/980x655_124548.jpg`

### 5. Local File Updates

Several local files/directories are updated after successful upload:

| Path | Purpose |
|------|---------|
| `/public_html/db/{name}/{YYYY}/{MM}/{DD}/files.txt` | Appended list of processed filenames |
| `/public_html/db/{name}/{YYYY}/{MM}/{DD}/thumbs/` | Thumbnail directory (created, not populated by this script) |
| `/public_html/webcams/{name}/980x655.jpg` | Latest still image for website display |
| `/public_html/webcams/{name}/timestamp.txt` | Unix timestamp of last upload |
| `/public_html/webcams/{name}/status.txt` | Set to `OK` |
| `/public_html/monitoring/logs/{name}/DSLRStatus.txt` | Set to `CONNECTED` |
| `/public_html/monitoring/logs/{name}/DSLRStats.txt` | Set to `Camera Model-Microcontroller\|Auto Power Off-0\|Available Shots-100` |
| `/public_html/monitoring/logs/{name}/VPNIP.txt` | Camera serial number |

Where `{name}` is the underscored version of `cameraName` (e.g., `tlc_formplus_pinkenba2_jn1168`).

All directories are created automatically with `0775` permissions if they don't exist.

### 6. Cleanup

After successful processing, the original image is deleted from the S3 passthrough bucket using `deleteObjects`.

### 7. Cloudflare Integration

When a new B2 bucket is created, the script also calls:
```
https://livetimelapse.com.au/php/github/cloudflare/insertBucket2Db.php?bucketName={cameraName}
```
This registers the new bucket with Cloudflare for CDN/DNS purposes.

## Logging

All activity is logged to `logs/server-upload.log` with the format:
```
[YYYY-MM-DD HH:MM:SS][instanceName][cameraName]: message
```

Examples:
```
[2026-02-16 13:06:09][instance2][tlc-formplus-pinkenba2-jn1168]: Downloading image for conversion: CAM_YM6A9287Z_20260216_12_45_48.jpg
[2026-02-16 13:06:11][instance2][tlc-formplus-pinkenba2-jn1168]: Image downloaded from S3.
[2026-02-16 13:06:11][instance2][tlc-formplus-pinkenba2-jn1168]: Uploading original to B2: fullsize/2026/02/16/20260216_124548.jpg
```

The `cameraName` field in logs shows the currently active camera context. When no camera is active, it shows `system` or is empty.

Log rotation is handled externally -- gzipped daily logs are retained (e.g., `server-upload.log.1.gz`).

## Adding a New Microcontroller Camera

### Step-by-step:

1. **Create the database record** in `cameraSettings`:
   ```sql
   INSERT INTO cameraSettings (cameraSerialNum, cameraName, cameraLongName,
     cloudStorageAccount, cloudStorageBucketCreated, archived, createDate)
   VALUES ('XXX-XXX-XXX', 'tlc-client-location-jnXXXX', 'Client - Location',
     '{b2_account_id}', 0, 0, NOW());
   ```

2. **Add the serial to `$threadList`** in `microControllerUploader.php` under an instance with capacity (max ~4 per instance; instance6 currently has only 2).

3. **Create the B2 bucket manually** in the Backblaze web console for the target account. Set it to public-read. Then set `cloudStorageBucketCreated = 1` in the database. (Auto-creation via the script is unreliable.)

4. **Ensure B2 bucket capacity**: Each Backblaze account has a **100 bucket limit**. Check the target account's bucket count before assigning. If full, either delete an old unused bucket or use a different account.

5. **B2 API key permissions**: The `PHP-Scripting-rw` key for the target account must have write access to the new bucket. If keys are scoped to specific buckets, the key may need to be regenerated with "all buckets" access.

6. **Bucket naming**: Backblaze bucket names are **globally unique** across ALL Backblaze accounts worldwide. A name used in any account cannot be reused elsewhere.

7. **Verify**: Monitor `logs/server-upload.log` to confirm images are being processed. Look for "Upload completed" and "Image deleted successfully" messages.

## Changing a Camera's Upload Location (cameraName)

To change which bucket/directory a camera uploads to:

1. **Update the database:**
   ```sql
   UPDATE cameraSettings
   SET cameraName = 'tlc-newclient-location-jnXXXX',
       cameraLongName = 'New Client - Location'
   WHERE camId = {camId};
   ```

2. **Create the B2 bucket manually** in the Backblaze web console under the correct account. Ensure:
   - The bucket is in the same account as `cloudStorageAccount`
   - The bucket is set to public-read
   - The bucket name doesn't already exist in any other B2 account

3. **Set the bucket flag:**
   ```sql
   UPDATE cameraSettings SET cloudStorageBucketCreated = 1 WHERE camId = {camId};
   ```

4. **If changing B2 accounts**, also update:
   ```sql
   UPDATE cameraSettings SET cloudStorageAccount = '{new_account_id}' WHERE camId = {camId};
   ```

5. **Check the serial** in `$threadList` matches the database `cameraSerialNum` exactly (all 11 characters).

6. **Note on existing data:** Changing the camera name does NOT migrate existing images. Old images remain in the old bucket/directories. Only new images will go to the new location.

## Backblaze B2 Accounts

There are 10 B2 accounts:

| # | Account ID | Region | Key ID (rw) |
|---|-----------|--------|-------------|
| 1 | `2dc3d983cefa` | us-west-004 | `0042dc3d983cefa000000000e` |
| 2 | `4235aee3081e` | us-west-004 | `0044235aee3081e0000000015` |
| 3 | `442bb2cfb82b` | us-west-004 | `004442bb2cfb82b000000001a` |
| 4 | `644574ec9689` | us-west-004 | `004644574ec96890000000010` |
| 5 | `7a730c869c19` | eu-central-003 | `0037a730c869c19000000000c` |
| 6 | `b6356c83c215` | us-west-004 | `004b6356c83c215000000000f` |
| 7 | `bdca4042903e` | us-west-004 | `004bdca4042903e0000000010` |
| 8 | `cdae1b104ded` | us-west-002 | `002cdae1b104ded0000000010` |
| 9 | `e9b9a43ed880` | us-west-004 | `004e9b9a43ed8800000000017` |
| 10 | `f1a3094fac55` | us-west-004 | `004f1a3094fac550000000021` |

Each account has a `PHP-Scripting-rw` (read-write) and `PHP-Scripting-RO` (read-only) key. The uploader script uses the `rw` key (matched by `bb.keyName='php-scripting-rw'` -- note: MySQL comparison is case-insensitive by default).

## Error Handling and Known Issues

1. **Fatal errors halt all cameras in an instance:** If one camera fails (e.g., B2 upload error), the script calls `exit(1)`, stopping processing for ALL cameras in that instance. The cron job will restart it next minute, but images for other cameras in that instance are delayed.

2. **Bucket creation via S3 API is unreliable:** The `createBucket` call via the S3-compatible API frequently returns `400 Bad Request`, likely due to API key permissions not including `createBucket` capability. **Recommendation:** Always create buckets manually in the Backblaze console.

3. **100-bucket limit per B2 account:** Each Backblaze account allows a maximum of 100 buckets. Monitor bucket counts when adding new cameras. This limit cannot be increased.

4. **Globally unique bucket names:** Bucket names must be unique across ALL Backblaze accounts worldwide. A name used in any account cannot be reused elsewhere until deleted.

5. **Serial number mismatch:** The `$threadList` must use the exact 11-character serial. The script validates `strlen == 11` and uses `in_array` for exact matching. A serial that is too short or too long will be silently ignored.

6. **`$cam` variable leaks between cameras:** The `$cam` variable (used for logging context) retains the previous camera's name when processing moves to a new serial. This can produce confusing log entries where one camera's name appears in another camera's log messages.

7. **`$deleteObject` array accumulates:** The `$deleteObject` array (line 276) uses `[]` append without being reset between iterations, which could cause issues if multiple images are processed in one run.

8. **No retry logic:** If an upload fails, the script exits immediately. There is no retry mechanism -- the image will be retried on the next cron invocation (next minute).

## Dependencies

- **PHP** with Imagick extension
- **AWS SDK for PHP** (via Composer, in `vendor/`)
- **MySQL** database (`timelapse`) on localhost
- **ImageMagick** with Gotham-Book.otf font at `/usr/share/fonts/Gotham-Book.otf`
- **Cron** for scheduling (configured in `/etc/crontab`)

## AWS S3 Connection Details

- **Endpoint:** `https://s3.us-east-1.amazonaws.com`
- **Bucket:** `tlc-passthrough-new`
- **Region:** `us-east-1`