# Files System Refactoring

> **Context:** OpenAI Assistants are deprecated and will be removed ~June 2026. This refactoring focuses on clean architecture for the future: Responses API (OpenAI), Files API (Anthropic), and Google. Assistants-related code will not be extended or fixed.

---

## Current Issues

### 1. Provider Files Never Deleted

When a user uploads a PDF to the chatbot:

1. **Local entry created** in `wp_mwai_files` with our internal `refId` and `expires` (5 minutes)
2. **File uploaded to provider** (OpenAI Files API or Anthropic Files API)
3. **Provider returns a `file_id`** (e.g., `file-abc123`)
4. **Provider `file_id` is NOT stored** - only exists in memory during the request
5. **Local file cleanup runs** - deletes local file after expiration
6. **Provider file remains orphaned** - never deleted, accumulates indefinitely

This affects:
- **OpenAI**: Files uploaded via `/v1/files` with purpose `user_data` for Responses API
- **Anthropic**: Files uploaded via Files API for PDF handling

### 2. Confusing `refId` Dual Purpose

Currently `refId` serves two purposes:
- For uploaded files: our internal hash (e.g., `4637406206133ea2326ee9c97933e9ce`)
- For assistant-out files: gets **replaced** with OpenAI's `file_id` via `update_refId()`

This is confusing and makes cleanup logic unreliable.

**Note:** The `refId` overwrite only happens in OpenAI Assistants code (`chatbot.php:711`, `premium/openai.php:243`). Since **OpenAI Assistants are deprecated and will be removed ~June 2026**, we should not invest in fixing this. Instead, focus on clean architecture for the future (Responses API, Anthropic, Google).

### 3. Redundant/Unclear Fields

| Field | Issue |
|-------|-------|
| `type` | Only `'image'` or `'file'` - redundant, can derive from MIME type |
| `purpose` | Random values (`files`, `vision`, `assistant-in`, `assistant-out`, `generated`, `transcription`) - mostly informational, only used technically in 2 places |
| `status` | Always `'uploaded'` - unclear if other values exist |

### 4. `purpose` Field Analysis

**Current values:**
- `files` - File API upload (default, not helpful)
- `vision` - Image sent directly in query
- `assistant-in` - User uploaded to assistant
- `assistant-out` - Generated by assistant
- `generated` - AI generated content
- `transcription` - Audio file for transcription

**Actual technical usage (only 2 places):**
1. `discussions.php:554` - Checks `purpose === 'vision'` for image markdown preview
2. `openai.php:197` - Searches for `purpose = 'assistant-out'` to find sandbox files

Both could be handled differently (MIME type check, metadata check).

---

## Implementation Phases

The refactoring is split into two phases to minimize risk and allow incremental testing.

---

## Phase 1: Provider Cleanup (Priority - Fixes Real Leak)

**Goal:** Store provider file IDs and delete them during cleanup. No schema changes.

### 1.1 Store Provider File ID After Upload

**OpenAI** - `classes/engines/openai.php` in `prepare_query()`:
```php
$uploadedFile = $uploadEngine->upload_file( $filename, $data, 'user_data' );
$fileId = $uploadedFile['id'] ?? null;

if ( $fileId ) {
  // Store provider file_id in metadata
  $localFileId = $this->core->files->get_id_from_refId( $refId );
  if ( $localFileId ) {
    $this->core->files->add_metadata( $localFileId, 'file_id', $fileId );
    $this->core->files->add_metadata( $localFileId, 'provider', 'openai' );
  }
  // ... rest of existing code
}
```

**Anthropic** - `classes/engines/anthropic.php` in `prepare_query()`:
```php
$uploadedFile = $this->upload_file( $filename, $data, $mimeType );
$fileId = $uploadedFile['id'] ?? null;

if ( $fileId ) {
  // Store provider file_id in metadata
  $localFileId = $this->core->files->get_id_from_refId( $refId );
  if ( $localFileId ) {
    $this->core->files->add_metadata( $localFileId, 'file_id', $fileId );
    $this->core->files->add_metadata( $localFileId, 'provider', 'anthropic' );
  }
  // ... rest of existing code
}
```

### 1.2 Add `delete_file()` to Anthropic Engine

Currently only OpenAI has this in `classes/engines/chatml.php`. Add to `classes/engines/anthropic.php`:

```php
public function delete_file( $fileId ) {
  $url = 'https://api.anthropic.com/v1/files/' . $fileId;

  $response = wp_remote_request( $url, [
    'method' => 'DELETE',
    'headers' => [
      'x-api-key' => $this->apiKey,
      'anthropic-version' => '2023-06-01',
      'anthropic-beta' => 'files-api-2025-04-14',
    ],
    'timeout' => 30
  ] );

  if ( is_wp_error( $response ) ) {
    throw new Exception( 'File deletion failed: ' . $response->get_error_message() );
  }

  return json_decode( wp_remote_retrieve_body( $response ), true );
}
```

### 1.3 Delete Provider Files During Cleanup

In `classes/modules/files.php`, update `delete_expired_files()`:

```php
public function delete_expired_files( $fileRefs ) {
  // Give a chance to other processes to delete files (existing filter)
  $fileRefs = apply_filters( 'mwai_files_delete', $fileRefs );

  // NEW: Delete from providers before local cleanup
  foreach ( $fileRefs as $refId ) {
    $this->delete_provider_file( $refId );
  }

  // ... existing local deletion code
}

/**
 * Delete file from provider (OpenAI, Anthropic, etc.) if applicable.
 */
private function delete_provider_file( $refId ) {
  $metadata = $this->get_metadata( $refId );
  $providerFileId = $metadata['file_id'] ?? null;
  $provider = $metadata['provider'] ?? null;

  if ( !$providerFileId || !$provider ) {
    return; // No provider file to delete
  }

  $fileInfo = $this->get_info( $refId );
  $envId = $fileInfo['envId'] ?? null;

  if ( !$envId ) {
    Meow_MWAI_Logging::warn( "Cannot delete provider file {$providerFileId}: no envId" );
    return;
  }

  try {
    if ( $provider === 'openai' ) {
      $engine = Meow_MWAI_Engines_Factory::get_openai( $this->core, $envId );
      $engine->delete_file( $providerFileId );
      Meow_MWAI_Logging::log( "Deleted OpenAI file: {$providerFileId}" );
    }
    elseif ( $provider === 'anthropic' ) {
      $engine = Meow_MWAI_Engines_Factory::get( $this->core, $envId );
      if ( method_exists( $engine, 'delete_file' ) ) {
        $engine->delete_file( $providerFileId );
        Meow_MWAI_Logging::log( "Deleted Anthropic file: {$providerFileId}" );
      }
    }
    // Google: Add when they have a Files API that needs cleanup
  }
  catch ( Exception $e ) {
    // Log but don't fail - local cleanup should still proceed
    Meow_MWAI_Logging::error( "Failed to delete {$provider} file {$providerFileId}: " . $e->getMessage() );
  }
}
```

### 1.4 Ensure `envId` is Set on File Records

The cleanup needs `envId` to know which API credentials to use. Verify that `envId` is being set when files are created in chatbot uploads. Check `classes/modules/chatbot.php` and `premium/forms.php`.

### 1.5 Scope: Assistants Are Deprecated

**OpenAI Assistants are deprecated and will be removed ~June 2026.**

The `refId` overwrite issue (`update_refId()`) only exists in Assistants code:
- `chatbot.php:711` - Assistant file search
- `premium/openai.php:243` - Assistant output images

**We will NOT fix or extend Assistants-related file handling.** Phase 1 targets:
- **Responses API uploads** (`openai.php:prepare_query()`)
- **Anthropic Files API uploads** (`anthropic.php:prepare_query()`)

Both leave the database `refId` unchanged, so Phase 1 will work correctly for these flows.

### 1.6 Error Handling

- Provider deletion failures should be logged but not fatal (local cleanup must proceed)
- Consider rate-limiting log messages to avoid noisy cron tasks on repeated failures
- Handle cases where provider API is unreachable (timeout, network errors)

### Phase 1 Files to Modify

| File | Changes |
|------|---------|
| `classes/engines/openai.php` | Store `file_id` + `provider` metadata after upload |
| `classes/engines/anthropic.php` | Store metadata + add `delete_file()` method |
| `classes/modules/files.php` | Add `delete_provider_file()`, call it in `delete_expired_files()` |

---

## Phase 2: Schema Cleanup (Future - Clean Architecture)

**Goal:** Simplify schema, fix `refId` consistency, replace `type`/`purpose` with `mimeType`/`source`.

**Prerequisites:** Phase 1 complete and stable.

### 2.1 Proposed Final Schema

| Field | Type | Description |
|-------|------|-------------|
| `id` | BIGINT | Primary key |
| `refId` | VARCHAR(64) | Our internal unique ID only (never provider IDs) |
| `userId` | VARCHAR(64) | User or session ID |
| `envId` | VARCHAR(128) | AI environment used |
| `mimeType` | VARCHAR(128) | Full MIME type (e.g., `application/pdf`, `image/png`) |
| `status` | VARCHAR(32) | Keep for now (`uploaded`, potentially others) |
| `source` | VARCHAR(32) | `'input'` (user uploaded) or `'output'` (AI generated) |
| `created` | DATETIME | Creation timestamp |
| `updated` | DATETIME | Last update timestamp |
| `expires` | DATETIME | For cleanup task |
| `path` | TEXT | Local filesystem path |
| `url` | TEXT | Public URL |

### 2.2 Migration Steps

1. **Add new columns** via `check_db()` ALTER:
   - `mimeType VARCHAR(128) NULL`
   - `source VARCHAR(32) NULL`

2. **Backfill data:**
   - `mimeType`: Derive from file extension or existing `type` field
   - `source`: Map from `purpose`:
     - `assistant-out`, `generated` → `'output'`
     - Everything else → `'input'`

3. **Update code to use new fields** (all call sites):
   - `type` checks → `mimeType` checks
   - `purpose` checks → `source` or metadata checks
   - Frontend filters (Assistants.js) → use new fields

4. **`refId` consistency:**
   - The `update_refId()` issue only exists in Assistants code
   - **Assistants will be removed ~June 2026** - no need to fix this
   - After Assistants removal, `refId` will naturally be consistent (always our internal ID)

5. **Deprecation period:**
   - Keep `type` and `purpose` columns but stop writing to them
   - Log warnings if old fields are accessed

6. **Remove old columns** (after migration period, e.g., 6 months):
   - Drop `type` column
   - Drop `purpose` column

### 2.3 Code Changes Required

| File | Changes |
|------|---------|
| `classes/modules/files.php` | Schema migration, new helpers (`is_image()`, `is_audio()`, `is_pdf()`) |
| `classes/modules/discussions.php:554` | Replace `purpose === 'vision'` with MIME type check |
| `classes/modules/files.php:520-580` | Update `search()`/`list()` SQL for new fields |

**Note:** Assistants-related files (`premium/openai.php`, `app/js/screens/assistants/`) will be removed ~June 2026, so no changes needed there.

### 2.4 Helper Methods

```php
// In files.php - use mimeType field instead of type

public function is_image_by_mime( $mimeType ) {
  return $mimeType && strpos( $mimeType, 'image/' ) === 0;
}

public function is_audio_by_mime( $mimeType ) {
  return $mimeType && strpos( $mimeType, 'audio/' ) === 0;
}

public function is_pdf_by_mime( $mimeType ) {
  return $mimeType === 'application/pdf';
}

// Convenience methods that fetch info first
public function is_image( $refId ) {
  $info = $this->get_info( $refId );
  return $this->is_image_by_mime( $info['mimeType'] ?? '' );
}
```

---

## Metadata Keys Reference

After Phase 1, the following metadata keys will be used in `wp_mwai_filemeta`:

| Key | Description | Set By |
|-----|-------------|--------|
| `file_id` | Provider's file ID (e.g., `file-abc123`) | Engines after upload |
| `provider` | Provider name: `openai`, `anthropic`, `google` | Engines after upload |

**Legacy (will be removed with Assistants ~June 2026):**
| Key | Description |
|-----|-------------|
| `assistant_id` | OpenAI Assistant ID |
| `assistant_threadId` | OpenAI Thread ID |
| `assistant_sandboxPath` | Sandbox file path |

---

## Benefits

### Phase 1
- **Provider files cleaned up** - No more orphaned files on OpenAI/Anthropic
- **Minimal risk** - No schema changes, backwards compatible
- **Easy to test** - Can verify files are deleted via provider dashboards

### Phase 2
- **Clear separation** - `refId` is always ours, provider IDs in metadata
- **Simpler schema** - `mimeType` and `source` are self-explanatory
- **Extensible** - Easy to add new providers via metadata
- **Debuggable** - Can see exactly which provider file corresponds to which local file
