# Multi-render maintenance map

## Add-on account connection (1 November 2026)

Customer access now uses dedicated, expiring add-on tokens, separate from Manager / Render App tokens. The website Profile exposes Blender Add-on Access; the add-on preferences section is Account Connection. With no token, show entry/storage/Connect controls; with a token, show Check Connection and Forget Token. Website Link is the only account-navigation button in Blender.

Add-on ownership: `operators/account_connection.py`, `helpers/addon_credentials.py`, `helpers/submission.py` and `preferences.py`. Website ownership: `app/Services/AddonTokens.php`, `app/Http/Middleware/AddonTokenBoundary.php`, `app/Http/Controllers/AddonTokenController.php` and `resources/views/dashboard/addon-tokens.blade.php`. Website `docs/ADDON_TOKEN_RELEASE.md` documents rollout, origin configuration and tests. Production and test builds need tokens from their respective website origins. Secure OS keyring integration is optional; unavailable storage means session-only credentials, never a plaintext fallback.

Guide narrative and this maintenance source live in `RenderFarm-MainSoftware/packages/renderfarm_core/docs/`. Run `generate_guides.py`, `generate_guides.py --check`, and `verify_guides.py` there to publish website guide files and verify the add-on’s web-help link. The add-on contains no bundled docs; the guide button uses its configured website URL. Local guide generation does not deploy the website or certify a live farm run.


Current post-implementation findings and fixes: [implementation review](MULTI_RENDER_REVIEW.md). Finalization scans beyond incomplete early IDs, publication rejects empty artifacts, Job scenes use `scene_name`, and MP4/MOV codec selection follows the declared profiles. The manager now requires a securely supplied `RUNPOD_API_KEY` environment value for cloud-provider use; rotate any previously embedded active key. Rebuild manager artifacts before release. Historical reports are indexed under [history](docs/multi-render/history/README.md).

Stage-19 guide ownership: `RenderFarm-MainSoftware/packages/renderfarm_core/docs/` holds authored HTML and the contract-driven generator. Run `generate_guides.py`, then `generate_guides.py --check` and `verify_guides.py`; rerun Website `UserGuidesTest` and relevant workflow checks. Generated guides live only in Website `resources/guides/`. The add-on packager excludes `docs/`; its Multi-Render Guide button opens the configured website guide. See [stage-19 evidence and all-requirement checklist](docs/multi-render/history/MULTI_RENDER_STAGE19_VERIFICATION.md). Screenshots are labeled isolated fixtures, never fleet certification. Rebuild distribution artifacts before release; the preexisting versioned ZIP is not updated by editing source.

Current audit and concise ownership/recovery table: [stage 17](docs/multi-render/history/MULTI_RENDER_STAGE17_VERIFICATION.md); controlled deployment procedure: [release runbook](MULTI_RENDER_RELEASE_RUNBOOK.md). Core `control.py` fences malformed heartbeat replies; `runtime.py`/`AssignmentProtocol` preserve durable upload pauses and idempotent owner Retry Upload. Website `renders:cutover-status` is read-only; `RENDERFARM_ACCEPT_MULTI_SUBMISSIONS` closes new multi admission without blocking committed receipt replay. Current unreleased wheel digest: `8791fc1d5d573da33efcf2fb5d4f8ca861ea8fcf6324a78f286665a3644baa79`; the stage-18 diagnostic headless executable verifies it. GUI/Tcl and actual Docker execution remain unverified. Older notes below retain their dated context.

Current checkout only; **v2 submission/shared preparation, server billing/control, combined manager distribution, canonical worker runtime and per-ID finalization are implemented locally; host and release verification remains**. Dispatch remains disabled. Consult MULTI_RENDER_IMPLEMENTATION_PROGRESS.md, [stage-14 verification](docs/multi-render/history/MULTI_RENDER_STAGE14_VERIFICATION.md), docs/multi-render/history/MULTI_RENDER_ACCEPTANCE_AUDIT.md and [the system bug audit](docs/multi-render/history/MULTI_RENDER_SYSTEM_AUDIT.md) before treating a module as production-ready. The early audit records legacy ownership defects subsequently fixed by registration/gate checks: the three ownership probes now pass. They do not certify every legacy upload/billing path.

| Concern | Canonical place to edit | Diagnostic / safe recovery |
|---|---|---|
| Wire schemas, typed properties, limits, statuses/actions, compatibility | RenderFarm-MainSoftware/packages/renderfarm_core/src/renderfarm_core/resources/contract.json and compatibility.json; contracts.py | Core unit/parity tests. Keep dispatch_enabled=false and unlisted combinations unverified until real host fixtures pass. A limit override is trusted configuration, never request authority. |
| Blender capture/apply/preflight/reference/animation rules | Same package: blender_state.py and resources/blender_bindings.json | Blender state/rules smoke scripts with --python-exit-code 1. Apply reopens the base only in an isolated process. Do not use apply in the authoring process; restore_working is its explicit authoring helper. Copied files/bake flags are not simulation proof. |
| Discovery/copy/inventory/extension/archive | Same package: preparation.py and package_files.py; Blender discovery adapter helpers/project_preparation.py | PreparedProject.validate checks immutable owned bytes. Extensions commit inventory/states together; sealed archive never refreshes live sources. Missing/corrupt session requires explicit preparation. Never overwrite frozen reserve bytes. |
| Add-on state/draft ownership and lifecycle | RenderFarm-BlenderAddon/helpers/multi_render_session.py, multi_render_lifecycle.py and multi_render_background.py; operators/multi_render.py | Cancel task, wait for borrowers/processes, then owner-scoped cleanup. Refresh Initial State resets every ID only after valid replacement succeeds. No persistent recovery or manual temp-folder descriptor edits. Source authoring file stays untouched. |
| Add-on presentation / submission | multi_render_properties.py, multi_render_ui.py, ui_panel.py; operators/submission.py and helpers/submission.py | Preview and sealed submission are implemented. Confirmed receipts are separate from local saving. Uncertain requests reuse the exact payload/key/archive; retry before sending changed data. Rejected/cancelled attempts retain owned preparation; close/load clears session state. |
| Submission, intent and receipts | Core submission.py; Website RenderSubmissionService.php, SingleSubmissionAdapter.php and RenderSubmissionController.php | One owner/key/request hash and one upload owner. Explicit-null ordinary intent is not dispatchable. Do not regenerate a key after an uncertain create. Inspect the immutable request, receipt and owner-only /submissions/<id> page. |
| Upload authority | Website SubmissionUploadService.php and Storage/SubmissionStorage.php | Durable multipart hashes/ETags, operation token and expiry. Network outside account/SQL locks; only current authority advances offsets. Renew expired unsubmitted uploads through start with the SAME key/bytes. Never manually append a final chunk or delete a confirmed source. |
| Manager preparation | Core manager_preparation.py and resources/prepare_submission.py; thin Manager_Functions/Submission_Preparation.py; Website SubmissionPreparationService.php | Off by default: RENDERFARM_PREPARE_V2=1 requires installed pinned core, matching Blender and configured storage. Reuse immutable request, renew lease, token-specific object, replay completion. Stop releases preparation; expired attempts may be reclaimed. Failure logs have bounded Blender diagnostics. Never mark old downloaded=true to bypass the gate. |
| Shared retention | Website renders:cleanup-submissions and config/renderfarm.php | Dry run first; --apply deletes exact owned keys only after all siblings/leases/retention permit. Failed/unsubmitted uploads retain local data and remote cleanup authority. Future scoped retry/control services must honor deleting/removed and independent holds. |
| Local edit warnings / Preview from Base | Core blender_authoring.py; add-on helpers/multi_render_edit_tracker.py and multi_render_preview.py | Do not add frame/depsgraph handlers or serialize authored signatures into states. Capture boundary locks are authoritative. Preview uses canonical apply in a separate exact-version process; cancel/join before owned cleanup. Tokens survive image renames; actual content users require packing before cleanup. If packing fails, retain files and save the image permanently before manual cleanup. Runtime/disk failures retain the session. Current tested adapter is Cycles CPU, not fleet parity. |
| PHP contract consumer | RenderFarm-Website/app/Services/RenderFarm/Contracts.php and ContractJson.php | RenderFarmContractTest and Python/PHP parity. Policy lives in canonical resources; do not hand-maintain another camera/state list. |
| Database structures / contract mapping | Website database/migrations/2026_09_27_*.php and app/Models/Render*.php | RenderConfiguration.fromContract/toContract retain object/array/null/false and exact FPS/hash. Locked model checks do not secure generic SQL/API writes; future services must fence those paths. |
| Legacy audit / backfill | Website app/Services/RenderFarm/RenderMigrationAudit.php, LegacySubmissionBackfill.php; renders:migration-audit and renders:backfill-submissions | Use a restored isolated database, compare history fingerprints, run dry-run, retain per-ID checkpoints/conflicts, then repeat to verify. Never infer archive presence from old downloaded flags. Backfill does not reprice, initialize ledger holds or upgrade worker protocol. |
| Schema verification / rollback | Website tools/test_stage7_mysql.py and tests/Integration/README.md; Render migration down() guards | Real isolated MySQL 8.4.8 schema/backfill checks now pass. Production-version restored-data/backup rehearsal remains required. Use a tested backup/forward-fix for partial DDL; never flatten multi-ID history, drop old columns or consolidate live duplicate assignments as an automatic recovery. |
| Server assignment/control/output authority | Website app/Services/RenderFarm/{ProtocolLocks,AssignmentProtocol,RenderControlService,FrameAcceptanceService,ExecutionAdmission}.php | Use scoped token APIs; never edit v2 execution rows. Status/snapshot reconciles expiry; release follows process stop. Closed tokens/history and accepted manifests are immutable. Dispatch stays gated. See stage-9 verification. |
| Per-ID output processing and downloads | Website FinalizationService.php, RenderFinalizationController.php, RenderedController.php, SendRenderCompletionNotices.php; manager Finalization_Executor.py | Reclaim expired leases or inspect `finalization_error`/`final_outputs`; never rerender or rewrite accepted frame rows to recover ZIP/video. Completion notice claims require uncertain-delivery reconciliation. Stage-14 verification lists external checks. |
| Attempt cleanup and node registration | OutputAttemptCleanup.php, NodeAuthority.php; renderfarm:cleanup-output-attempts --after/--limit and render-nodes:cleanup-stale | Dry-run first, follow next_after. Only old closed-token unreferenced objects are removable. V2 stale-node cleanup expires leases and retains node ownership/history; heartbeat cannot transfer accounts. |

To propagate canonical changes, use packages/renderfarm_core/tools/vendor_addon.py and sync_contracts.py. Checks without --write verify identical bytes; --write regenerates consumers after canonical edits. build_wheel.py builds the common artifact; desktop/Docker host installation/parity is still stage 13. Never patch the add-on vendor or PHP resources independently. Current unreleased core is 0.1.0, schema 1/protocol 2; compatibility manifest certifies no combination.

Useful tests: core unittest discovery; add-on session/preparation unittest suites; actual Blender state/preparation/UI/preview/audit scripts; isolated Website MultiRenderSchemaTest and RenderFarmContractTest. The Website schema suite forces in-memory SQLite and owns its temporary snapshot copy. SQLite does not prove MySQL locking/races. Add-on upload-ETA wording failures were fixed in the 2026-09-28 audit; PHP-8.5 PDO config deprecations remain. The three opt-in tests/Audit/RenderNodeOwnershipAuditTest.php probes now pass in stage 9; registration/read/delete ownership is enforced. Never run Composer setup or production migrate as a diagnostic shortcut.


## Stage-10 billing maintenance

Use [the billing verification/recovery guide](docs/multi-render/history/MULTI_RENDER_STAGE10_VERIFICATION.md). Canonical website modules are `Services/Credits/LedgerBalance`, `AccountLedger`, and `Services/RenderFarm/AccountRenderBillingService`. All financial writers must enter the account lock before touching financial/frame rows; never call providers under that lock. Preserve accepted identities and cumulative credit rows. The migration records shared charges, billing provenance and notice episodes without replacing the ledger.

Safe recovery commands are `renderfarm:reconcile-accounts`, `renderfarm:reconcile-refund`, and `renderfarm:credit-notices` (inspection by default). Never clear uncertain refund reservations without checking provider evidence. Reconcile funds, then explicitly resume; do not clear every pause reason. Notice delivery is gated separately from dispatch. Manager adapters are in `Manager_Website_Api.py` / `App_2_Check.py`; the add-on estimate is display-only. Stage 11 connects the new billing endpoint to actual multi distribution.


## Stage-11 scheduling and cloud recovery

- `Manager_Functions/Distribution_Planner.py`: pure counted allocation, stable quotas, eligibility pools and reservation subtraction. `Distribution_Service.py`: billing/snapshot/commit sequence and durable request replay; `App_2_Check.py` invokes it. `RENDERFARM_DISTRIBUTE_V2` stays off until controlled rollout. Do not send v2 work through `App_2_Distribute` or generic table writes.
- Website `AssignmentProtocol.php`: fresh eligible workers, shared scheduling owner/takeover and bounded waiting observations. Plan journal: `~/.renderfarm-manager/<SHA256(manager ID)>/<submission>.json`. Preserve uncertain keys; retry the original body. Do not remove the persistent lock file while a manager is running.
- `CloudSchedulingService.php` and `App_3_Compute_Provider_Manager.py`: verified image pin, serialized provisioning and guarded retirement. `cloud_image=null` fails closed. Startup permission never substitutes for a worker heartbeat. Generic reads omit v2 assignments; cloud demand provides authoritative retention names.
- A `retiring` profile stays fenced until the original owner account's adapter verifies the actual container/pod is absent and reports the same token. Recover that adapter/account or prepare a reviewed ownership-recovery procedure with provider evidence. Never clear this state merely because time passed. A delayed provider stop must not overlap new work.
- Tests and exact commands: docs/multi-render/history/MULTI_RENDER_STAGE11_VERIFICATION.md. Core version/schema/protocol are unchanged; real host/provider artifacts are still unverified.


Stage-12 runtime maintenance: `renderfarm_core/runtime.py` owns claim/render/handoff/release; `control.py` owns per-token deadlines; `workspace.py` owns package/attempt paths and pins; `spool.py`/`uploader.py` own ready/accepted bytes; `blender_runner.py` plus static resources own processes. Restart the same owned workspace to reconcile. Never relabel quarantined output under a new token or delete assignment history to reset a retry budget. Website `FrameAttemptService.php` and `render_frame_attempts` own retry counts; existing `FrameAcceptanceService` owns acceptance/payout. See [stage-12 verification](docs/multi-render/history/MULTI_RENDER_STAGE12_VERIFICATION.md) for commands, exact wheel digest and stage-13 host requirements.