Document Scope
This document defines the standing backend architecture standard for Python work inside the modular workspace. It governs how Python backend code should be organized, refactored, extended, validated, and prepared for long-term use across scripts, workers, APIs, dashboards, command-line tools, database infrastructure, integrations, and future AI-agent workflows.
This document is written for human developers, ChatGPT sessions, Codex sessions, AI-assisted development tools, automation workers, future AI agents, and future maintainers. It should be read before proposing Python package moves, backend refactors, service extraction, configuration changes, database access changes, API work, command-line tools, internal dashboards, worker changes, scheduler-related backend changes, or Codex implementation tasks.
This document does not replace the platform architecture standards, technology stack reference, runtime validation standard, patch/Codex workflow standard, Git checkpoint standard, Git commit-message standard, database-layer standard, API-layer standard, or task-specific handoff. It defines the Python backend structure standard. Task-specific documents still control the current work state and immediate implementation scope.
1. Purpose
The purpose of this document is to define the permanent standard for Python backend structure.
The backend must not remain a collection of large scripts, one-off helpers, ad hoc database calls, scattered configuration parsing, and duplicated business logic. Python code should move toward a package-based, testable, service-oriented structure where reusable logic lives in importable modules and thin entry points call those modules.
The standard transformation is:
loose scripts
become
thin entry-point scripts
that call
importable service modules
that use
standard configuration, model, database, integration, and operations layers
This does not mean broad rewrites are allowed. It means every future backend change should move in the same architectural direction.
The goal is backend code that is easier to run, easier to test, easier to inspect, easier to document, easier for Codex to modify safely, easier for ChatGPT to reason about, easier for future AI agents to operate, and easier to move from the current operating environment to future deployment environments.
2. Layer and Tool Definitions
The following table defines the current selected implementation for each backend layer. The rest of this document uses generic layer terms unless the specific implementation matters.
| Generic layer term | Current selected implementation | Role in backend architecture |
|---|---|---|
| Python project manager | uv |
Dependencies, virtual environments, lockfiles, Python versions, project metadata, and script execution |
| linting and formatting layer | Ruff | Consistent formatting, import organization, lint checks, and code-quality enforcement |
| testing layer | pytest | Unit tests, integration tests, regression tests, service tests, API tests, and safety tests |
| model layer | Pydantic | Request, response, service, configuration, and internal data models |
| configuration layer | Pydantic Settings | Centralized validated settings and environment-specific configuration |
| database access layer | SQLAlchemy Core | Standardized structured database access where practical |
| schema migration layer | Alembic where compatible | Version-controlled database schema changes |
| API layer | FastAPI | Structured API endpoints that expose service-layer behavior |
| API contract | OpenAPI | Machine-readable API contract for humans, frontends, tests, and AI agents |
| Python HTTP client layer | HTTPX | New backend HTTP integrations where appropriate |
| terminal output layer | Rich | CLI tables, progress, warnings, summaries, and diagnostic output |
| terminal application layer | Textual where justified | Future interactive terminal applications |
| internal dashboard layer | Streamlit | Internal browser-based dashboards and data tools |
| scheduler layer | current operating-system scheduler; future scheduler undecided | Scheduled work, recurring jobs, worker starts, and future cross-platform scheduling |
| integration wrapper layer | internal wrappers | Telegram, email, object storage, external APIs, infrastructure APIs, AI providers, and payment providers |
| remote repository CI/CD pipeline service | GitLab CI/CD currently | Remote validation, build, package, and deployment automation |
| pipeline configuration | .gitlab-ci.yml currently |
Repository-specific pipeline jobs and stages |
Specific tool names should be used in setup guides, implementation guides, and decision records. Durable backend standards should usually use the layer terms.
3. Core Backend Rule
Reusable backend logic belongs in importable modules.
Business logic should not be embedded directly in:
large scripts,
API route handlers,
frontend code,
Streamlit dashboard files,
Textual application files,
Rich output wrappers,
one-off maintenance scripts,
scheduler command definitions,
Telegram command handlers,
static website JavaScript.
Those entry points may call backend services. They should not own the rules.
The preferred structure is:
entry point
loads configuration
parses command or request
calls service-layer function
receives structured result
formats output or response
The risky structure is:
entry point
parses configuration manually
opens database connection directly
implements business workflow
calls providers directly
mutates persistent state
formats output
handles errors in its own style
Legacy code may contain the risky pattern. New work should not copy it.
4. Preferred Backend Organization Pattern
The backend should move toward a project structure similar to this, adapted to the actual repository after inspection:
<workspace-root>/app
├── pyproject.toml
├── uv.lock
├── .python-version
├── README.md
├── docs
├── src
│ └── platform_app
│ ├── __init__.py
│ ├── config
│ ├── models
│ ├── db
│ ├── services
│ ├── integrations
│ ├── operations
│ ├── pipelines
│ ├── reports
│ ├── api
│ └── utils
├── scripts
│ ├── maintenance
│ ├── diagnostics
│ └── migrations
├── apps
│ ├── streamlit
│ └── textual
├── tests
└── state
This is the preferred organizational pattern, not an immediate move command.
4.1 Repository Class Controls Backend Restructuring Scope
Backend restructuring rules apply most strongly to Class A deployable application repositories and Class B shared library repositories. They also apply to Class D utility repositories and Class E infrastructure repositories when those repositories contain Python code that can read, write, deploy, schedule, configure, upload, delete, or mutate important state.
Context/reference repositories should not receive backend package structure merely because they contain scripts unless the operator approves executable tooling in that repository. Experimental repositories must remain clearly isolated and must not become production backend structure by accident.
The exact package name must be selected deliberately. It should be generic enough to survive company-name, product-name, repository-name, CI/CD provider, or service-name changes. It should not confuse the backend package with one customer-facing subscription service.
A future backend package should make the system easier to reason about by separating configuration, models, database access, services, integrations, operations, pipelines, reports, APIs, and utilities.
5. Python Project Manager Standard
The Python project manager is the foundation for backend execution and dependency management.
Once adopted in the repository, it should manage:
project metadata,
runtime dependencies,
development dependencies,
virtual environments,
lockfile,
Python version expectations,
script execution.
New Python backend work should avoid global package installation instructions unless they are explicitly machine setup prerequisites. Project dependencies should be declared through project metadata and installed through the project manager.
During transition, existing local interpreter commands may remain valid if they are the known working commands for the repository. Do not rewrite commands merely to use the future runner before the repository has formally adopted it.
5.1 Project Manager and Supply Chain Boundary
The Python project manager is not only a convenience tool. It is also part of the repository's supply-chain control surface because it records dependency declarations, lockfiles, Python version expectations, development dependency groups, and execution commands.
A restructuring task must not add or upgrade packages merely because a refactor would be easier with a new dependency. New dependencies, project-manager plugins, package indexes, generated scaffolds, and helper tools must first pass the Supply Chain Security Standard. If the dependency is approved, update the relevant Supply Chain Record and Integration Record where required.
Selected backend tools are not blanket approval for every related plugin or extension. The package, version, source, and integration path still matter.
6. Importable Modules Over Loose Scripts
Scripts should become thin wrappers around importable backend logic.
A script may:
parse command-line arguments,
load settings,
call one or more service functions,
print a result,
write a summary,
return an exit code.
A script should not:
own a major business workflow,
open unrelated database connections directly,
parse unrelated environment variables manually,
call providers directly when wrappers exist,
contain presentation logic mixed with persistent-state rules,
be the only place where a reusable behavior exists.
When modifying a script, identify whether the changed logic is truly script-specific or whether it belongs in a service module.
7. Service Module Standard
Service modules own backend workflows.
Examples of service modules may include:
subscription service management,
content intake,
content normalization,
content processing,
deduplication,
classification,
publication,
report generation,
operations incident handling,
remediation job handling,
email sending,
object storage artifact creation,
dashboard data preparation,
database health checks,
API response preparation.
Service functions should take clear inputs and return structured outputs. They should not assume they are being called only from one interface.
A service function may be called by:
scheduled workers,
API routes,
CLI scripts,
internal dashboards,
terminal applications,
pytest tests,
future AI agents.
This reuse is the main reason service modules exist.
8. Configuration Standard
Configuration must move toward a centralized validated configuration layer.
The configuration layer should cover:
environment identity,
local versus server behavior,
database settings,
Telegram settings,
email settings,
object storage settings,
scheduler settings,
feature flags,
logging settings,
runtime mode,
provider credentials,
API keys,
development versus production behavior.
Modules should not independently parse configuration in unrelated styles.
Production must not be the accidental default for scripts that can write data. Any write-capable tool must make environment identity visible.
The configuration layer should be designed for the current operating environment and future deployment environments. Durable backend code should not hardcode Windows-only assumptions unless the task is explicitly machine-specific.
9. Model and Validation Standard
The backend should use explicit models where data crosses boundaries.
Use models for:
API requests,
API responses,
service inputs,
service outputs,
configuration objects,
database result records,
operations incidents,
remediation jobs,
Telegram records,
report metadata,
subscription-service definitions,
provider request and response records where useful.
Do not attempt to model every value in the repository at once. Add models where they reduce ambiguity, support API contracts, catch errors, make refactors safer, or help future AI agents understand data shape.
The model layer should not become the database access layer. Models define structure. Database access executes storage operations.
10. Database Access Standard
Database access should go through the standardized database access layer or the approved current adapter while migration is in progress.
The database access layer should standardize:
connection behavior,
query execution,
parameter binding,
transaction handling,
error handling,
query timing,
logging,
metrics,
health checks,
staging/production separation,
future portability.
The selected database access layer should be adopted incrementally. Existing SQL statements may remain during early phases if the surrounding infrastructure is standardized first.
Do not combine database access standardization with unrelated feed processor modularization, API work, frontend work, scheduler replacement, provider changes, or schema changes unless the task explicitly scopes that combined work.
11. Schema Migration Standard
Schema changes are not ordinary code edits.
Where compatible with the selected database path, schema changes should use the schema migration layer. Migration files should be version-controlled, reviewable, repeatable, tested against staging, documented, and reversible where practical.
Manual database changes should be avoided unless explicitly documented as emergency action or approved maintenance.
Schema migration work should not be mixed casually with runtime refactors, API changes, dashboard work, or provider integrations.
12. API-Ready Service Design
Backend services should be designed so they can later be exposed through the API layer when appropriate.
This means service functions should:
take clear inputs,
return structured outputs,
avoid direct terminal printing,
avoid direct web-framework assumptions,
avoid hidden global state where practical,
raise or return understandable errors,
use the configuration layer,
use the database access layer,
use integration wrappers,
emit useful operations records.
Not every service needs an API endpoint. The rule is that backend logic should not be trapped inside one presentation layer.
When API endpoints are created, route handlers should remain thin. They should validate input, check authentication and authorization, call service functions, return structured responses, and record telemetry.
13. Integration Wrapper Standard
Provider-specific integrations should be wrapped behind internal interfaces where practical.
This applies to:
Telegram,
email providers,
object storage,
external content APIs,
infrastructure provider APIs,
AI providers,
payment providers,
authentication integration points.
Application code should call internal wrappers rather than scattering provider-specific code across scripts and services.
The wrapper standard preserves future portability and makes provider replacement easier.
14. Scheduler Layer Standard
The scheduler layer is a capability, not a permanent commitment to one scheduler product.
The current environment may use the current operating system’s scheduler. Future deployment may use a different scheduler after the backend has clearer services, centralized configuration, worker identity, operations visibility, and CI/CD pipeline support.
Backend code should not be designed around the current scheduler. Scheduled work should be implemented as callable backend functions or thin command entry points that any scheduler can invoke.
A scheduled job should have:
clear name,
clear command,
clear working directory,
clear environment identity,
clear logging,
idempotency where required,
retry limits,
operations visibility,
safe production behavior.
The scheduler decision should not lead the backend architecture. It should follow a cleaner backend structure.
15. CLI, Dashboard, and Terminal Tool Standard
Command-line tools, internal dashboards, and terminal applications are entry points into shared backend services.
A CLI tool may use the terminal output layer to print tables, progress, warnings, validation reports, and operational summaries.
An internal dashboard may show reports, metrics, database inspection views, pipeline status, operational health, and staging/production comparisons.
A terminal application may be useful for SSH-friendly or keyboard-driven operational workflows.
All of these tools should call shared service modules. None should duplicate business logic or database access rules.
16. CI/CD Pipeline Standard For Backend Work
Backend structure and CI/CD pipeline structure must evolve together.
Each backend repository that is deployable or validation-critical should move toward a repository-specific pipeline configuration. The pipeline should begin with low-risk validation jobs and expand as the repository matures.
Backend changes should consider CI/CD impact when they affect:
Python project metadata,
dependency files,
lockfiles,
Python version files,
package layout,
script entry points,
runtime commands,
test commands,
lint or format commands,
API startup commands,
frontend build commands where applicable,
database migration commands,
worker commands,
environment variables,
protected variables,
deployment behavior.
The first backend pipeline should normally avoid deployment. It should validate that the remote repository can install dependencies, run compile/import checks, run lint checks, run tests, or run other no-mutation validation.
Production deployment must not be automated casually. Staging deployment may be introduced after validation jobs are reliable. Production deployment should require a manual gate until a later deployment standard changes that rule.
Before backend work is committed, the session should record whether the pipeline file changed, whether a pipeline update is needed, the expected pipeline behavior after push, and whether deployment is expected.
Architectural Impact Review Requirement
Before any repository commit is treated as ready, the session must perform an Architectural Impact Review. This review is broader than CI/CD. It asks whether the work affected or exposed required changes to validation, dependencies, configuration, environment variables, runtime commands, database behavior, API contracts, frontend behavior, monitoring, operations records, documentation, runbooks, CI/CD, Docker, Kubernetes, deployment, backups, security, or future AI-agent workflows.
This review is action-oriented. If the review identifies an issue that can be fixed safely inside the current task scope, the assistant or coding agent should fix and validate it before the commit. If the issue cannot be fixed safely before commit, it must be recorded as a generated follow-up task with enough context for a future human, AI session, automation worker, or operations database process to act on it.
Every commit and merge commit must also answer this question in its own subsection:
Has this commit revealed a weakness, ambiguity, missing safeguard, or repetitive manual step in the project standards or workflows?
The answer must state whether the issue was fixed before commit or deferred as a structured follow-up. It must not be hidden inside a generic risk note.
17. Operations and Observability Standard
Backend code should emit enough operational information for humans and future AI agents to understand system state.
Important backend workflows should record:
component identity,
environment,
subscription-service identity where relevant,
start time,
finish time,
success state,
failure reason,
retry state,
processing counts,
database timing,
provider timing,
operations event records,
incident records where appropriate.
The operations database layer is expected to become a coordination point for future AI-agent workflows. Backend services should produce structured records rather than vague messages.
18. Testing Standard for Backend Restructuring
Backend restructuring requires validation and tests.
The testing layer should begin with critical paths:
configuration loading,
environment separation,
database adapter import and behavior,
database access helpers,
Telegram publishing safety,
deduplication,
operations incident creation,
remediation job behavior,
service-layer workflows,
API response behavior when APIs exist,
scheduler/worker behavior.
Initial tests do not need full coverage. They need to protect the most dangerous behavior.
Every future refactor should define validation before implementation begins. If validation cannot be stated clearly, the refactor is not ready.
19. Linting and Formatting Standard
The linting and formatting layer should enforce consistent Python style once configured.
Do not mix broad formatting changes with behavior changes unless the task explicitly scopes that combination. A formatting-only cleanup should be separate from a behavior change so the diff remains reviewable.
AI-generated code should follow the same linting and formatting standards as human-written code.
20. Current Transition State
This section records current transition facts that future sessions may need. These facts are current-state guidance, not permanent architecture.
The current backend still includes working scripts, monitoring tools, operations scripts, configuration files, runtime support modules, and state handoff files inside the application repository.
A completed runtime package milestone created:
<workspace-root>/app/runtime/__init__.py
<workspace-root>/app/runtime/db_adapter.py
The active database adapter import pattern is:
from runtime import db_adapter
The runtime package may remain as a top-level package, move under a future src package, or be split into more specific packages. That decision requires repository inspection.
Known configuration-local candidates for future inspection include:
status_adapter.py
ai_routing.py
ai_provider_client.py
These files should not be moved casually. Each requires an inspection-only task before any relocation or restructuring.
This section should be updated or removed when the final backend package layout is adopted.
21. Migration Sequence
The recommended migration sequence is:
1. Establish Python project manager and project metadata.
2. Add linting and formatting configuration carefully.
3. Add testing framework and first critical-path tests.
4. Inventory current scripts and runtime modules.
5. Define final package layout and package name.
6. Move one shared runtime or service boundary at a time.
7. Centralize configuration.
8. Standardize database access incrementally.
9. Add explicit models where boundaries require them.
10. Build API endpoints on top of service modules.
11. Build dashboards and CLI tools on top of service modules.
12. Add or update CI/CD pipeline validation as repository commands, tests, and build behavior become stable.
13. Revisit scheduler strategy after services, configuration, and pipeline validation are cleaner.
14. Revisit deployment and containerization after startup commands, health checks, logs, and CI/CD validation are standardized.
This sequence is guidance, not permission to perform all steps in one task.
22. AI-Assisted Refactoring Rules
AI-assisted backend restructuring must be narrow.
Codex or another implementation agent should not be asked to:
clean up the backend,
modularize the processor,
standardize all database access,
move all runtime modules,
rewrite scripts into services,
fix all architecture issues.
A safe AI-assisted task should define:
repository path,
branch name,
current files to inspect,
allowed files to change,
forbidden files and actions,
whether the task is inspection-only,
whether dependencies may change,
whether lockfiles may change,
whether pipeline configuration may change,
validation commands,
handoff file,
expected Git state.
The agent should stop if it finds more consumers, more side effects, more changed files, or more scope than expected.
23. Repository Session Rules
Before implementation work begins, a session must confirm:
repository path,
branch,
working-tree state,
task scope,
allowed files,
validation expectations,
whether Codex will be used,
whether production resources are out of scope.
Do not assume repository state from memory or prior sessions. Use current terminal output or a current handoff.
Do not recommend commits, pushes, merges, branch deletion, database actions, scheduler changes, Telegram actions, or production actions unless the task and user authorize them.
24. Stop-and-Ask Triggers
Stop and ask before proceeding if:
1. A refactor touches more files than expected.
2. A module has unknown consumers.
3. A change would mutate production data.
4. A change would alter Telegram publishing behavior.
5. A change would alter customer-facing output.
6. A change would require database schema changes.
7. A change would require scheduler changes.
8. A change would introduce a new dependency.
9. A change would alter lockfiles.
10. A change would leave a compatibility wrapper.
11. A change would move files between repositories.
12. A change would require changing authentication behavior.
13. A change would expose internal tools.
14. A change would mix formatting cleanup with behavior changes.
Stopping is correct when scope changes.
25. Maintenance Requirements
Update this document when:
1. The Python project structure is formally established.
2. The package name is selected.
3. The runtime package is replaced or absorbed into a final package layout.
4. The configuration layer is implemented.
5. The database access layer is standardized.
6. Schema migration tooling is adopted.
7. The first API service module is created.
8. The first dashboard calls shared backend services.
9. The scheduler strategy changes.
10. A repository adopts or materially changes its CI/CD pipeline.
11. The project moves from Windows to Linux.
12. Containerization is introduced.
13. Future AI agents begin automated operations.
14. A production incident reveals missing backend structure rules.
26. Summary
The backend standard is service-oriented, package-based, testable, and migration-safe.
The core rules are:
use a project-managed Python structure,
put reusable logic in importable modules,
make scripts thin entry points,
keep business logic in service modules,
centralize configuration,
standardize database access,
use explicit models where boundaries matter,
wrap provider-specific integrations,
keep scheduler assumptions flexible,
validate before committing,
review CI/CD pipeline impact before commit readiness,
use AI implementation workers only inside narrow scopes.
This document should be part of the standard onboarding set for repository-related project sessions because it tells a new AI session how Python backend work must be structured before it proposes code, Codex tasks, service extraction, package moves, configuration changes, database work, APIs, CLI tools, dashboards, worker changes, or CI/CD pipeline updates.