Introduction: why secrets do not belong in code
Storing sensitive information such as API keys, database passwords, or tokens directly in Python code is dangerous. If those secrets leak, attackers can break into connected systems, and the fallout can include lost trust, financial damage, and legal consequences. The safer pattern is to externalize secrets so they never appear in code or version control. A long-standing best practice — codified in the twelve-factor app methodology — is to keep configuration and secrets in environment variables, outside the codebase. Because managing environment variables by hand is tedious, a single .env file has become the convenient standard for local development.
This guide, based on an article by machine learning engineer and technical writer Kanwal Mehreen, walks through seven practical techniques for managing secrets in Python projects, with code examples and common pitfalls.
Technique 1: Use a .env file locally — and load it safely
A .env file is a plain-text file of key=value pairs kept locally and excluded from version control. It defines environment-specific settings and secrets for development. A recommended project layout:
my_project/
app/
main.py
settings.py
.env # NOT committed – contains real secrets
.env.example # committed – lists keys without real values
.gitignore
pyproject.tomlReal secrets live only in the local .env:
# .env (local only, never commit)
OPENAI_API_KEY=your_real_key_here
DATABASE_URL=postgresql://user:pass@localhost:5432/mydb
DEBUG=trueBy contrast, .env.example is a committed template that shows other developers which keys are needed — with the values left empty:
# .env.example (commit this)
OPENAI_API_KEY=
DATABASE_URL=
DEBUG=falseAdding .env to .gitignore ensures the real file is never committed accidentally. In Python, the python-dotenv library loads the .env file at runtime; in app/main.py, for example:
# app/main.py
import os
from dotenv import load_dotenv
load_dotenv() # reads variables from .env into os.environ
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError("Missing OPENAI_API_KEY. Set it in your environment or .env file.")
print("App started (key loaded).")Here load_dotenv() automatically finds .env in the working directory and sets each key=value pair into os.environ (unless the variable is already set). This guards against committing or insecurely sharing secrets while keeping a clean, reproducible development environment — switching machines or setups requires no code changes.
Technique 2: Read secrets from the environment
Placeholders like API_KEY=’test’ hard-coded into source, or the assumption that variables are always set, work on one machine and fail in production — and a missing secret can silently activate an insecure placeholder. The robust approach is to fetch secrets from environment variables at runtime with os.environ or os.getenv:
def require_env(name: str) -> str:
value = os.getenv(name)
if not value:
raise RuntimeError(f"Missing required environment variable: {name}")
return value
OPENAI_API_KEY = require_env("OPENAI_API_KEY")If a required secret is missing, the application fails fast at startup — far safer than proceeding with a missing or dummy value.
Technique 3: Validate configuration with a settings module
As projects grow, scattered os.getenv calls become disorganized and error-prone. A settings class — such as Pydantic’s BaseSettings, available in the pydantic-settings package — centralizes configuration, validates types, and loads values from both .env and the environment:
# app/settings.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
openai_api_key: str = Field(min_length=1)
database_url: str = Field(min_length=1)
debug: bool = False
settings = Settings()Then, in the application:
# app/main.py
from app.settings import settings
if settings.debug:
print("Debug mode on")
api_key = settings.openai_api_keyThis prevents mistyped keys, incorrect parsing (the string “false” versus the boolean False), and duplicated environment lookups. A settings class also guarantees the app fails fast when secrets are missing, avoiding “works on my machine” surprises.
Technique 4: Use platform or CI secrets for deployment
A local .env file should never be copied to production. Hosting and CI platforms provide their own secret management: GitHub Actions, for example, stores encrypted secrets in repository settings and injects them into workflows at runtime, so real values never appear in code or logs. Cloud platforms offer equivalent mechanisms.
Technique 5: Handle Docker carefully
In Docker, secrets should not be baked into images or passed as plain ENV instructions — environment variables can leak through process listings and logs. Docker and Kubernetes both provide dedicated secrets mechanisms that are more secure. For local development, .env plus python-dotenv is fine; in production containers, mount secrets or use Docker secrets, and avoid ENV API_KEY=… lines or files containing secrets in Dockerfiles. This limits permanent exposure inside images and simplifies rotation.
Technique 6: Add guardrails
Humans make mistakes, so secret safety should be automated. GitHub’s push protection can block commits containing detected secrets, and scanning tools such as TruffleHog and Gitleaks catch leaked credentials in CI before a merge. Guardrails stop leaks before they enter the repository, making day-to-day work with .env files and environment variables considerably safer.
Technique 7: Use a real secrets manager
For larger applications, a dedicated secrets manager — HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault — controls who can access each secret, logs every access, and automates key rotation. Without one, teams tend to reuse credentials and forget rotation. A secrets manager keeps production protected even if a developer’s machine or a local .env file is exposed.
A practical checklist
- .env is in .gitignore; real credentials are never committed
- .env.example exists and is committed with empty values
- Code reads secrets only through environment variables (os.getenv, a settings class, etc.)
- The app fails fast with a clear error if any required secret is missing
- Different secrets are used for dev, staging, and production — never the same key
- CI and deployment use encrypted secrets (GitHub Actions secrets, AWS Parameter Store, etc.)
- Push protection and/or secret scanning is enabled on the repository
- A rotation policy exists: rotate immediately on any leak, and regularly otherwise
Limitations and what to watch
- .env files protect against accidental commits, not against malware or attackers with access to the machine — they are stored in plain text.
- Environment variables are visible to the whole process and its children; for high-security contexts, prefer file-mounted or manager-fetched secrets.
- Secret scanning catches known token formats; custom secrets may need custom detection rules.
- Secrets-manager pricing and operational overhead are real; small projects may reasonably start with platform-provided encrypted secrets and graduate later.
Wrapping up
Keeping secrets safe is less about following rules and more about building a workflow that makes projects secure, maintainable, and portable across environments. Local .env files, environment-based access, validated settings, platform secrets, container hygiene, automated guardrails, and — at scale — a proper secrets manager together cover the lifecycle from first prototype to production. These practices matter doubly for AI projects, where API keys for model providers are among the most commonly leaked credentials; related tooling is covered in this overview of Python libraries for building LLM applications.
Based on material by Kanwal Mehreen, a machine learning engineer and technical writer focused on the intersection of AI and data science, co-author of the ebook “Maximizing Productivity with ChatGPT,” Google Generation Scholar 2022 (APAC), Teradata Diversity in Tech Scholar, and founder of FEMCodes, which supports women in STEM.
