clawd/gravl
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore(07-03): Stage deployment scripts and documentation updates

Browse Source
This commit is contained in:
Clawd Agent
2026-03-03 19:24:29 +01:00
parent fa766b21f7
commit 1104f6360e
4 changed files with 722 additions and 377 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/README.md
+179 -144
View File
@@ -1,155 +1,167 @@
# Gravl Backend # Gravl Backend
Node.js / Express API server for the Gravl application. Backend service for the Gravl exercise and fitness tracking platform.
## Development ## Overview
### Prerequisites The Gravl backend is a Node.js/Express application that provides:
- Node.js 18+ - REST API for exercise data management
- PostgreSQL 14+ (or use Docker Compose) - User authentication and authorization
- Docker & Docker Compose (for containerized development) - Integration with frontend via HTTP
- Health check endpoint for deployment monitoring
### Getting Started
```bash
# Install dependencies
npm install
# Create .env file (copy from .env.example)
cp .env.example .env
# Run database migrations
npm run migrate
# Start development server
npm run dev
```
The API will be available at `http://localhost:3001`.
### Health Check Endpoint
The API exposes a health check endpoint for deployment verification:
```bash
curl http://localhost:3001/api/health
```
Expected response:
```json
{
"status": "ok",
"timestamp": "2026-03-03T18:30:00Z"
}
```
This endpoint is used by the deployment scripts to verify the backend is healthy after deployment.
--- ---
## Deployment ## Local Development
### Quick Start ### Prerequisites
See `/docs/DEPLOYMENT.md` for comprehensive deployment documentation. - Node.js 18+
- npm or yarn
- Docker & Docker Compose (for local container development)
### Installation
```bash ```bash
# Deploy the application cd backend
scripts/deploy.sh npm install
# Check deployment status
scripts/build-check.sh
``` ```
### How It Works ### Running Locally
1. **Automatic build:** `scripts/deploy.sh` builds fresh Docker images **Development mode (with hot reload):**
2. **Zero downtime:** Old containers are replaced with `--force-recreate` ```bash
3. **Health verification:** API health endpoint is polled before deployment completes npm run dev
4. **Rollback:** Use git to revert and redeploy if issues arise ```
### Prerequisites for Deployment The server starts on `http://localhost:3001`
- Docker and Docker Compose installed **Production mode:**
- Git remote configured and accessible ```bash
- Backend listening on port 3001 npm run build
- Health endpoint (`/api/health`) responding with 200 OK npm start
```
### Example Deployment Workflow ### Environment Variables
Create a `.env` file in the backend directory:
```bash ```bash
# 1. Make code changes and commit NODE_ENV=development
git add . && git commit -m "feat: new API endpoint" PORT=3001
DATABASE_URL=postgresql://user:password@localhost:5432/gravl
# 2. Deploy from project root
cd /workspace/gravl
scripts/deploy.sh
# 3. Verify deployment
scripts/build-check.sh
# 4. Check logs if needed
docker compose logs gravl-backend
``` ```
### Container Labels See `.env.example` (if available) for all supported variables.
All deployed containers include build metadata labels for tracking: ---
- `org.opencontainers.image.revision` — Git commit SHA
- `org.opencontainers.image.created` — Build timestamp
These are used by `scripts/build-check.sh` to detect stale deployments. ## API Endpoints
### Health Check (Monitoring & Deployment)
```
GET /api/health
```
Used by deployment scripts to verify the backend is running and responsive.
**Response:**
```json
{
"status": "ok",
"timestamp": "2026-03-03T18:21:00Z"
}
```
**Status Codes:**
- `200 OK` — Backend is healthy
- `500 Internal Server Error` — Backend has errors (check logs)
### Other Endpoints
(Document your API endpoints here; placeholder for now)
--- ---
## Testing ## Testing
```bash ```bash
# Run unit tests npm test # Run all tests
npm test npm run test:watch # Run tests in watch mode
# Run integration tests
npm run test:integration
# Run with coverage
npm run test:coverage
``` ```
--- ---
## Database ## Docker
### Migrations ### Building the Image
```bash ```bash
# Run pending migrations docker build -t gravl-backend:latest .
npm run migrate
# Rollback last migration
npm run migrate:rollback
# Create new migration
npm run migrate:create -- my_migration_name
``` ```
### Connection ### Running in Container
Configure via `.env`: ```bash
``` docker run -p 3001:3001 \
DATABASE_URL=postgresql://user:password@localhost:5432/gravl -e NODE_ENV=production \
-e DATABASE_URL=postgresql://... \
gravl-backend:latest
``` ```
### With Docker Compose
See the root `docker-compose.yml` for multi-container setup.
--- ---
## Environment Variables ## Deployment
See `.env.example` for all available variables. ### Automated Deployment
Key variables: The backend is deployed using scripts in the root `scripts/` directory:
- `NODE_ENV` — Development/production mode
- `PORT` — Server port (default: 3001) - **`scripts/deploy.sh`** — Pulls latest code, builds fresh Docker image, starts container with health checks
- `DATABASE_URL` — PostgreSQL connection string - **`scripts/build-check.sh`** — Verifies deployed container matches local git HEAD
- `JWT_SECRET` — Token signing secret
### How to Deploy
```bash
cd /workspace/gravl
scripts/deploy.sh
```
### Checking Deployment Status
```bash
cd /workspace/gravl
scripts/build-check.sh
```
For complete deployment documentation, see: **[`docs/DEPLOYMENT.md`](../docs/DEPLOYMENT.md)**
That guide includes:
- Prerequisites and setup
- How to run deploy.sh
- How to check build status
- Troubleshooting (health check failures, stale containers, etc.)
- Recovery procedures (rollbacks, cleanup)
### Health Check Configuration
The backend exposes a health check endpoint at `GET /api/health`. The deployment script (`scripts/deploy.sh`) waits up to 60 seconds for this endpoint to return HTTP 200.
**In your backend code:**
```javascript
app.get('/api/health', (req, res) => {
res.json({ status: 'ok', timestamp: new Date().toISOString() });
});
```
**Deployment timeout:** 60 seconds (12 retries × 5 seconds)
- If this endpoint takes >5 seconds to respond, deployment will timeout
- Ensure health check is lightweight (no expensive DB queries)
--- ---
@@ -158,61 +170,84 @@ Key variables:
``` ```
backend/ backend/
├── src/ ├── src/
│ ├── api/ # Express route handlers │ ├── index.js # Server entry point
│ ├── middleware/ # Express middleware │ ├── routes/ # API endpoints
│ ├── models/ # Database models │ ├── controllers/ # Business logic
│ ├── services/ # Business logic │ ├── models/ # Data models (if using ORM)
│ └── index.js # App entry point │ └── middleware/ # Express middleware
├── tests/ # Unit and integration tests ├── test/ # Test files
├── migrations/ # Database migrations ├── Dockerfile # Container image definition
├── docker/ # Dockerfile ├── package.json # Dependencies
├── .env.example # Environment template
└── README.md # This file └── README.md # This file
``` ```
--- ---
## Logs
### Local Development
```bash
npm run dev # Logs to stdout
```
### Docker Container
```bash
docker logs gravl-backend # Current logs
docker logs -f gravl-backend # Follow logs in real-time
docker logs --tail 50 gravl-backend # Last 50 lines
```
### In Deployment
All deploy activity is logged to `logs/deploy.log` at the root:
```bash
tail logs/deploy.log
```
---
## Troubleshooting ## Troubleshooting
### API Won't Start ### Health Check Endpoint Not Responding
Check the logs: **Symptom:** Deployment fails with "Health check failed after 60s"
```bash
docker compose logs gravl-backend
```
Common issues: **Causes & Fixes:**
- Port 3001 already in use: Kill the process or change the port 1. **Port 3001 is already in use**
- Database connection failed: Verify `.env` DATABASE_URL ```bash
- Node modules missing: Run `npm install` lsof -i :3001
# Kill the conflicting process or use a different port
```
### Health Check Fails 2. **Backend code has a syntax error**
```bash
npm run dev # Look for error messages
```
Ensure the `/api/health` endpoint is implemented: 3. **Health check endpoint is not implemented**
- Ensure `app.get('/api/health', ...)` is in src/index.js
```javascript 4. **Database connection is failing**
// backend/src/api/health.js - Backend might be stuck trying to connect to DB
app.get('/api/health', (req, res) => { - Check `DATABASE_URL` in `.env`
res.json({ status: 'ok', timestamp: new Date().toISOString() }); - Ensure database is running
});
```
### Database Issues See **[`docs/DEPLOYMENT.md`](../docs/DEPLOYMENT.md#troubleshooting)** for more deployment troubleshooting.
Check Docker container status:
```bash
docker compose ps
docker compose logs gravl-db
```
--- ---
## Contributing ## Contributing
See `CODING-CONVENTIONS.md` in the project root for code style and standards. See the root project README or CONTRIBUTING.md for guidelines on:
- Code style ([CODING-CONVENTIONS.md](../docs/CODING-CONVENTIONS.md))
- Testing requirements
- Pull request process
--- ---
**Last Updated:** 2026-03-03 ## License
**Phase:** 07-03
**Related:** `/docs/DEPLOYMENT.md` [Specify your license here]
---
*Last updated: 2026-03-03*
docs/DEPLOYMENT.md
+380 -172
View File
@@ -1,17 +1,52 @@
# Gravl Deployment Guide # Gravl Deployment Guide
This guide covers how to deploy the Gravl application, verify deployments, and troubleshoot common issues. This guide covers how to deploy Gravl's backend and frontend services using automated scripts, verify deployment status, and handle troubleshooting and recovery scenarios.
---
## Overview
Gravl uses Docker and Docker Compose for containerization. Two automated scripts manage the deployment lifecycle:
- **`scripts/deploy.sh`**: Pulls latest code, builds fresh images (with `--no-cache` to prevent stale assets), and starts containers with health checks
- **`scripts/build-check.sh`**: Verifies that running containers match the current git HEAD (detects stale deployments)
---
## Prerequisites ## Prerequisites
- Docker and Docker Compose installed Before deploying, ensure you have:
- Git repository with remote configured
- Access to `/workspace/gravl` directory
- Backend API listening on `http://localhost:3001/api/health`
## Deployment Script 1. **Docker & Docker Compose** installed and running
```bash
docker --version
docker compose version
```
### Running a Deployment 2. **Git** configured with push/pull access to the repository
```bash
git remote -v
```
3. **Network access** to required ports:
- Backend: `localhost:3001` (health check at `http://localhost:3001/api/health`)
- Frontend: `localhost:3000` (or configured in `docker-compose.yml`)
4. **Sufficient disk space** for Docker images and volumes
```bash
docker system df
```
5. **No conflicting services** using ports 3000-3001
```bash
lsof -i :3000 -i :3001 # (macOS/Linux only)
```
---
## How to Run `deploy.sh`
### Basic Usage
```bash ```bash
cd /workspace/gravl cd /workspace/gravl
@@ -20,57 +55,98 @@ scripts/deploy.sh
### What It Does ### What It Does
1. **Pulls latest code:** `git pull` 1. **Git Pull**: Fetches and merges latest code from remote
2. **Captures build metadata:** - Exits if merge conflicts occur (manual resolution required)
- Git commit SHA
2. **Captures Metadata**:
- Current git commit hash
- Build timestamp - Build timestamp
3. **Builds fresh images:** `docker compose build --no-cache` - These are stored as Docker image labels for later verification
- `--no-cache` ensures all layers are rebuilt (prevents stale assets)
4. **Restarts containers:** `docker compose up -d --force-recreate`
5. **Health check:** Polls `/api/health` for up to 60 seconds
6. **Logs deployment:** Records all steps to `logs/deploy.log`
### Output Example 3. **Builds Docker Images** (`--no-cache`):
- Rebuilds all layers (no caching) to prevent stale assets
- Applies git commit and build timestamp as labels
4. **Starts Containers**:
- Uses `docker compose up -d --force-recreate` to ensure clean start
- Both backend and frontend containers are started
5. **Health Check**:
- Waits up to 60 seconds for backend to respond on `/api/health`
- Retries every 5 seconds (12 attempts max)
- Fails with exit code 1 if health check times out
### Exit Codes
| Code | Meaning | Next Steps |
|------|---------|-----------|
| 0 | Success | Deployment complete; containers healthy |
| 1 | Failure | See troubleshooting below |
### Logs
All deploy activity is logged to `logs/deploy.log`:
```bash
tail -50 logs/deploy.log # Last 50 lines
grep ERROR logs/deploy.log # Find errors
``` ```
[2026-03-03 18:30:00] === Deploy started ===
[2026-03-03 18:30:01] Pulling latest code... ### Environment Variables
[2026-03-03 18:30:05] Commit: 53f4df6 | Date: 2026-03-03T18:30:00Z
[2026-03-03 18:30:06] Building images (--no-cache)... Optional env vars can be set before running `deploy.sh`:
[2026-03-03 18:30:45] Starting containers...
[2026-03-03 18:30:50] Health check... | Variable | Default | Purpose |
[2026-03-03 18:30:55] Backend healthy |----------|---------|---------|
[2026-03-03 18:30:56] === Deploy complete: 53f4df6 === | `GIT_COMMIT` | auto-detected | Override git commit label (not recommended) |
``` | `BUILD_DATE` | auto-detected | Override build timestamp (not recommended) |
--- ---
## Checking Deployment Status ## How to Check Build Status (`build-check.sh`)
### Build Status Check Run this command anytime to verify deployed containers match your local code:
```bash ```bash
cd /workspace/gravl
scripts/build-check.sh scripts/build-check.sh
``` ```
### Output Example ### Output Example
**Healthy deployment:**
``` ```
Local HEAD: 53f4df6 (53f4df6f8a5c4d2e1f0a9b8c7d6e5f4a3b2c1d0) Local HEAD: abc1234 (abc1234567890abcdef1234567890abcdef123456)
[gravl-backend] Built: 53f4df6 on 2026-03-03T18:30:00Z [gravl-backend] Built: abc1234 on 2026-03-03T18:21:00Z
[gravl-backend] OK: up to date [gravl-backend] OK: up to date
[gravl-frontend] Built: abc1234 on 2026-03-03T18:21:00Z
[gravl-frontend] Built: 53f4df6 on 2026-03-03T18:30:00Z
[gravl-frontend] OK: up to date [gravl-frontend] OK: up to date
``` ```
### What the Check Tells You **Stale containers (code updated, not redeployed):**
```
Local HEAD: xyz5678 (xyz5678...)
- **OK: up to date** — Containers match the local git commit (everything is current) [gravl-backend] Built: abc1234 on 2026-03-03T18:21:00Z
- **STALE: container is behind local code** — Code has changed but containers haven't been redeployed yet [gravl-backend] STALE: container is behind local code — run scripts/deploy.sh
- **WARNING: no build label found** — Container is old (pre-07-02) and lacks build tracking labels [gravl-frontend] Built: abc1234 on 2026-03-03T18:21:00Z
[gravl-frontend] STALE: container is behind local code — run scripts/deploy.sh
```
**Missing labels (container built manually, not via deploy.sh):**
```
Local HEAD: abc1234
[gravl-backend] WARNING: no build label found — redeploy with scripts/deploy.sh to add tracking
[gravl-frontend] Not running
```
### Exit Codes
| Code | Meaning |
|------|---------|
| 0 | All checks completed (warnings don't fail; see output for status) |
| (no error exit) | Missing containers are noted but don't cause failure |
--- ---
@@ -78,215 +154,347 @@ Local HEAD: 53f4df6 (53f4df6f8a5c4d2e1f0a9b8c7d6e5f4a3b2c1d0)
### Health Check Failures ### Health Check Failures
**Symptom:** Deployment fails with "ERROR: Health check failed after 60s" **Symptom:** `ERROR: Health check failed after 60s`
**Possible Causes & Solutions:** **Causes & Solutions:**
| Cause | Solution | 1. **Backend service didn't start**
|-------|----------| ```bash
| Backend not starting | Check logs: `docker compose logs gravl-backend` | docker logs gravl-backend | tail -20
| Health endpoint not implemented | Implement `GET /api/health` in backend (returns `200 OK`) | # Look for:
| Network issues | Verify network: `docker network inspect gravl` or restart: `docker compose restart` | # - Port conflicts (ERR_EADDRINUSE)
| Port already in use | Check: `lsof -i :3001` and kill the process or change port | # - Missing dependencies (module not found)
| Insufficient resources | Free disk space: `df -h` or reduce image size | # - Database connection errors
```
2. **Port 3001 is already in use**
```bash
lsof -i :3001 # Find what's using it
docker port gravl-backend # Check exposed port
kill -9 <PID> # Kill conflicting process (if safe)
scripts/deploy.sh # Retry
```
3. **Network issue between host and container**
```bash
docker inspect gravl-backend --format '{{.NetworkSettings.IPAddress}}'
curl -sf http://<container-ip>:3001/api/health # Test directly
```
4. **Backend code has syntax error**
```bash
docker logs gravl-backend 2>&1 | grep -i "syntax\|error\|exception"
# Check backend/src/index.js for obvious errors
# Revert recent changes: git log --oneline -5 && git checkout <good-commit>
```
**Quick recovery:**
**Manual Restart:**
```bash ```bash
docker compose restart gravl-backend # 1. Stop everything
# Wait a few seconds docker compose down
curl -sf http://localhost:3001/api/health
# 2. Check backend logs
docker compose up -d gravl-backend
sleep 5
docker logs gravl-backend | tail -50
# 3. If logs show errors, fix code and retry
git diff HEAD~1..HEAD backend/src/
# ... fix issues ...
scripts/deploy.sh
``` ```
--- ---
### Stale Containers ### Stale Containers
**Symptom:** `build-check.sh` shows "STALE: container is behind local code" **Symptom:** `build-check.sh` shows `STALE: container is behind local code`
**Cause:** Code has been updated but containers haven't been redeployed. **Causes:**
- Code was updated (`git pull`) but `deploy.sh` hasn't been run
- Deployment failed partway through
- Manual restart without redeploy
**Solution:** **Solution:**
```bash ```bash
scripts/deploy.sh scripts/deploy.sh
scripts/build-check.sh # Should now show OK scripts/build-check.sh # Verify update
``` ```
--- ---
### Missing Docker Labels ### Missing Build Labels
**Symptom:** `build-check.sh` shows "WARNING: no build label found" **Symptom:** `WARNING: no build label found — redeploy with scripts/deploy.sh`
**Cause:** Containers were built before phase 07-02 (before labels were added). **Causes:**
- Container was built with `docker compose build` directly (not via `deploy.sh`)
- Container predates the labeling system
**Solution:** **Solution:**
```bash
scripts/deploy.sh # Rebuilds with labels
```
---
### Deployment Hangs
**Symptom:** `scripts/deploy.sh` doesn't complete or appears stuck.
**Possible Causes & Solutions:**
| Symptom | Solution |
|---------|----------|
| Stuck at "Building images" | Docker build is slow. Check: `docker builder prune` to free cache |
| Stuck at "Health check" | Backend not responding. Try: `docker compose logs` to see errors |
| Git pull conflicts | Resolve conflicts manually: `cd /workspace/gravl && git status` |
**Force Stop:**
```bash
# Kill the deploy script
pkill -f scripts/deploy.sh
# Manually check status
docker compose ps
docker compose logs
```
---
## Rollback Procedures
### Quick Rollback
If the current deployment is broken:
```bash ```bash
# Revert to previous commit # Re-deploy to add labels
git reset --hard HEAD~1
# Redeploy
scripts/deploy.sh
# Verify
scripts/build-check.sh
```
### Multi-Commit Rollback
If you need to go back several commits:
```bash
# View recent commits
git log --oneline -10
# Rollback to a specific commit (example: abc1234)
git reset --hard abc1234
# Redeploy
scripts/deploy.sh scripts/deploy.sh
``` ```
### Rollback Verification ---
After rolling back, verify the system is stable: ### Container Won't Start (CrashLoopBackOff / Exited)
**Symptom:** `docker compose ps` shows container in "Exited" state
**Steps:**
1. **Check container logs**
```bash
docker logs gravl-backend --tail 50
docker logs gravl-frontend --tail 50
```
2. **Check docker-compose.yml for typos**
```bash
docker compose config # Validates syntax
```
3. **Inspect health check endpoint**
```bash
curl -v http://localhost:3001/api/health
# Should see HTTP 200, not 404 or 500
```
4. **If all else fails, clean rebuild**
```bash
docker compose down
docker rmi gravl-backend gravl-frontend
docker system prune -f
scripts/deploy.sh
```
---
### Database Connection Issues
**Symptom:** Backend logs show `Connection refused` or `ECONNREFUSED`
**Causes:**
- Database service not running
- Wrong host/port in `.env` or backend code
- Network issue between containers
**Solutions:**
1. **Check database service status** (if applicable)
```bash
docker compose ps # All services running?
docker network ls # Check gravl network exists
```
2. **Verify connection string in `.env`**
```bash
cat .env | grep -i database
# Should match docker-compose.yml service name (e.g., gravl-db:5432)
```
3. **Test connection from backend container**
```bash
docker exec gravl-backend ping gravl-db
docker exec gravl-backend curl http://gravl-db:5432 # If HTTP, adjust port
```
---
### Disk Space Issues
**Symptom:** `no space left on device` during build
**Solution:**
```bash ```bash
# Check containers match the previous code # Check disk usage
scripts/build-check.sh docker system df
# Check API is healthy # Clean up unused images/containers
curl -sf http://localhost:3001/api/health | jq . docker system prune -a --volumes
# Check frontend is responsive # Then retry deploy
curl -sf http://localhost:3000/ | head -c 500 scripts/deploy.sh
``` ```
--- ---
## Manual Container Cleanup ## Recovery Procedures
If containers become corrupted or stuck: ### Manual Rollback to Previous Commit
Use this when the deployed code is broken and you need to quickly revert.
```bash ```bash
# Stop all containers # 1. Find the last good commit
git log --oneline -10 # Review recent commits
# 2. Check out the known-good commit
git checkout <commit-hash>
# 3. Redeploy
scripts/deploy.sh
# 4. Verify
scripts/build-check.sh
curl -sf http://localhost:3001/api/health
# 5. Document the incident
echo "Rolled back to <commit-hash> due to <reason>" >> logs/rollback.log
```
### Emergency Container Cleanup
Use this when containers are hung, corrupted, or in an unknown state.
```bash
# 1. Stop all services
docker compose down docker compose down
# Remove volumes (WARNING: deletes data) # 2. Remove images (forces fresh rebuild)
docker compose down -v docker rmi gravl-backend gravl-frontend
# Verify they're gone # 3. Clear unused volumes (optional; use with caution!)
docker compose ps # docker volume prune
# Full redeploy # 4. Rebuild from scratch
scripts/deploy.sh scripts/deploy.sh
# 5. Verify all containers running and healthy
docker compose ps
scripts/build-check.sh
curl -sf http://localhost:3001/api/health
```
**Safety Check:** If your data is in Docker volumes, `docker volume prune` will destroy them. Skip this step unless you're sure you don't need the data.
### Staged Rollback (Zero-Downtime)
If you're running a blue-green deployment setup:
```bash
# 1. Deploy to green environment
cd /path/to/green
git pull && docker compose build --no-cache && docker compose up -d
# 2. Test green (health check, smoke tests)
curl -sf http://green-backend:3001/api/health
# 3. Switch traffic to green (via load balancer or DNS)
# (Implementation depends on your infrastructure)
# 4. If green has issues, revert traffic to blue immediately
# (Blue kept serving; no downtime)
# 5. Debug green offline
docker logs gravl-backend
``` ```
--- ---
## Monitoring & Logs ## Monitoring After Deployment
### Deployment Log ### Immediate Checks (after `deploy.sh` completes)
```bash ```bash
tail -f logs/deploy.log # Containers are running
docker compose ps
# Backend is healthy
curl -sf http://localhost:3001/api/health | jq .
# Containers match local code
scripts/build-check.sh
# Logs have no errors
docker logs gravl-backend 2>&1 | grep -i error | head -5
``` ```
### Container Logs ### Ongoing Checks (periodically)
```bash ```bash
# Backend logs # Run build-check regularly (cron every 30 min, or manual)
docker compose logs gravl-backend scripts/build-check.sh
# Frontend logs # Monitor resource usage
docker compose logs gravl-frontend docker stats gravl-backend gravl-frontend
# All logs with timestamps # Audit logs for issues
docker compose logs --timestamps --follow docker logs gravl-backend --since 1h --until now | grep ERROR
``` ```
### Build Info ### Example Monitoring Script
```bash ```bash
# List deployed images #!/bin/bash
docker images | grep gravl # Save as scripts/health-monitor.sh
set -euo pipefail
# Inspect container labels (build metadata) HEALTHY=true
docker inspect gravl-backend | jq '.Config.Labels'
# Check containers running
docker compose ps | grep -q "Up" || HEALTHY=false
# Check health endpoint
curl -sf http://localhost:3001/api/health || HEALTHY=false
# Check for stale containers
scripts/build-check.sh | grep -q "STALE" && HEALTHY=false
if [ "$HEALTHY" = "true" ]; then
echo "[$(date)] Gravl is healthy ✓"
else
echo "[$(date)] Gravl has issues! See above." >&2
exit 1
fi
``` ```
--- ---
## Best Practices ## Best Practices
1. **Always test in staging first** — Validate the deploy in a non-production environment 1. **Always run `build-check.sh` before deploying changes**
2. **Check status before deploying** — Run `scripts/build-check.sh` to ensure no stale containers - Ensures you know current state
3. **Review logs after deployment** — Check `logs/deploy.log` for warnings or errors - Catches stale containers early
4. **Plan rollbacks** — Know which commits are stable before deploying
5. **Monitor health endpoints** — Regularly ping `/api/health` in production 2. **Review changes before deploying**
6. **Backup before major changes** — Tag releases in git before significant deployments ```bash
7. **Use semantic commits** — Make it easy to identify which commits introduced changes git log --oneline -5 # Recent commits
git diff origin/main..HEAD # What will be deployed
```
3. **Test in staging first**
- Separate staging environment for pre-production testing
- Deploy to staging, verify, then deploy to production
4. **Keep logs rotated**
- `logs/deploy.log` can grow large
- Use `logrotate` or manual cleanup: `tail -1000 logs/deploy.log > logs/deploy.log.1 && > logs/deploy.log`
5. **Automate regular checks**
- Cron job to run `build-check.sh` every 30 minutes
- Send alerts if "STALE" or "WARNING" found
6. **Document rollbacks**
- Always log why you rolled back
- Review patterns (e.g., "rolled back 3 times this week" = code review process failing)
--- ---
## FAQ ## See Also
**Q: Can I deploy without building (e.g., just restart containers)?** - **Testing**: [DEPLOYMENT_TEST_PLAN.md](./DEPLOYMENT_TEST_PLAN.md) — comprehensive test scenarios
A: No. The script always rebuilds to prevent stale code. This is intentional for safety. - **Code style**: [CODING-CONVENTIONS.md](./CODING-CONVENTIONS.md)
- **Architecture**: Backend README or architecture docs (if available)
**Q: How long should a deployment take?**
A: Typically 60-90 seconds (build time + health check). If longer, check Docker build performance.
**Q: What if I need to deploy a specific commit?**
A: Check it out first, then deploy:
```bash
git checkout <commit-sha>
scripts/deploy.sh
```
**Q: Can I skip the health check?**
A: Not recommended. The health check prevents deploying broken code. Fix the health endpoint instead.
**Q: What data is lost if I rollback?**
A: Container rollback only reverts code. Database data persists unless you `docker compose down -v`.
--- ---
**Last Updated:** 2026-03-03 *Last updated: 2026-03-03 | Maintained by: Gravl Development Team*
**Document Version:** 1.0
**Phase:** 07-03
scripts/build-check.sh
+64 -26
View File
@@ -1,68 +1,106 @@
#!/bin/bash #!/bin/bash
# Compare deployed container versions against local git HEAD # Gravl Build Status Checker
# Warns if containers are stale (built from an older commit) #
# Purpose:
# Verifies that deployed containers match the current git HEAD.
# Warns if containers are stale (built from older commits).
# Helps you catch situations where code was updated but not redeployed.
#
# How it works:
# 1. Gets current local git commit (HEAD)
# 2. Queries each container's build labels
# 3. Compares container label commit vs local HEAD
# 4. Reports status: "OK", "STALE", or "WARNING"
#
# Exit codes:
# 0 = All checks completed (see output for individual status)
# (Warnings don't cause non-zero exit)
# #
# Usage: # Usage:
# ./scripts/build-check.sh # ./scripts/build-check.sh
# #
# What it shows: # Example output:
# - Local git HEAD commit SHA # Local HEAD: abc1234 (abc1234567890abcdef...)
# - Each container's built commit SHA (from Docker labels)
# - Whether containers are up-to-date or stale
# - Warnings if labels are missing (pre-07-02 containers)
# #
# Label fields read: # [gravl-backend] Built: abc1234 on 2026-03-03T18:21:00Z
# - org.opencontainers.image.revision = Git commit SHA embedded by deploy.sh # [gravl-backend] OK: up to date
# - org.opencontainers.image.created = Build timestamp (ISO 8601 format) # [gravl-frontend] Built: abc1234 on 2026-03-03T18:21:00Z
# # [gravl-frontend] OK: up to date
# Exit codes:
# 0 = All containers up to date
# 1+ = Warnings or stale containers detected
#
# See: /docs/DEPLOYMENT.md for troubleshooting
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_DIR="$(dirname "$SCRIPT_DIR")" REPO_DIR="$(dirname "$SCRIPT_DIR")"
cd "$REPO_DIR" cd "$REPO_DIR"
# Get the current local git commit (what's checked out locally)
LOCAL_COMMIT=$(git rev-parse HEAD) LOCAL_COMMIT=$(git rev-parse HEAD)
echo "Local HEAD: $(git rev-parse --short HEAD) ($LOCAL_COMMIT)" echo "Local HEAD: $(git rev-parse --short HEAD) ($LOCAL_COMMIT)"
echo "" echo ""
# Check a single container's build status # ============================================================================
# Args: $1 = container name # check() helper function
# ============================================================================
# Queries a container's build labels and compares against local HEAD.
#
# Parameters:
# $1 = Container name (e.g., "gravl-backend")
#
# Label fields used:
# org.opencontainers.image.revision = commit hash when image was built
# Format: 40-character SHA (same as git rev-parse HEAD)
# Set by: scripts/deploy.sh -> docker compose build args
#
# org.opencontainers.image.created = RFC3339 timestamp when image was built
# Format: 2026-03-03T18:21:00Z
# Set by: scripts/deploy.sh -> docker compose build args
# Purpose: Shows humans when the image was built (for diagnostics)
#
# Status outcomes:
# - "Not running": Container doesn't exist or isn't running
# - "WARNING": Container exists but has no revision label
# Fix: Re-deploy with scripts/deploy.sh
# - "OK": Container label commit = local HEAD (up to date)
# - "STALE": Container label commit != local HEAD
# Fix: Run scripts/deploy.sh to update container
check() { check() {
local name="$1" local name="$1"
# Container not running # Check if container exists and is running
if ! docker inspect "$name" &>/dev/null; then if ! docker inspect "$name" &>/dev/null; then
echo "[$name] Not running" echo "[$name] Not running"
return return
fi fi
# Read Docker labels set by deploy.sh # Extract build labels from container config
# If labels are missing, container was built before phase 07-02 # These labels are set in the docker-compose.yml build args,
# and the Dockerfile COPYs them into image labels.
local commit date local commit date
commit=$(docker inspect "$name" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' 2>/dev/null) commit=$(docker inspect "$name" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' 2>/dev/null)
date=$(docker inspect "$name" --format '{{index .Config.Labels "org.opencontainers.image.created"}}' 2>/dev/null) date=$(docker inspect "$name" --format '{{index .Config.Labels "org.opencontainers.image.created"}}' 2>/dev/null)
# Check if revision label exists
if [ -z "$commit" ] || [ "$commit" = "unknown" ]; then if [ -z "$commit" ] || [ "$commit" = "unknown" ]; then
echo "[$name] WARNING: no build label found — redeploy with scripts/deploy.sh to add tracking" echo "[$name] WARNING: no build label found — redeploy with scripts/deploy.sh to add tracking"
return return
fi fi
# Display container info # Display when this container's image was built
echo "[$name] Built: ${commit:0:7} on ${date:-unknown}" echo "[$name] Built: ${commit:0:7} on ${date:-unknown}"
# Compare container commit to local HEAD # Compare container's commit against local HEAD
# If they match, container is up to date.
# If they differ, code has changed locally but container hasn't been redeployed.
if [ "$commit" = "$LOCAL_COMMIT" ]; then if [ "$commit" = "$LOCAL_COMMIT" ]; then
echo "[$name] OK: up to date" echo "[$name] ✓ OK: up to date"
else else
echo "[$name] STALE: container is behind local code — run scripts/deploy.sh" echo "[$name] ⚠ STALE: container is behind local code — run scripts/deploy.sh"
fi fi
} }
# Check both containers # ============================================================================
# Check Each Service
# ============================================================================
# These are the service names defined in docker-compose.yml.
# Adjust if you rename services.
check "gravl-backend" check "gravl-backend"
check "gravl-frontend" check "gravl-frontend"
scripts/deploy.sh
+97 -33
View File
@@ -1,24 +1,31 @@
#!/bin/bash #!/bin/bash
# Gravl deployment script # Gravl Deployment Script
# Prevents stale containers by always building fresh with --no-cache #
# Purpose:
# Automates the deployment of Gravl services to production/staging.
# Ensures fresh builds and verifies service health after startup.
#
# Prevents stale containers by always building fresh with --no-cache:
# The --no-cache flag rebuilds all Docker layers from scratch.
# This prevents stale application code, assets, or dependencies
# from being cached and deployed. Essential for reliable deployments.
#
# Workflow:
# 1. Pull latest code from git
# 2. Capture build metadata (commit hash, timestamp)
# 3. Build Docker images (--no-cache for freshness)
# 4. Start containers with new images
# 5. Health check: wait for backend to respond
#
# Exit codes:
# 0 = Success (deployment complete, services healthy)
# 1 = Failure (see error message in logs)
# #
# Usage: # Usage:
# ./scripts/deploy.sh # ./scripts/deploy.sh
# #
# What it does: # Logs:
# 1. Pulls latest code from git # All output saved to logs/deploy.log (see tail to follow)
# 2. Captures build metadata (commit SHA, timestamp)
# 3. Builds fresh Docker images with --no-cache (no layer caching)
# 4. Restarts containers to use new images
# 5. Polls /api/health endpoint until backend is ready
# 6. Logs all steps to logs/deploy.log
#
# Rationale for --no-cache:
# Docker caching can hide stale assets (JS, CSS, images) when source files change.
# Using --no-cache ensures all layers rebuild fresh, guaranteeing new code is deployed.
# Trade-off: Slightly slower builds (30-60s vs 10-20s with cache), but safer.
#
# See: /docs/DEPLOYMENT.md for troubleshooting
set -euo pipefail set -euo pipefail
@@ -27,49 +34,106 @@ REPO_DIR="$(dirname "$SCRIPT_DIR")"
LOG_FILE="$REPO_DIR/logs/deploy.log" LOG_FILE="$REPO_DIR/logs/deploy.log"
BACKEND_HEALTH="http://localhost:3001/api/health" BACKEND_HEALTH="http://localhost:3001/api/health"
# Logging helper: prints timestamp + message to both stdout and log file
log() { log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE" echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
} }
# Ensure logs directory exists
mkdir -p "$REPO_DIR/logs" mkdir -p "$REPO_DIR/logs"
cd "$REPO_DIR" cd "$REPO_DIR"
log "=== Deploy started ===" log "=== Deploy started ==="
# Pull latest code from remote # ============================================================================
# Fails if there are local changes or merge conflicts # STEP 1: Git Pull
# ============================================================================
# Fetches latest code from remote and merges into current branch.
# Fails if there are merge conflicts (manual intervention required).
log "Pulling latest code..." log "Pulling latest code..."
git pull git pull
# Capture build metadata to embed in Docker image labels # ============================================================================
# These labels allow build-check.sh to verify deployed containers match local code # STEP 2: Capture Build Metadata
# ============================================================================
# Build labels are attached to Docker images and stored in container labels.
# These are used by build-check.sh to verify deployed containers match local HEAD.
#
# Labels:
# org.opencontainers.image.revision = git commit hash (40-char SHA)
# Purpose: Track which commit the image was built from
# Example: abc1234567890abcdef1234567890abcdef123456
#
# org.opencontainers.image.created = RFC3339 timestamp
# Purpose: Track when the image was built
# Example: 2026-03-03T18:21:00Z
GIT_COMMIT=$(git rev-parse HEAD) GIT_COMMIT=$(git rev-parse HEAD)
BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
log "Commit: $(git rev-parse --short HEAD) | Date: $BUILD_DATE" log "Commit: $(git rev-parse --short HEAD) | Date: $BUILD_DATE"
# Build fresh images — no-cache prevents Docker layer caching # ============================================================================
# This is critical for frontend deployments where CSS/JS changes might not be obvious # STEP 3: Build Docker Images (--no-cache)
# to Docker's layer detection algorithm # ============================================================================
log "Building images (--no-cache)..." # Why --no-cache?
# Docker layer caching can hide stale assets (CSS, JS bundles, dependencies).
# Example: If package.json changes but npm install is cached, old dependencies are used.
# --no-cache forces full rebuild of all layers every time.
#
# Build args are passed to Dockerfile via export, allowing them to be used
# in RUN instructions or referenced in labels (see docker-compose.yml).
log "Building images (--no-cache to prevent stale assets)..."
export GIT_COMMIT BUILD_DATE export GIT_COMMIT BUILD_DATE
docker compose build --no-cache docker compose build --no-cache
# Restart containers with new images # ============================================================================
# --force-recreate stops old containers and removes them before starting new ones # STEP 4: Start Containers with New Images
# ============================================================================
# docker compose up -d --force-recreate:
# -d = Run in background (detached mode)
# --force-recreate = Stop and remove existing containers, start fresh
# Ensures old containers with old images are not reused.
#
# This step also networks containers (creates/reuses docker network).
log "Starting containers..." log "Starting containers..."
docker compose up -d --force-recreate docker compose up -d --force-recreate
# Health check: poll /api/health endpoint until it responds with 200 OK # ============================================================================
# Timeout: 60 seconds (12 retries × 5 seconds each) # STEP 5: Health Check
# This prevents deployment from completing if the backend is broken # ============================================================================
log "Health check..." # Waits for backend to respond on /api/health endpoint.
# This proves the service started correctly and is ready for traffic.
#
# Timeout configuration:
# Loop: 12 iterations
# Interval: 5 seconds per iteration
# Total: 60 seconds max wait time
#
# Why 60 seconds?
# - Docker startup: ~5-10 seconds
# - Node.js app initialization: ~5 seconds
# - Database connection: ~5-10 seconds
# - Buffer for system load: ~30 seconds
#
# If this timeout is too short, you may see false negatives (healthy app fails check).
# If too long, deployment takes unnecessarily long to fail.
#
# Endpoint details:
# URL: http://localhost:3001/api/health
# Method: GET
# Expected status: 200
# Should complete in <1 second
log "Health check: waiting for backend (60s timeout)..."
for i in $(seq 1 12); do for i in $(seq 1 12); do
if curl -sf "$BACKEND_HEALTH" >/dev/null 2>&1; then if curl -sf "$BACKEND_HEALTH" >/dev/null 2>&1; then
log "Backend healthy" log "✓ Backend healthy"
break break
fi fi
[ "$i" -eq 12 ] && { log "ERROR: Health check failed after 60s"; exit 1; } if [ "$i" -eq 12 ]; then
log "Waiting... ($i/12)" log "✗ ERROR: Health check failed after 60s"
log " Try: docker logs gravl-backend | tail -20"
exit 1
fi
log " Waiting... ($i/12 attempts, 5s intervals)"
sleep 5 sleep 5
done done