Skip to main content

Python CI/CD Integration

Introduction​

Continuous Integration and Continuous Deployment (CI/CD) are essential practices in modern software development that help teams deliver code changes more frequently and reliably. For Python developers, integrating CI/CD into projects means automating testing, linting, building, and deployment processes to ensure high-quality code reaches production quickly and safely.

In this tutorial, you'll learn:

  • What CI/CD is and why it matters for Python projects
  • How to set up basic CI/CD pipelines for Python applications
  • Popular CI/CD tools for Python development
  • Best practices for Python CI/CD workflows

What is CI/CD?​

Continuous Integration (CI) is the practice of frequently merging code changes into a shared repository, with automated tests to catch issues early.

Continuous Deployment (CD) extends this by automatically deploying all code changes to a testing or production environment after the build stage.

For Python projects, CI/CD brings several benefits:

  • Automatically running tests on every code change
  • Ensuring code style consistency with linters
  • Building and packaging Python applications reliably
  • Deploying to development, staging, or production environments

Getting Started with Python CI/CD​

Prerequisites​

Before setting up CI/CD for your Python project, you'll need:

  • A Python project with tests (e.g., using pytest, unittest)
  • A version control system (typically Git)
  • A repository on GitHub, GitLab, Bitbucket, or similar platform
  • Basic understanding of YAML for configuration files

1. GitHub Actions​

GitHub Actions is a popular choice for Python CI/CD due to its tight integration with GitHub repositories.

Example: Basic Python CI Workflow​

Create a file at .github/workflows/python-ci.yml:

name: Python CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8, 3.9, 3.10]

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Lint with flake8
      run: |
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
    - name: Test with pytest
      run: |
        pytest

This workflow:

  1. Triggers on pushes to the main branch or pull requests
  2. Tests across multiple Python versions
  3. Installs project dependencies
  4. Runs linting with flake8
  5. Executes tests with pytest

2. GitLab CI/CD​

If you're using GitLab, you can create a .gitlab-ci.yml file in your project root:

image: python:3.9

stages:
  - test
  - deploy

before_script:
  - pip install -r requirements.txt

test:
  stage: test
  script:
    - pip install pytest pytest-cov
    - pytest --cov=myproject

lint:
  stage: test
  script:
    - pip install flake8
    - flake8 myproject

deploy:
  stage: deploy
  script:
    - pip install twine
    - python setup.py sdist bdist_wheel
    - twine upload dist/*
  only:
    - tags

This GitLab CI pipeline:

  1. Uses a Python 3.9 base image
  2. Defines test and deploy stages
  3. Runs tests with coverage reports
  4. Performs linting
  5. Deploys the package to PyPI when a tag is pushed

3. Jenkins​

For teams with existing Jenkins infrastructure, you can create a Jenkinsfile for your Python project:

pipeline {
    agent {
        docker {
            image 'python:3.9'
        }
    }
    stages {
        stage('Setup') {
            steps {
                sh 'pip install -r requirements.txt'
                sh 'pip install pytest flake8'
            }
        }
        stage('Lint') {
            steps {
                sh 'flake8 .'
            }
        }
        stage('Test') {
            steps {
                sh 'pytest'
            }
            post {
                always {
                    junit 'test-reports/*.xml'
                }
            }
        }
        stage('Deploy') {
            when {
                expression { env.TAG_NAME != null }
            }
            steps {
                sh 'pip install twine'
                sh 'python setup.py sdist bdist_wheel'
                sh 'twine upload dist/*'
            }
        }
    }
}

Building a Complete CI/CD Pipeline for Python​

Now let's create a more comprehensive GitHub Actions workflow that includes testing, building, and deploying a Python application:

Example: Full Python Web App CI/CD Pipeline​

name: Python Web App CI/CD

on:
  push:
    branches: [ main ]
    tags:
      - 'v*'
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:13
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test_db
        ports:
          - 5432:5432
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.9'
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install pytest pytest-cov flake8
    
    - name: Lint with flake8
      run: |
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 . --count --max-complexity=10 --max-line-length=127 --statistics
    
    - name: Test with pytest
      env:
        DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_db
      run: |
        pytest --cov=app --cov-report=xml
    
    - name: Upload coverage report
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.xml
  
  build:
    needs: test
    runs-on: ubuntu-latest
    if: success() && (github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')))
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2
    
    - name: Login to Docker Hub
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}
    
    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v4
      with:
        images: yourusername/python-app
    
    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true

        labels: ${{ steps.meta.outputs.labels }}
  
  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: success() && startsWith(github.ref, 'refs/tags/v')
    
    steps:
    - name: Deploy to production
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.SERVER_HOST }}
        username: ${{ secrets.SERVER_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd /app
          docker pull yourusername/python-app:latest
          docker-compose down
          docker-compose up -d

This comprehensive workflow:

  1. Sets up a PostgreSQL service for integration testing
  2. Runs linting and testing with coverage reports
  3. Builds and pushes a Docker image when tests pass
  4. Deploys to a production server when a version tag is pushed

Best Practices for Python CI/CD​

1. Organize Your Python Project for CI/CD​

A well-structured Python project makes CI/CD implementation easier:

my_project/
├── .github/workflows/        # GitHub Actions workflows
│   └── python-ci.yml
├── src/                      # Project source code
│   └── my_project/
│       ├── __init__.py
│       └── app.py
├── tests/                    # Test directory
│   ├── __init__.py
│   └── test_app.py
├── .gitignore
├── pyproject.toml            # Modern Python project config
├── requirements.txt          # Dependencies
└── README.md

2. Manage Dependencies Properly​

Use dependency pinning to ensure consistent builds:

# requirements.txt
flask==2.2.3
SQLAlchemy==2.0.5
pytest==7.3.1

For development dependencies, consider using a separate file:

# requirements-dev.txt
-r requirements.txt
pytest==7.3.1
pytest-cov==4.1.0
flake8==6.0.0

3. Test Configuration​

Create a pytest.ini file to configure your test environment:

[pytest]
testpaths = tests
python_files = test_*.py
python_functions = test_*
python_classes = Test*
addopts = --cov=src --cov-report=term-missing

4. Environment Variables and Secrets​

Store sensitive information as secrets in your CI/CD platform:

# GitHub Actions example
env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
  API_KEY: ${{ secrets.API_KEY }}

Example: Real-World Python Web Application CI/CD​

Let's create a complete CI/CD pipeline for a Flask web application:

Project Setup​

flask_app/
├── .github/workflows/
│   └── deploy.yml
├── app/
│   ├── __init__.py
│   ├── routes.py
│   ├── models.py
│   └── templates/
├── tests/
│   ├── test_routes.py
│   └── test_models.py
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── wsgi.py

Sample Flask Application (app/__init__.py)​

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

def create_app(config=None):
    app = Flask(__name__)
    app.config.from_mapping(
        SECRET_KEY='dev',
        SQLALCHEMY_DATABASE_URI='sqlite:///app.db',
        SQLALCHEMY_TRACK_MODIFICATIONS=False,
    )
    
    if config:
        app.config.update(config)
    
    db.init_app(app)
    
    from . import routes
    app.register_blueprint(routes.bp)
    
    return app

Dockerfile​

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 5000

CMD ["gunicorn", "--bind", "0.0.0.0:5000", "wsgi:app"]

CI/CD Workflow (.github/workflows/deploy.yml)​

name: Flask App CI/CD

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.9'
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install pytest pytest-cov
    
    - name: Run tests
      run: |
        pytest --cov=app tests/
    
  deploy:
    needs: test
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    
    steps:
    - name: Deploy to Heroku
      uses: akhileshns/heroku-[email protected]
      with:
        heroku_api_key: ${{ secrets.HEROKU_API_KEY }}
        heroku_app_name: ${{ secrets.HEROKU_APP_NAME }}
        heroku_email: ${{ secrets.HEROKU_EMAIL }}
        usedocker: true

Advanced CI/CD Techniques for Python​

1. Matrix Testing​

Test across multiple Python versions and operating systems:

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: [3.8, 3.9, 3.10, 3.11]
    
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      # ...rest of test job

2. Caching Dependencies​

Speed up CI builds by caching pip packages:

- name: Cache pip packages
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
    restore-keys: |
      ${{ runner.os }}-pip-

3. Testing Database Migrations​

Ensure your database migrations work correctly:

- name: Test migrations
  run: |
    flask db upgrade
    flask db downgrade
    flask db upgrade

Summary​

Integrating CI/CD into your Python projects significantly improves code quality and deployment efficiency. In this tutorial, you've learned:

  • The fundamentals of CI/CD for Python applications
  • How to set up CI/CD pipelines using GitHub Actions, GitLab CI, and Jenkins
  • Best practices for organizing Python projects for CI/CD
  • How to create comprehensive workflows for testing, building, and deploying Python applications

By implementing CI/CD in your Python projects, you'll catch bugs earlier, ensure consistent code quality, and streamline your deployment process.

Additional Resources​

Exercises​

  1. Set up a basic CI workflow for an existing Python project using GitHub Actions.
  2. Modify the workflow to test against multiple Python versions.
  3. Add a deployment step to your CI/CD pipeline that deploys your application to a cloud provider (e.g., Heroku, AWS, or Google Cloud).
  4. Implement dependency caching to speed up your CI process.
  5. Configure a workflow that publishes your Python package to PyPI when you create a new release.

By completing these exercises, you'll gain practical experience with Python CI/CD integration that you can apply to your own projects.

💡 Found a typo or mistake? Click "Edit this page" to suggest a correction. Your feedback is greatly appreciated!