Deploying Django with Salt - Now with PostgreSQL!

  • Sun 21 April 2013

  • Continuing this post I'm going to explain how to install PostgreSQL for use with Django using Salt Stack.

    What's Necessary?

    • Install PostgreSQL
    • Install psycopg2 and it's dependancies
    • Setup PostgreSQL permissions so there is a database and the user we specify has access to said database
    • Tell Django to use PostgreSQL rather than sqlite3

    Install PostgreSQL

    .
    ├── django
    │   ├── init.sls
    │   └── requirements.txt
    ├── minion.conf
    ├── README.rst
    ├── requirements
    │   └── init.sls
    ├── top.sls
    └── Vagrantfile
    

    Above is how we left things. I'm going to create a postgresql directory in my project's root and create an init.sls file in the postgresql directory.

    In the init.sls file I'm going to tell Salt to install postgresql like so:

    postgresql:
    pkg:
        - name: postgresql-9.1
        - installed
    service.running:
        - enabled: True
        - watch:
            - file: /etc/postgresql/9.1/main/pg_hba.conf
    

    In the above, I'm simply saying install the package postgresql-9.1. I'm also telling it to ensure the service is running, and with watch if the file /etc/postgresql/9.1/main/pg_hba.conf changes, restart the service.

    For Django be able to create/connect to the database we define in our settings.py file we have to edit the above mentioned /etc/postgresql/9.1/mina/pg_hba.conf file to do so.

    # This file controls: which hosts are allowed to connect, how clients
    # are authenticated, which PostgreSQL user names they can use, which
    # databases they can access.  Records take one of these forms:
    #
    # local      DATABASE  USER  METHOD  [OPTIONS]
      local   djangodb    django  md5
    

    In the above example, we're allowing the user django to connect to the database djangodb on the local sytem.

    We want to manage this file, in the event that things change, we have history of those changes. We can manage it with Salt like so:

    pg_hba.conf:
    file.managed:
        - name: /etc/postgresql/9.1/main/pg_hba.conf
        - source: salt://postgresql/pg_hba.conf
        - user: postgres
        - group: postgres
        - mode: 644
        - require:
            - pkg: postgresql-9.1
    

    In the above code block, we're simply saying, manage the file pg_hba.conf with Salt. The source directive says the file is located in the $ROOT/postgresql directory. We want the file to be owned by postgres:postgres and we're setting the permissions to 644. BUT, this file REQUIRES the package postgresql-9.1 to be installed before we manage the file. If we try and manage this file before it actually exists on the system, it will not be created. This also ensures that the package exists, likely running, and when we manage the file, Salt will restart the service, loading the new settings.

    OK...so that takes care of installing PostgreSQL, it also checks off the "allowing Django to connect to the DB we specified". Next!

    Install psycopg2 and it's dependancies

    psycopg2 is required by Django to use PostgreSQL database. To install, we simply add it to our requirements.txt

    Django==1.5
    psycopg2==2.4.5
    

    psycopg2 does require the system package python-dev and libpq-dev so we will list that package in our requirements/init.sls file

    packages:
        pkg.installed:
            names:
                - python-virtualenv
                - python-dev
                - libpq-dev
    

    A gotcha!

    We need to ensure that the package python-dev and libpq-dev is installed before we try and install psycopg2 or bad things happen. Order of operations if you will.

    So we do that like so (in django/init.sls):

    /home/vagrant/learning-salt/venv:
        virtualenv.managed:
            - no_site_packages: True
            - runas: vagrant
            - requirements: salt://django/requirements.txt
            - require:
                - pkg: python-dev
                - pkg: python-virtualenv
                - pkg: libpq-dev
    

    Requiring the package python-dev ensures that python-dev is installed before we try and do a pip install of our requirements.txt, subsequently installing psycopg2.

    That takes care of that. Next!

    Tell Django to use PostgreSQL rather than sqlite3

    In our settings.py file, we can declare what DB to use, user, password, port. Rather than having these values static, we can generate them at the time of deployment with Salt. Salt uses something called pillars

    In my project root, I create a directory called pillar and in that directory I create a file called top.sls as well as settings.sls.

    These files look like:

    ## top.sls
    base:
        '*':
            - settings
    
    ## settings.sls
    dbengine: django.db.backends.postgresql_psycopg2
    dbname: djangodb
    dbuser: django
    dbpassword: password
    dbhost: localhost
    dbport: 5432
    secret_key: secret_key
    

    Now we need to coorelate these to pieces in our settings.py like so:

    ########## DATABASE CONFIGURATION
    # See: https://docs.djangoproject.com/en/dev/ref/settings/#databases
    DATABASES = {
            'default': {
                        'ENGINE': '{{ pillar["dbengine"] }}',
                        'NAME': '{{ pillar["dbname"] }}',
                        'USER': '{{ pillar["dbuser"] }}',
                        'PASSWORD': '{{ pillar["dbpassword"] }}',
                        'HOST': '{{ pillar["dbhost"] }}',
                        'PORT': '{{ pillar["dbport"] }}',
                    }
    }
    ########## END DATABASE CONFIGURATION
    

    To ensure that the settings go into our settings.py file, we can manage the file as we have the previously mentioned like so:

    base.py:
        file.managed:
            - name: /home/vagrant/learning-salt/icecream/icecream/icecream/settings/base.py
            - source: salt://django/base.py
            - template: jinja
            - require:
                - postgres_user: djangouser
    

    There is still one missing piece. We need to tell Salt to create the user and database. Here's how:

    djangouser:
        postgres_user.present:
            - name: {{ pillar['dbuser'] }}
            - password: {{ pillar['dbpassword'] }}
            - runas: postgres
            - require:
                - service: postgresql
    
    djangodb:
        postgres_database.present:
            - name: {{ pillar['dbname'] }}
            - encoding: UTF8
            - lc_ctype: en_US.UTF8
            - lc_collate: en_US.UTF8
            - template: template0
            - owner: {{ pillar['dbuser'] }}
            - runas: postgres
            - require:
                - postgres_user: djangouser
    

    In the above example, we're telling it to create the djangouser. It is using the username and password as defined in our pillar with {{ pillar['dbuser'] }} and {{ pillar['dbpassword'] }} respectively. We're running the commands as the postgres user, and we're requiring that the postgresql service exists, else wise there is no where to create the user!

    Like djangouser we're creating the database djangodb similarly, and as a requirement, the postgres_user: djangouser needs to exist before we create the database and define its owner.

    The only thing left is to put our postgresql in our top.sls like so:

    base:
        '*':
            - requirements
            - django
            - postgresql
    

    Like mentioned in the earlier post, to run this, vagrant ssh into the host.

    Then cd learning salt && source venv/bin/activate then cd icecream and python manage.py runserver 0.0.0.0.

    All set!

    Next time, I'll likely show how to set up Nginx and uWSGI, because I'm tired of manually running the server.

    Github: https://github.com/esacteksab/learning-salt

    In case you're wondering, here are the successful parts of Salt's output:

    ----------
     State: - pkg
     Name:      python-dev
     Function:  installed
         Result:    True
         Comment:   The following package(s) were installed/updated: python-dev.
         Changes:   libpython2.7: {'new': '2.7.3-0ubuntu3.1', 'old': ''}
                    python2.7: {'new': '2.7.3-0ubuntu3.1', 'old': '2.7.3-0ubuntu3'}
                    python2.7-minimal: {'new': '2.7.3-0ubuntu3.1', 'old': '2.7.3-0ubuntu3'}
                    python2.7-dev: {'new': '2.7.3-0ubuntu3.1', 'old': ''}
                    libexpat1-dev: {'new': '2.0.1-7.2ubuntu1.1', 'old': ''}
                    libexpat-dev: {'new': '1', 'old': ''}
                    libexpat1: {'new': '2.0.1-7.2ubuntu1.1', 'old': '2.0.1-7.2ubuntu1'}
                    python-dev: {'new': '2.7.3-0ubuntu2', 'old': ''}
    
     ----------
     State: - pkg
     Name:      python-virtualenv
     Function:  installed
         Result:    True
         Comment:   The following package(s) were installed/updated: python-virtualenv.
         Changes:   python-pip: {'new': '1.0-1build1', 'old': ''}
                    python-virtualenv: {'new': '1.7.1.2-1', 'old': ''}
                    python-setuptools: {'new': '0.6.24-1ubuntu1', 'old': ''}
                    python-distribute: {'new': '1', 'old': ''}
    
     ----------
     State: - pkg
     Name:      libpq-dev
     Function:  installed
         Result:    True
         Comment:   The following package(s) were installed/updated: libpq-dev.
         Changes:   libpq5: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
                    libkadm5srv-mit8: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    krb5-multidev: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    libk5crypto3: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': '1.10+dfsg~beta1-2'}
                    libkdb5-6: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    libkrb5-dev: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    libkadm5clnt-mit8: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    libkrb5-3: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': '1.10+dfsg~beta1-2'}
                    libpq-dev: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
                    comerr-dev: {'new': '2.1-1.42-1ubuntu2', 'old': ''}
                    libgssrpc4: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': ''}
                    libkrb5support0: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': '1.10+dfsg~beta1-2'}
                    libgssapi-krb5-2: {'new': '1.10+dfsg~beta1-2ubuntu0.3', 'old': '1.10+dfsg~beta1-2'}
    
     ----------
     State: - virtualenv
     Name:      /home/vagrant/learning-salt/venv
     Function:  managed
         Result:    True
         Comment:   Created new virtualenv
         Changes:   new: Python 2.7.3
                    packages: {'new': ['Django==1.5', 'psycopg2==2.4.5'], 'old': ''}
    
     ----------
     State: - pkg
     Name:      postgresql-9.1
     Function:  installed
         Result:    True
         Comment:   The following package(s) were installed/updated: postgresql-9.1.
         Changes:   postgresql-client-common: {'new': '129ubuntu1', 'old': ''}
                    postgresql-9.1: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
                    ssl-cert: {'new': '1.0.28ubuntu0.1', 'old': ''}
                    postgresql-client-9.1: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
                    postgresql-common: {'new': '129ubuntu1', 'old': ''}
    

    The above show that the packages python-dev, python-virtualenv, libpq-dev, and postgresql-9.1 were successfull installed. It also shows that the virtualenv /home/vagrant/learning-salt/venv was created and the packages Django==1.5 and psycopg2==2.4.5 were successfully installed.

    The block below shows that the file pg_hba.conf was successfully updated

    ----------
    State: - file
    Name:      /etc/postgresql/9.1/main/pg_hba.conf
    Function:  managed
        Result:    True
        Comment:   File /etc/postgresql/9.1/main/pg_hba.conf updated
        Changes:   diff: ---
    +++
    @@ -10,6 +10,7 @@
    # databases they can access.  Records take one of these forms:
    #
    # local      DATABASE  USER  METHOD  [OPTIONS]
    +local  djangodb    django  md5
    # host
    [default]   DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
    # hostssl    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
    # hostnossl  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
    @@ -89,9 +90,9 @@
    # "local" is for Unix domain socket connections only
    local   all             all                                     peer
    # IPv4 local connections:
    -host    all             all             127.0.0.1/32            md5
    +host    all             all             127.0.0.1/32            trust
    # IPv6 local connections:
    -host    all             all             ::1/128                 md5
    +host    all             all             ::1/128                 trust
    # Allow replication connections from localhost, by a user with the
    # replication privilege.
    #local   replication     postgres                                peer
    
                    group: postgres
                    user: postgres
    

    A very truncated output displaying that the packages from our Two Scoops of Django was successful:

    Successfully installed Django bpython django-braces django-model-utils
    logutils South coverage django-discover-runner django-debug-toolbar Sphinx
    pygments Jinja2 docutils
    Cleaning up...
    Syncing...
    Creating tables ...
    Creating table auth_permission
    Creating table auth_group_permissions
    Creating table auth_group
    Creating table auth_user_groups
    Creating table auth_user_user_permissions
    Creating table auth_user
    Creating table django_content_type
    Creating table django_session
    Creating table django_site
    Creating table django_admin_log
    Creating table south_migrationhistory
    Installing custom SQL ...
    Installing indexes ...
    Installed 0 object(s) from 0 fixture(s)
    
    Synced:
     > django.contrib.auth
      > django.contrib.contenttypes
       > django.contrib.sessions
        > django.contrib.sites
         > django.contrib.messages
          > django.contrib.staticfiles
           > django.contrib.admin
            > south
             > debug_toolbar
    
             Not synced (use migrations):
              -
                (use ./manage.py migrate to migrate these)
    

    Here shows that PostgreSQL was successfully installed and the user django was successfull created:

    ----------
    State: - service
    Name:      postgresql
    Function:  running
        Result:    True
        Comment:   Service restarted
        Changes:   postgresql: True
    
    ----------
    State: - postgres_user
    Name:      django
    Function:  present
        Result:    True
        Comment:   The user django has been created
        Changes:   django: Present
    

    In the examples below, we see that the package postgresql-9.1-dbg, postgresql-server-dev-9.1 were successfully installed and that the database djangodb was successfully created

    ----------
    State: - pkg
    Name:      postgresql-9.1-dbg
    Function:  installed
        Result:    True
        Comment:   The following package(s) were installed/updated: postgresql-9.1-dbg.
        Changes:   postgresql-9.1-dbg: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
    
    ----------
    State: - pkg
    Name:      postgresql-server-dev-9.1
    Function:  installed
        Result:    True
        Comment:   T
    [default] he following package(s) were installed/updated: postgresql-server-dev-9.1.
        Changes:   postgresql-server-dev-9.1: {'new': '9.1.9-0ubuntu12.04', 'old': ''}
    
    ----------
    State: - postgres_database
    Name:      djangodb
    Function:  present
        Result:    True
        Comment:   The database djangodb has been created
        Changes:   djangodb: Present
    

    Comments !

    social