Skip to content

SETUP.org

Latest commit

 

History

History
247 lines (189 loc) · 10.8 KB

SETUP.org

File metadata and controls

247 lines (189 loc) · 10.8 KB

Cornerwise: Setup

Running the Server

We’re using Docker, because it ensures that we’re all working with the same environment and provides near parity between development and production. You can think of Docker as a tool for managing lightweight machines and wiring them together.

If you follow the instructions below, you will have the necessary dependencies, including Postgres w/GIS extensions and Django 1.10.

Docker Installation

OS X and Windows

You have two options:

  • Docker for Windows / Docker for Mac

    Macs (>=2012) running 10.10.3 or higher and Windows PCs running Windows 10 Pro, Enterprise, or Education edition can choose to run Docker Native.

  • Install Docker Toolbox

    The Toolbox includes VirtualBox, Docker Machine, and Docker Compose. On Windows, it also includes MinGW (w/a Unix-like shell), Git, and some other handy tools. Please note: using Docker Toolbox requires some extra steps, because instead of running application containers directly, it launches them inside a VM. You should make sure you’re using the Docker Quickstart Terminal whenever you want to interact with Docker.

Linux

  • Install Docker Engine
  • Install Docker Compose

    Compose is a tool built on top of Docker that allows us to organize the application into component services and to wire them together. It reads the contents of docker-compose.yml (possibly merging in other files like docker-compose.override.yml) and constructs containers as needed.

Startup

Note: On Linux, you’ll typically have to run all docker commands, including docker-compose, with sudo. This is due to the fact that the Docker daemon must run as root, and the client communicates with it over a socket owned by root. You can add your user to the docker group to avoid having to type sudo.

  1. (Docker Toolbox) Launch the Docker Quickstart Terminal. This will provision a new default Docker Machine if one does not already exist and set up some environment variables for communicating with that machine. See instructions below if you prefer to do this step manually, in another terminal, or using an alternative VM driver.
  2. cd path/to/cornerwise in your terminal
  3. docker-compose up
    • When you first run up, Compose will retrieve some images from Docker Hub. It is normal for this process to take a few minutes.
  4. Once the images have been built and the containers have started, you should see something like:

    cornerwise_1 | Starting development server at http://0.0.0.0:4000/

    Keep this window open; closing it will stop the server.

  5. Open a web browser window and browser to the appropriate URL:
    • (Docker Toolbox) Determine the IP address of the machine by running docker-machine ip default. The address will have the form: http://*ip-address*:4000/. (On Mac, you can run open http://$(docker-machine ip default):4000 or set up port forwarding by following the instructions below.)
    • (Linux or Docker Native) Open a browser to http://localhost:4000/.
  6. In the original window, press Ctrl-C to shut down cornerwise.

Docker Toolbox Without Quickstart Terminal (optional)

If you’re using Docker Toolbox and prefer to use a terminal other than the included Docker Quickstart Terminal, you will need to run a couple extra commands.

  • docker-machine create -d virtualbox default Run this the first time. It creates a new VM with the name default using your specified driver (VirtualBox in this example, but there are many other supported drivers)
  • eval $(docker-machine env default) Sets up some environment variables for communicating with a running machine named default. You must run this each time you open a new terminal session.

Port Forwarding (OS X + Docker Toolbox) (optional)

If you’re using Docker Toolbox + VirtualBox and would like to be able to access the site at http://localhost:4000:

  • VBoxManage controlvm default natpf1 cornerwise,tcp,127.0.0.1,4000,,4000
    default
    name of the Docker Machine
    4000 (first occurrence)
    host port
    4000 (second occurrence)
    VM port

Interacting with Containers

  • (Docker Toolbox) eval $(docker-machine env default) or run Quickstart Terminal
  • cd path/to/cornerwise
  • docker-compose exec cornerwise bash

API Credentials

Many of the features of Cornerwise require the application to communicate with third-party APIs. To get them working, you’ll need to set up accounts with the appropriate vendors. All of them offer services that are free at the volume we deal with (certainly, more than needed for the purposes of local development.)

At the moment, the only settings that are absolutely required are GOOGLE_BROWSER_API_KEY and either GOOGLE_API_KEY or the ArcGIS id and secret. If you use GOOGLE_API_KEY and omit ArcGIS, change GEOCODER to “google”. Things will run more or less normally without the other credentials, with certain features disabled.

Setup

Rename server/cornerwise/local_settings.example.py to local_settings.py and replace the values there.

Certain Google applications require default credentials. Rename the credentials json file to google_credentials.json and move it to server/cornerwise.

Getting Data

Proposals

When you first run Cornerwise, there will be no data in the database. Every night at midnight, it runs its available importers to find new data from known sites.

With cornerwise running, open a new window and enter:

docker-compose exec cornerwise ./manage.py shell
>>> import datetime, proposal
>>> proposal.tasks.pull_updates(datetime.datetime(2016, 10, 10).timestamp())
# Or, with celery running, dispatch an asychronous job:
>>> proposal.tasks.pull_updates.delay(datetime.datetime(2016, 10, 10).timestamp())

Parcels

Parcels, for our purposes, are shapes representing an area of real property (real estate) with some metadata attached. They are stored in shapefiles. The shapefiles for Somerville are currently included in the repository for convenience. Some features of Cornerwise require parcels to be available. For example, selecting or hovering over a proposal marker will show the shape of the affected parcel if one is found. We also use metadata about the square footage in some places.

To import the Somerville parcels, run:

docker-compose exec cornerwise ./manage.py addparcels somervillema

(You can run ./manage.py help addparcels to view additional options.)

The script will import all the Somerville parcel shapes into the Cornerwise database running in your postgis container. There are over 30,000 parcels in Somerville alone, so it can take a while for the process to complete.

Administrator Interface

Setup

You’ll first need to create a superuser.

With the server running in Docker Compose, cd to the repository root and run:

docker-compose exec cornerwise ./manage.py createsuperuser

You will be prompted to enter a username, email (optional), and password. See ./manage.py createsuperuser -h for full usage details, e.g., if you would like to set the superuser’s email address. If SendGrid is configured, superusers with email addresses receive system notifications, such as messages submitted through the Contact form.

Now navigate to http://localhost:4000/admin and enter your credentials.

Creating new Scrapers

Cornerwise uses a simple scraper interface to import new cases, events, and projects. Every day, it will send a GET request to a configured URL. The scraper should return a JSON result conforming to the scraper schema.

Troubleshooting

Broken Icons

The license agreement for the layer icons forbids us from redistributing them in the repository, but we are allowed to share them individually. Contact an existing team member to get the icons, then copy them to client/css/font.

Production

Differences

Deploying to production differs in a few ways:

  1. The webserver, not Django, is responsible for serving static assets. This should be configured in your webserver.
  2. Changes to application code are not automatically loaded, since the contents of server/ are copied when the image is built, not when the container starts. You must run docker-compose build first.
  3. It uses gunicorn instead of the built-in development webserver and serves from port 3000 instead of 4000.
  4. Configuration that in development uses local_settings.py uses environment variables in production . When deploying, ensure that there is a prod.env file in docker-support. It should not be in git.

Running Production Mode Locally

The docker-compose.nginx.yml file contains the necessary configuration to run Cornerwise in production mode behind a reverse proxy, mirroring the production setup. To run it:

  1. Copy docker-support/nginx.env.example to docker-support/nginx.env, replacing the variables with appropriate values. (The application will fail gracefully if you omit most settings, but you will need to have a GOOGLE_BROWSER_API_KEY and a geocoder at minimum.)
  2. Run docker-compose -f docker-compose.nginx.yml up to start running the application.
  3. The application will be running at http://localhost:3000.

Starting Fresh

To start over with a clean database, cd to the the project directory and run docker-compose down -v. This will shut down the running containers and delete them. It will also delete all of the named volumes and any data they may contain.

Uninstalling

Stop and Remove Containers

  • In the cornerwise directory, run docker-compose down -v

Remove the image:

docker rmi bdsand/cornerwise

Or, if you’d like to remove all the images we use:

docker rmi bdsand/cornerwise redis mdillion/postgis nginx