Automatic project deployment and management
Introduction
This page describes a mechanism that facilitates working with versioned CartoWeb(s) and deploying to remote environments.
Making your project deployable
To make your project deployable, configuration elements which depend on where the project is run must be handled specifically.
For instance, if your client.ini contains:
cartoclientBaseUrl = http://example1.com/cartoweb
The project will run fine on the server example1 in a specific path, but it won't run any more once deployed on the server example2.com or if it is deployed on another path on this server, because the cartoclientBaseUrl is not the same.
That's why a mechanism exists to handle this situation. This mechanism works by generating configuration files, and replaces server/path dependent configuration elements using variables.
.in files
Any configuration file that contains location-specific variables should be renamed to a file with a ".in" suffix, and all those server/path dependent variables must use template variables which will be replaced dynamically.
As an example, let's say you have the following client.ini file in your project:
cartoclientBaseUrl = http://example1.com/cartoweb profile = production
You must rename it to client.ini.in and change its content to:
cartoclientBaseUrl = "@CARTOCLIENT_BASE_URL@" profile = @PROFILE@
During deployment, the client.ini.in file will generate the client.ini file, replacing @CARTOCLIENT_BASE_URL@ and @PROFILE@ with the values contained in a project deployment file (see next paragraph) that is specific to the local path.
- Note 1 : this mechanism can be used for mapfiles too; this comes particularly handy for PostGIS layers :
[...] CONNECTIONTYPE postgis CONNECTION "user=@DB_USER@ password=@DB_PASSWD@ dbname=@DB_NAME@ host=@DB_HOST@ port=@DB_PORT@" [...]
- Note 2 : only the .in file must be saved in CVS repository.
Project deployment files
The project deployment file is a configuration file which contains the values of the variables which have to be replaced during the generation of configuration files from the <.in> files. It is located in the "deployment" directory of your project, and its name contains the current server name and the directory where cartoweb is located.
To see the exact name of the project deployment file you have to use, see "The first fetch" section (for instance, if you want to deploy on hostname "example1.com" in directory "/home/alice/cartoweb3", the file name will be config_example1.com_home_alice_cartoweb3.properties in the deployment directory of your project).
A deployment configuration file contains the values of the variables, according to the following syntax:
PROFILE = production CARTOCLIENT_BASE_URL = "http://example1.com/~alice/cartoweb3"
If you want to deploy the same project elsewhere, you'll need another deployment configuration file for this second location.
Configuration of the automatic fetch & deploy
Concepts
The fetch & deploy architecture can suppport a set of multiple cartowebs, possibly retrieved with different cvs versions, called cartoweb instances.
Each instance can contain several cartoweb projects
The directory layout is the following
- Makefile: the file containing the deploy configuration
- cartowebs : contains the instances; each instance contains a full-featured cartoweb and all projects which are attached to this instance.
- deploy : contains the deploy scripts; do not modifiy these.
- htdocs :
for convenience, a symbolic link is created in this directory for each project, pointing to the htdocs of the instance containing the project. Thus, the webserver has to be given access to this directory, and everything will be available (of course, this can be tuned depending on the desired web server configuration).
Configuration
The Makefile contains the following definitions blocks:
instances
projects
hostnames
The sample file cartoweb3/scripts/deploy/Makefile.sample contains documentation on these different variables. It should be used as a starting point.
Fetching
The first fetch
Once you have your Makefile setup and your project available on CVS, you can run the initial instance fetch, so that you can see the deployment filename to use.
Type the command inside the directory containing the Makefile, by replacing YOUR_INSTANCE by the instance name you want to use:
make fetch_instance/YOUR_INSTANCE
This will display the following error:
Error message: Can't find project config file. It should be in one of the path (tried in order): projects/example1/deployment/config_example1.com_home_alice_public_html_cartowebs_example1_cartoweb3.properties projects/example1/deployment/config_example1.com_home_alice_public_html_cartowebs_fpds.properties projects/example1/deployment/config_example1.com.properties projects/example1/deployment/config.properties
This means that you have to create a file named "config_example1.com_home_alice_public_html_cartowebs_example1_cartoweb3.properties" in the deployment directory of your project containing server/path dependent variables. See the previous section for an example.
Fetching commands
Once this file is configured, the following commands can be used to manage deployment: These commands have to be used from the directory containing the Makefile:
make fetch_all_instances
- Fetch all instances and projects
make fetch_instance/INSTANCE
- Fetch the instance named INSTANCE with all its projects
make fetch_project/PROJECT
- Fetch the project called PROJECT
Deploying
Here is a situation where you need an automatic deploy process. Say you have three servers :
Server 0
- Development server
- cartoweb cvs access + http access for archive fetching
- project cvs access
-> deployment to server 1
Server 1
- Validation server
- No cvs access
-> deployment to server 2
Server 2
- Production server
Makefile configuration
In the hostname part of the Makefile, you have to define :
server0_TARGET_HOST := server1 server1_TARGET_HOST := server2
Deploying commands
make deploy_all_instances
- Deploys all instances to the next server
make deploy_instance/INSTANCE
- Deploys instance named INSTANCE to the next server
In this configuration, you have to be on server1 to deploy to server2.
Example
Take the following file as a starting point:
http://cvs.camptocamp.com/cgi-bin/viewcvs.cgi/*checkout*/cartoweb3/scripts/deploy/Makefile.sample?rev=HEAD
cvs -d :pserver:anonymous@dev.camptocamp.com:/var/lib/cvs/public co cartoweb3/scripts/deploy/Makefile.sample mv cartoweb3/scripts/deploy/Makefile.sample Makefile rm -rf cartoweb3/
Edit the Makefile, replacing the instance definition and project definitions with (if necessary change USERNAME to your username):
[...] # ################################### # Instances definitions elk_REVISION := "-D 2020-01-26" # This MUST contain all instance names ALL_INSTANCES := elk # Project definitions # ################################### default_CVSROOT := :pserver:USERNAME@source.c2c:/var/lib/cvs/projects/cw3 elk_INSTANCE := elk foobar_INSTANCE := elk foobar_CVSROOT := :pserver:bob@example.com./var/lib/cvs/ # This MUST contain all project names ALL_PROJECTS := elk foobar [...]
As you can see, several projects can be defined. If you want to add a new project, add at least the <project>_INSTANCE line and update the ALL_PROJECTS variable. Other variables can optionally be given, like the CVSROOT of the project, using the <project>_CVSROOT variable.
Now you should be able to type:
make fetch_instance/elk (or make fetch_instance/elk NO_CONFIRM=1 to avoid confirmation message)
You may see the error message:
Error message: Can't find project config file. It should be in one of the path (tried in order): projects/elk/deployment/config_sarge_home_sypasche_public_html_deploy_cartowebs_elk_cartoweb3.properties projects/elk/deployment/config_sarge_home_sypasche_public_html_deploy_cartowebs_elk.properties projects/elk/deployment/config_sarge.properties projects/elk/deployment/config.properties
Now create the deployment configuration file:
cd cartowebs/elk/cartoweb3/projects/elk/deployment/ echo 'CARTOCLIENT_BASE_URL = "http://zuort.c2c/~sypasche/deploy/htdocs/elk"' > config_sarge_home_sypasche_public_html_deploy_cartowebs_elk_cartoweb3.properties cvs ci -m added config_sarge_home_sypasche_public_html_deploy_cartowebs_elk_cartoweb3.properties cd ../../../../../..
And relaunch the deploy:
make fetch_instance/elk NO_CONFIRM=1
And it will automatically:
- fetch deploy scripts
- fetch cartoweb instance with the given revision
- fetch project inside instance
- launch cw3setup install scripts
- create link into htdocs
Now the application should be available in the htdocs/elk directory.
To deploy to the next server with rsync:
First configure the hostname definition section correctly. Then you can type the following command:
make deploy_instance/elk
Deploy update
If you see the message: Warning: A new version of the deploy script is available. Press enter to continue, or control-c to abort so that you can update
You must first update the deploy version. See the Changes section below, and pick a date (or a CVS Tag). Then edit your Makefile to update the DEPLOY_REVISION variable. For instance:
DEPLOY_REVISION := -D 2020-02-20
Then you need to delete the deploy directory
rm -rf deploy
The script will automatically fetch the latest deploy directory again from CVS. You might then need to update your Makefile if you see the message mentionned on the next section.
Makefile update
If you recieve a message like Your Makefile version X is not compatible with the new deploy version Y, see the following migration guide, and update your Makefile and COMPAT_VERSION variable.
The COMPAT_VERSION variable can be found at the top of the Makefile:
COMPAT_VERSION := 0
Changes
2020-01-26: First version
Migration guide
Version 0 -> version 1 Migration:
Nothing yet