§Dynamic proxy configuration
The dynamic proxy configuration feature allows operations to provide a customized proxy configuration that would be updated automatically according to the applications that are running within the ConductR cluster.
ConductR provides a default configuration that can be overridden should the need arise. For example, HAProxy can provide HTTP Basic Auth for staging deployments enabling protected preview prior to promoting to production. This is accomplished by deploying a template haproxy.cfg
file as part of the configuration of the HAProxy bundle.
This document is a guide and reference for using dynamic HAProxy configuration templates.
§Assumptions
ConductR uses HAProxy as its proxying solution. It is assumed the reader has some familiarity with HAProxy installation and configuration.
ConductR makes use of Handlebars as the templating solution to generate the customised HAProxy configuration. It is assumed the reader has some degree familiarity in working with other templating solutions.
§Requirement
ConductR makes the assumption that an endpoint must be uniquely identified by its name given a particular system and system version.
The endpoints that failed to satisfy this requirement will be considered as duplicate, and the duplication is assumed to be a result of configuration error. Because of this, duplicated endpoints will not be exposed via proxy configuration.
§Creating and deploying custom HAProxy configuration
It is possible to provide custom HAProxy configuration by supplying configuration overrides to the ConductR HAProxy bundle.
Follow these steps to provide custom HAProxy configuration to ConductR HAProxy running on a clustered environment (i.e. non-sandbox). To provide a custom configuration for the Sandbox go to the chapter Creating and deploying custom HAProxy configuration for Sandbox.
ConductR CLI is required to perform the following steps, and as such it needs to be installed beforehand.
These are the steps to deploying custom HAProxy configuration.
§Setup
Create a directory where the custom HAProxy configuration will be placed, e.g.
mkdir -p /tmp/custom-haproxy-conf
§Populate the custom HAProxy configuration template
Create the custom HAProxy template, e.g.
touch /tmp/custom-haproxy-conf/haproxy-override.cfg
The file can be given any name as long as it’s within the directory created from the previous step.
In this example, we’ll populate the file with the following custom template.
# HAProxy Specific Configuration
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# We like statistics
stats socket {{haproxyHost}}:8999 level user
stats timeout 2m #Wait up to 2 minutes for input
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
# Custom Endpoint Configuration
{{#eachAcls bundles defaultHttpPort=9443}}
{{#ifAcl 'public-site' '1' 'main'}}
frontend my_www_frontend
bind {{haproxyHost}}:80
# Uncomment the following line if SSL is used
# bind {{haproxyHost}}:443 ssl crt {{overrideConfigDir}}/test-cert.pem
mode http
acl my_www_host_match hdr(host) -i www.acme.com
use_backend my_www_backend if my_www_host_match
backend my_www_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
Here is the explanation for the example template above:
- HAProxy Specific Configuration section is declared prior to Custom Endpoint Configuration section.
- HAProxy Specific Configuration section is populated with operating system specific setup. In the example above it is populated with a cut-down version of Ubuntu’s configuration.
- Custom Endpoint Configuration section contains application specific proxy configuration.
- The endpoint that matches the following criteria has a custom configuration declared within the ifAcl block:
system
ispublic-site
system version
is1
endpoint name
ismain
- HAProxy frontend
my_www_frontend
is configured to listen on HTTP port80
and will relay request forwww.acme.com
tomy_www_backend
. - HAProxy backend
my_www_backend
server is generated based on the backend server information contained within the eachBackendServer helper function. - The backend server information will be updated whenever there’s changes to the endpoint, e.g. bundle instances being started or stopped within the ConductR cluster.
- All other endpoints is for configured using ConductR default configuration. The HTTP endpoint is exposed on the port
9443
§Deploying SSL certificate
This step is optional.
It’s possible to deploy SSL cert as part of the configuration override. The SSL cert will enable HTTPS traffic which terminates at the HAProxy.
Continuing from previous example, place the SSL certificate in /tmp/custom-haproxy-conf
, e.g.
cp /tmp/test-cert.pem /tmp/custom-haproxy-conf
In the example above, the SSL cert called test-cert.pem
is placed in /tmp/custom-haproxy-conf
.
The SSL cert will need to be referenced in the HAProxy configuration template. Continuing from previous example, uncomment the SSL configuration from the template, e.g.
...
{{#ifAcl 'public-site' '1' 'main'}}
frontend my_www_frontend
bind {{haproxyHost}}:80
# Uncomment the following line if SSL is used
bind {{haproxyHost}}:443 ssl crt {{overrideConfigDir}}/test-cert.pem
...
§Declare the custom HAProxy configuration template
Create a file called runtime-config.sh
within the proxy configuration directory which will be sourced by ConductR HAProxy upon startup, e.g.
touch /tmp/custom-haproxy-conf/runtime-config.sh
Populate the file with the following entry:
#!/bin/bash
CONFIG_DIR=$( cd $( dirname "${BASH_SOURCE[0]}" ) && pwd )
export CONDUCTR_HAPROXY_CONFIG_OVERRIDE="$CONFIG_DIR/haproxy-override.cfg"
The CONFIG_DIR
refers to the directory where the configuration override will be expanded. CONDUCTR_HAPROXY_CONFIG_OVERRIDE
is the environment variable which tells ConductR HAProxy where to find the custom HAProxy configuration template.
§Specifying custom HAProxy reload script
This step is optional. If a custom HAProxy reload script is not supplied, ConductR HAProxy will make use of its own default reload script packaged internally. The default reload script works for HAProxy installed for Systemd or SystemV.
ConductR HAProxy makes use of a reload script to reload HAProxy whenever a new HAProxy configuration is generated. When ConductR HAProxy starts, the reload script is copied from the path specified by HAPROXY_RELOAD_SCRIPT_SOURCE
environment variable to the path specified by HAPROXY_RELOAD_SCRIPT_LOCATION
environment variable.
Create a file called runtime-config.sh
within the proxy configuration directory which will be sourced by ConductR HAProxy upon startup, e.g.
touch /tmp/custom-haproxy-conf/runtime-config.sh
Populate the file with the following entry.
#!/bin/bash
CONFIG_DIR=$( cd $( dirname "${BASH_SOURCE[0]}" ) && pwd )
export HAPROXY_RELOAD_SCRIPT_SOURCE="$CONFIG_DIR/reloadHAProxy.sh"
export HAPROXY_RELOAD_SCRIPT_LOCATION="/usr/local/etc/haproxy/reloadHAProxy.sh"
# Ensure empty reload script exists with correct permission so it can be copied over
touch "${HAPROXY_RELOAD_SCRIPT_LOCATION}"
chmod u+x "${HAPROXY_RELOAD_SCRIPT_LOCATION}"
In the example above, the reload script will be copied from inside the bundle configuration to /usr/local/etc/haproxy/reloadHAProxy.sh
.
Populate the custom reload script /tmp/custom-haproxy-conf/reloadHAProxy.sh
, e.g.
#!/bin/bash
/usr/local/bin/haproxy reload
The custom reload script in the example above issued the /usr/local/bin/haproxy reload
command. Modify the command to suit your needs accordingly.
§Package the custom HAProxy configuration template
Use the CLI to package the configuration override:
bndl -f configuration /tmp/custom-haproxy-conf -o custom-haproxy-conf.zip
The generated file custom-haproxy-conf.zip
is the configuration override that can be loaded alongside ConductR HAProxy bundle.
§Load the custom HAProxy configuration template
If there’s an existing ConductR HAProxy running within ConductR, the running ConductR HAProxy instances have to be stopped. This is because each ConductR HAProxy instance is responsible for managing the HAProxy instance where the ConductR HAProxy instance is running.
Once custom configuration override is generated, it can be loaded into ConductR, e.g:
conduct load /tmp/conductr-haproxy-v2-0d24d10cb0d1af9bf9f7e0bf81778a61d2cc001f9393ef035cb343722da3ac87.zip /tmp/custom-haproxy-conf-ffd0dcf76f4d565424a873022fbb39f3025d4239c87d307be3078b320988b052.zip
In the example above, the files required are placed within the /tmp
directory. Replace the /tmp
with the actual path to the files.
Refer to the set of conduct
commands described in Deploying bundles for more details.
§Creating and deploying custom HAProxy configuration for Sandbox
Follow these steps to provide a custom HAProxy configuration to ConductR HAProxy running in the Sandbox. To provide a custom configuration for a non-sandbox environment go to the chapter Creating and deploying custom HAProxy configuration.
The steps are the same as in non-sandbox mode. However, the configuration values differ from the non-sandbox environment since HAProxy is running as a docker container on the Sandbox.
We will go through building the bundle configuration for HAProxy that can be loaded into the Sandbox. This will allow you to customize the behavior of HAProxy when using the Sandbox.
§Setup
Create a directory where the custom HAProxy configuration will be placed, e.g.
mkdir -p /tmp/custom-haproxy-conf
§Declare the custom HAProxy configuration entry point
Create a file called runtime-config.sh
within the proxy configuration directory, e.g.
touch /tmp/custom-haproxy-conf/runtime-config.sh
Populate the file with the following entry:
#!/bin/bash
export CONDUCTR_HAPROXY_USE_SUDO="off"
# HAProxy instance will be running inside Docker container, hence access will be controlled via docker run -p options.
export HAPROXY_BIND_HOST=0.0.0.0
CONFIG_DIR=$( cd $( dirname "${BASH_SOURCE[0]}" ) && pwd )
export CONDUCTR_HAPROXY_CONFIG_OVERRIDE="$CONFIG_DIR/haproxy-override.cfg"
export HAPROXY_RELOAD_SCRIPT_SOURCE="$CONFIG_DIR/reloadHAProxy.sh"
SANDBOX_PROXY_DIR="${HOME}/.conductr/proxy"
SANDBOX_HAPROXY_CFG_DIR="${SANDBOX_PROXY_DIR}/haproxy"
export HAPROXY_RELOAD_SCRIPT_LOCATION="${SANDBOX_PROXY_DIR}/reloadHAProxy.sh"
export HAPROXY_CONFIG_FILE="${SANDBOX_HAPROXY_CFG_DIR}/haproxy.cfg"
# Ensure directory exists
mkdir -p "${SANDBOX_HAPROXY_CFG_DIR}"
# Ensure empty reload script exists with correct permission so it can be copied over
touch "${HAPROXY_RELOAD_SCRIPT_LOCATION}"
chmod u+x "${HAPROXY_RELOAD_SCRIPT_LOCATION}"
CONDUCTR_HAPROXY_USE_SUDO
is set to off
since the HAProxy will be running as a docker container, as such the reload script needs to be run as the same user running the sandbox (hence no sudo
).
HAPROXY_BIND_HOST
is set to 0.0.0.0
since the actual HAProxy is running within the docker container, and there’s no way to tell what the actual bind ip will be until the container is started. The restriction of the incoming traffic to the HAProxy will be provided by the ports being opened into the container.
The HAPROXY_RELOAD_SCRIPT_LOCATION
is located at ~/.conductr/proxy/reloadHAProxy.sh
.
The HAPROXY_CONFIG_FILE
is located at ~/.conductr/proxy/haproxy/haproxy.cfg
. The directory ~/.conductr/proxy/haproxy
is mounted into the container so the haproxy.cfg
can be read by the HAProxy running within the container.
§Create HAProxy reload script
Create the custom HAProxy reload script, e.g.
touch /tmp/custom-haproxy-conf/reloadHAProxy.sh
Populate the file with the following template.
#!/bin/bash
set -e
docker kill -s HUP sandbox-haproxy
The signal HUP
issued to the sandbox-haproxy
container won’t terminate the container. Instead it will cause the HAProxy instance running within the sandbox-haproxy
container to be reloaded with the new configuration.
§Populate the custom HAProxy configuration template
Create the custom HAProxy template, e.g.
touch /tmp/custom-haproxy-conf/haproxy-override.cfg
Populate the file with the following template.
# HAProxy Specific Configuration
global
daemon
# We like statistics
stats socket {{haproxyHost}}:8999 level user
stats timeout 2m #Wait up to 2 minutes for input
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend conductr-haproxy-test
bind :65535
mode http
monitor-uri /test
# Custom Endpoint Configuration
{{#eachAcls bundles defaultHttpPort=9000}}
{{#ifAcl 'conductr-elasticsearch' '1' 'es'}}
# ConductR - Elasticsearch Bundle HAProxy Configuration
frontend elasticsearch_frontend
bind {{haproxyHost}}:9200
mode http
acl elasticsearch_context_root path_beg / /elastic-search
use_backend elasticsearch_backend if elasticsearch_context_root
backend elasticsearch_backend
mode http
reqrep ^([^\ :]*)\ /elastic-search/?(.*) \1\ /\2
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{#ifAcl 'conductr-kibana' '1' 'kibana'}}
# ConductR - Kibana Bundle HAProxy Configuration
frontend kibana_frontend
bind {{haproxyHost}}:5601
mode http
acl kibana_context_root path_beg /
use_backend kibana_backend if kibana_context_root
backend kibana_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{#ifAcl 'cinnamon-grafana-docker' '2' 'grafana'}}
# Cinnamon - Grafana HAProxy Configuration
frontend grafana_frontend
bind {{haproxyHost}}:3000
mode http
acl grafana_context_root path_beg /
use_backend grafana_backend if grafana_context_root
backend grafana_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{#ifAcl 'visualizer' '2' 'visualizer'}}
# ConductR - Visualizer Bundle HAProxy Configuration
frontend visualizer_frontend
bind {{haproxyHost}}:9999
mode http
acl visualizer_context_root path_beg /
use_backend visualizer_backend if visualizer_context_root
backend visualizer_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
{{#haproxyConf bundles}}
{{serviceFrontends}}
{{#unless (serviceFrontends)}}
frontend dummy
bind 127.0.0.1:65535
{{/unless}}
{{serviceBackends}}
{{/haproxyConf}}
Here is the explanation for the example template above:
- HAProxy Specific Configuration section is declared prior to Custom Endpoint Configuration section.
- Custom Endpoint Configuration section contains application specific proxy configuration.
- The HTTP endpoint is exposed on the port
9000
by default. - The backend server information will be updated whenever there’s changes to the endpoint, e.g. bundle instances being started or stopped within the ConductR cluster.
The custom configuration is declared within the ifAcl block:
Bundle Name | System Version | Endpoint Name | HTTP Port |
---|---|---|---|
conductr-elasticsearch | 1 | es | 9200 |
conductr-kibana | 1 | kibana | 5601 |
conductr-grafana-docker | 2 | grafana | 3000 |
visualizer | 2 | visualizer | 9999 |
These HTTP ports by will be exposed to the HAProxy docker container when the Sandbox starts.
§Package the custom HAProxy configuration template
Use the CLI to package the configuration override:
bndl -f configuration -o custom-haproxy-conf.zip /tmp/custom-haproxy-conf
The generated file custom-haproxy-conf.zip
is the configuration override that can be loaded alongside ConductR HAProxy bundle.
§Load the custom HAProxy configuration template
If there’s an existing ConductR HAProxy running within ConductR, the running ConductR HAProxy instances have to be stopped. This is because each ConductR HAProxy instance is responsible for managing the HAProxy instance where the ConductR HAProxy instance is running.
Once custom configuration override is generated, it can be loaded into ConductR, e.g:
conduct load /tmp/conductr-haproxy-v2-0d24d10cb0d1af9bf9f7e0bf81778a61d2cc001f9393ef035cb343722da3ac87.zip custom-haproxy-conf.zip
In the example above, the files required are placed within the /tmp
directory. Replace the /tmp
with the actual path to the files.
Refer to the set of conduct
commands described in Deploying bundles for more details.
§HAProxy environment variables
Here is the list of environment variables available to the operator when customizing the HAProxy via the runtime-config.sh
file.
Environment Variable | Description |
---|---|
HAPROXY_BIND_HOST | The bind host of the HAProxy. Defaults to the IP address assigned to the ConductR agent where ConductR HAProxy is running. |
HAPROXY_CONFIG_FILE | The absolute path to where generated HAProxy config will be placed. This defaults /etc/haproxy/haproxy.cfg . |
CONDUCTR_HAPROXY_CONFIG_OVERRIDE | The absolute path where HAProxy override template resides. If not supplied, the default template packaged internally within ConductR HAProxy will be used. |
CONDUCTR_HAPROXY_USE_SUDO | If true the HAProxy reload script will be invoked using sudo , and false otherwise. Defaults to true . |
HAPROXY_RELOAD_SCRIPT_SOURCE | The absolute path where HAProxy reload script will be copied from. Defaults to reloadHAProxy.sh packaged internally within ConductR HAProxy. |
HAPROXY_RELOAD_SCRIPT_LOCATION | The absolute path to the HAProxy reload script. Defaults to /usr/bin/reloadHAProxy.sh . |
STATUS_BIND_IP | The bind address of the HAProxy status endpoint. Defaults to the IP address assigned to the ConductR agent where ConductR HAProxy is running. The status endpoint is used for health check when exposing HAProxy to Load Balancer. |
STATUS_BIND_PORT | The bind port of the HAProxy status endpoint. Defaults to 9009 . |
HAPROXY_STATS_IP | The bind address of the HAProxy stats endpoint. Defaults to the IP address assigned to the ConductR agent where ConductR HAProxy is running. The stats endpoint is used for obtaining HAProxy performance stat. |
HAPROXY_STATS_PORT | The bind port of the HAProxy. Defaults to 8999 . |
§Default configuration
ConductR provides the following HAProxy configuration as a default:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# We like statistics
stats socket {{haproxyHost}}:8999 level user
stats timeout 2m #Wait up to 2 minutes for input
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
{{#eachAcls bundles defaultHttpPort=9000}}
{{#ifAcl 'conductr-kibana' '1' 'kibana'}}
# ConductR - Kibana Bundle HAProxy Configuration
frontend kibana_frontend
bind {{haproxyHost}}:5601
mode http
acl kibana_context_root path_beg /
use_backend kibana_backend if kibana_context_root
backend kibana_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{#ifAcl 'visualizer' '1.1' 'visualizer'}}
# ConductR - Visualizer Bundle HAProxy Configuration
frontend visualizer_frontend
bind {{haproxyHost}}:9999
mode http
acl visualizer_context_root path_beg /
use_backend visualizer_backend if visualizer_context_root
backend visualizer_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
{{#haproxyConf bundles}}
{{serviceFrontends}}
{{#unless (serviceFrontends)}}
frontend dummy
bind 127.0.0.1:65535
{{/unless}}
{{serviceBackends}}
{{/haproxyConf}}
The default configuration above is written in Handlebars template, and thus the template makes use of variables and helper functions made available to the template.
§Dynamic proxy configuration DSL
The variables and helper functions exposed to the template forms the DSL required to generate HAProxy configuration from the information available within the ConductR cluster.
§Variables
The following variables are available to the template:
Name | Type | Description |
---|---|---|
haproxyHost | String | Required: Host address of the proxy of which the configuration will be generated for. Example: 10.0.2.223 |
bundles | JSON array | Required: List of bundles returned by the ConductR bundles endpoint. This list is in the form of an arrays where each element is a bundle represented by a JSON object. The cluster-wide information of what has been deployed and what is currently running is being stored in this variable. Example: See Query bundle state |
overrideConfigDir | String | Optional: Physical directory location on the file system where configuration overrides has been unpacked. This variable is only defined when HAProxy custom template for ConductR HAProxy has been supplied. Example: /tmp/0d24d10cb0d1af9bf9f7e0bf81778a61-e357d11e8a38228880aa57d976d6f83b/e357d11e8a38228880aa57d976d6f83b576fd95e21dc99e8e4d02d4b227b4be3 In the example above the content of the ConductR HAProxy configuration override zip has been unpacked into the directory /tmp/0d24d10cb0d1af9bf9f7e0bf81778a61-e357d11e8a38228880aa57d976d6f83b/e357d11e8a38228880aa57d976d6f83b576fd95e21dc99e8e4d02d4b227b4be3 . |
§Functions
The following helper functions are available to the template:
§eachAcls
Iterates through each request ACL exposed by the endpoints given a list of bundles.
Used in conjunction with ifAcl to provide customised configuration override given endpoint matches a particular system
, system version
, and endpoint name
.
Once iteration of all bundles have been completed, generates default HAProxy configuration for the bundles without configuration override.
§Input arguments
Name | Type | Description |
---|---|---|
bundles | JSON array | Optional: List of bundles returned by the ConductR endpoint. Refer to bundles. |
defaultHttpPort | Integer | Default HTTP port where HTTP-based endpoints will be exposed. This must be specified keyword argument - refer to the example usage below. |
§Examples
{{#eachAcls bundles defaultHttpPort=9000}}
{{/eachAcls}}
In the example above, default HAProxy configuration will be generated. Since no overrides has been specified, so all HTTP-based endpoint will be exposed via port 9000
.
§ifAcl
Given an endpoint has request ACL defined and it matches a particular system
, system version
, and endpoint name
, the content nested within the ifAcl
block will be rendered.
The ifAcl
helper can be used as a stand-alone block helper, or nested within eachAcls block.
When called within eachAcls block, and the matching endpoint will be excluded when the default configuration is being generated.
§Input arguments
Name | Type | Description |
---|---|---|
bundles | JSON array | Optional: List of bundles returned by the ConductR endpoint. Refer to bundles. |
system | String | Match criteria for a particular system . |
systemVersion | String | Match criteria for a particular system version . |
endpointLabel | String | Match criteria for a particular endpoint name . |
§Examples
Standalone
frontend http_frontend
bind {{haproxyHost}}:80
mode http
{{#ifAcl bundles 'typesafe-website' '1' 'www'}}
# Forward to www.lightbend.com backend
acl lightbend_www hdr(host) -i www.lightbend.com
use_backend lightbend_www_backend if lightbend_www
{{/ifAcl}}
In the example above, the ifAcl
helper looks looks for the endpoint matches the following criteria within the bundle
input:
system
istypesafe-website
system version
is1
endpoint name
iswww
Should an endpoint that matches the above criteria is deployed and running, the content nested within the ifAcl
block will be generated. In this case, it will perform a header match against www.lightbend.com
and will forward the request to a backend called lightbend_www_backend
.
The content above contains additional DSLs such as haproxyHost which is explained in their respective sections.
Within eachAcls block
{{#eachAcls bundles defaultHttpPort=9000}}
{{#ifAcl 'visualizer' '1.1' 'visualizer'}}
# ConductR - Visualizer Bundle HAProxy Configuration
frontend visualizer_frontend
bind {{haproxyHost}}:9999
mode http
acl visualizer_context_root path_beg /
use_backend visualizer_backend if visualizer_context_root
backend visualizer_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
In the example above, the ifAcl
helper looks for the endpoint matches the following criteria:
* system
is visualizer
* system version
is 1
* endpoint name
is visualizer
Should an endpoint that matches the above criteria is deployed and running, the content nested within the ifAcl
block will be generated.
The content above contains additional DSLs such as haproxyHost, eachBackendServer, serverName, host, and port which are explained in their respective sections.
§eachBackendServer
Iterate through the backend server given a particular endpoint. The content declared within the eachBackendServer
block will be generated given each backend server, and will be concatenated in the end.
Can only be used within ifAcl block.
§Input arguments
The eachBackendServer
exposes the following variables. They are only scoped within eachBackendServer
block:
Name | Type | Description |
---|---|---|
serverName | String | A server name unique to a particular backend server. Only visible within eachBackendServer block. |
host | String | The host address of a particular backend server. Only visible within eachBackendServer block. |
port | Integer | The port of a particular backend server. Only visible within eachBackendServer block. |
§Examples
{{#ifAcl bundles 'typesafe-website' '1' 'www'}}
# Lightbend website
backend lightbend_www_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
In the example above we are iterating through the backend server for the endpoint matching the following criteria:
system
istypesafe-website
system version
is1
endpoint name
iswww
§haproxyConf
Generates HAProxy configuration given bundle endpoints which are declared with services
URI (as opposed to request acls).
§Input arguments
Name | Type | Description |
---|---|---|
bundles | JSON array | Optional: List of bundles returned by the ConductR endpoint. Refer to bundles. |
haproxyConf | String | Works in conjunction with serviceFrontends and serviceBackends below. |
§serviceFrontends
Generates HAProxy frontend configuration given bundle endpoints which are declared with services
URI (as opposed to request acls).
§serviceBackends
Generates HAProxy backend configuration given bundle endpoints which are declared with services
URI (as opposed to request acls).
§Examples
The functions haproxyConf, serviceFrontends, and serviceBackends are used in conjunction:
{{#haproxyConf bundles}}
{{serviceFrontends}}
{{#unless (serviceFrontends)}}
frontend dummy
bind 127.0.0.1:65535
{{/unless}}
{{serviceBackends}}
{{/haproxyConf}}
In the example above HAProxy configuration will be generated for endpoints declared with services
URI given the bundles
input. If there are no serviceFrontends
to be generated, the frontend called dummy
will be rendered instead.
§Example Use Cases
Here are some common use cases for the use of customized HAProxy configuration template.
§Collecting multiple backends under a single frontend
An example of this use case would be:
* Collect all public facing HTTP port 80/443 endpoints under the same frontend.
* All other endpoints are exposed via separate port exposed only to the internal network.
# Catch-all HTTP frontend coming at port 80
frontend http_frontend
bind {{haproxyHost}}:80
mode http
{{#ifAcl bundles 'typesafe-website' '1' 'www'}}
# Forward to www.lightbend.com backend
acl lightbend_www hdr(host) -i www.lightbend.com
use_backend lightbend_www_backend if lightbend_www
{{/ifAcl}}
{{#ifAcl bundles 'reactive-summit' '1' 'web'}}
# Forward to www.reactivesummit.org
acl reactive_summit_www hdr(host) -i www.reactivesummit.org
use_backend reactive_summit_www_backend if reactive_summit_www
{{/ifAcl}}
{{#eachAcls bundles defaultHttpPort=5443}}
{{#ifAcl 'typesafe-website' '1' 'www'}}
# Lightbend website
backend lightbend_www_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{#ifAcl 'reactive-summit' '1' 'web'}}
# Reactive Summit website
backend reactive_summit_www_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
Here is the explanation for the example template above:
* The first two ifAcl instructions collects the typesafe-website
and reactive-summit
endpoints under same HTTP frontend port 80
.
* The eachAcls instruction exposes the remaining endpoints to port 5443
which is declared as the default HTTP port.
* The ifAcl instructions nested within the eachAcls will render the backend for typesafe-website
and reactive-summit
endpoints respectively.
§HTTP Basic Auth
Here’s an example of enabling HTTP Basic Auth.
{{#eachAcls bundles defaultHttpPort=5443}}
{{#ifAcl 'conductr-kibana' '1' 'kibana'}}
# ConductR - Kibana Bundle HAProxy Configuration
userlist kibana_users
user kibana insecure-password k1b4n4
frontend kibana_frontend
bind {{haproxyHost}}:5601
mode http
acl kibana_basic_auth http_auth(kibana_users)
http-request allow if kibana_basic_auth
http-request auth realm kibana_users if !kibana_basic_auth
acl kibana_context_root path_beg /
use_backend kibana_backend if kibana_context_root
backend kibana_backend
mode http
{{#eachBackendServer}}
server {{serverName}} {{host}}:{{port}} maxconn 1024
{{/eachBackendServer}}
{{/ifAcl}}
{{/eachAcls}}
Here is the explanation for the example template above:
* The endpoint that matches the following criteria has a custom configuration declared within the ifAcl block:
* system
is conductr-kibana
* system version
is 1
* endpoint name
is kibana
* The userlist
provides list of user groups and user
directive declares each user within the group.
* The kibana_basic_auth
acl checks if the HTTP basic auth is supplied with the request
* The http-request auth realm kibana_users if !kibana_basic_auth
will trigger the HTTP basic auth prompt if the HTTP basic auth is not supplied with the request.
* The example has list of users declared with insecure password. In the actual production scenario, consider using a password
HAProxy directive which declares a secure password - refer to the HAProxy documentation for further details.
§Securing the ConductR CLI with Basic Authentication
HTTP Basic Authentication may be used to restrict access to a ConductR’s Control API. The ConductR CLI uses the Control API to load bundles and retrieve cluster information, thus applying Basic Authentication against this service restricts who is allowed to load, start and stop bundles in the ConductR cluster. This provides another method of securing the cluster beyond security group and firewall restrictions. This is useful when a fully restricted bastion host is not viable, or to further restrict access to particular clusters via the CLI. For example, while it may be acceptable to allow everyone on a team to deploy bundles to a development or staging cluster, a production cluster may require more restrictive access.
As a prerequisite, the ConductR Core’s Control Server should be configured to bind to a private IP address and an available port. By default, the Control Server will bind to the configured ConductR Core IP address and port 9005
. Binding to a different IP and/or port can be achieved by updating the configuration values conductr.control-server.ip
and/or conductr.control-server.port
respectively in /usr/share/conductr/conf/conductr.ini
. For this example, we will assume the ConductR Core’s IP address is private to the cluster but will bind the Control Server to 9055
internally to distinguish it from the external Control Protocol. Note that this change will be made on all ConductR Cores.
-Dconductr.control-server.port = 9055
Restart the ConductR Core Service with the following command:
> sudo service conductr restart
HAProxy must then be configured to provide Basic Authentication and SSL termination for the Control Protocol. In this case, it will still be accessed via the default port 9005
externally.
An SSL certificate will need to be generated and installed on the HAProxy host to enable two-way authentication for HTTPS. OpenSSL can be used for this. While there are lots of resources available online for creating certificates, the following script can be used to generate a self-signed certificate for testing purposes only. You will need to substitute the fields of the
openssl req
command with appropriate values for your server. It will create a folder calledssl-conductr.cli
containing the certificates and a .pem file:
#!/bin/bash
# Note - this script is for testing purposes only.
set -e
SCRIPT_DIR="$( cd "$(dirname $0)" && pwd)"
CERT_NAME=ssl-"conductr.cli"
SSL_CERT_DIR="$SCRIPT_DIR/$CERT_NAME"
#rm $SSL_CERT_DIR/*
mkdir -p $SSL_CERT_DIR
echo "Generating SSL certs and keys into $SSL_CERT_DIR"
openssl genrsa -out $SSL_CERT_DIR/$CERT_NAME.key 2048
#This line creates the certificate - Most important fields are CN=.. and subjectAltName=... These are the domains that the cert can be used with. For the purposes of the ConductR CLI, this must cover the Cores at least.
#For Dev servers, this would likely be (untested): *.ebiz.verizon.com
openssl req -new -key $SSL_CERT_DIR/$CERT_NAME.key -sha256 -nodes -subj '/C=US/ST=California/L=San Francisco/O=Lightbend/OU=Prod Suite/CN=192.168.99.100/emailAddress=dev.null@lightbend.com' -out $SSL_CERT_DIR/$CERT_NAME.csr
openssl x509 -req -days 365 -in $SSL_CERT_DIR/$CERT_NAME.csr -signkey $SSL_CERT_DIR/$CERT_NAME.key -out $SSL_CERT_DIR/$CERT_NAME.crt
echo "Concat crt and key into pem file"
cat $SSL_CERT_DIR/$CERT_NAME.crt $SSL_CERT_DIR/$CERT_NAME.key | tee $SSL_CERT_DIR/$CERT_NAME.pem
HAProxy is then configured with the path of our installed certificate and credentials to apply Basic Authentication to incoming ConductR Control API requests via port 9005
. To do this, we add the following to our custom HAProxy Configuration:
- a frontend ctrl_frontend_secure
to handle incoming Control API requests
- a userlist ctrl_users
, which specifies acceptable users and passwords.
While a plain text insecure password is shown below, HAProxy also supports secure (encrypted) passwords in userlists, which are recommended for production installations.
- a backend
ctrl_backend_auth
to direct to one of three available Conductr Cores (172.17.0.1
,172.17.0.2
and172.17.0.3
). In this case, each Control Server is listening on port9055
as configured above.
frontend dummy
bind 127.0.0.1:65535
frontend ctrl_frontend_secure
bind 0.0.0.0:9005 ssl crt /path/to/my/sslcert/ssl-conductr.cli.pem
mode http
acl ctrl_context path_beg /
use_backend ctrl_backend_auth if ctrl_context
userlist ctrl_users
user steve insecure-password stevespassword
backend ctrl_backend_auth
mode http
server ctrl_server_1 172.17.0.1:9055 maxconn 1024
server ctrl_server_2 172.17.0.2:9055 maxconn 1024
server ctrl_server_3 172.17.0.3:9055 maxconn 1024
acl ctrl_auth_ok http_auth(ctrl_users)
http-request auth realm ConductR if !ctrl_auth_ok
With the above setup, Control Protocol requests must be sent via HTTPS with Basic Authentication credentials. The ConductR CLI supports this via a credentials file that must be created at the following location ~/.conductr/settings.conf
, which must point to PEM file used to validate the SSL certificate locally (ss-conductr-cli.pem in the above example). Configuring this file is described in detail the ConductR CLI documentation section called HTTP Basic Authentication.
§Enabling ConductR to reload HAProxy without altering the sudoers file
On some systems, it may not be possible or desirable to provide root privileges to the conductr-agent such that it can execute the /usr/bin/reloadHAProxy.sh
script. In these cases, it is still possible for ConductR to utilize HAProxy with a slightly modified configuration. Note that with this modified configuration, HAProxy will be limited to binding to ports greater than or equal to 1024. It will also limit the maximum number of connections (maxconn) to the maximum number of open file descriptors for non-root Linux users (1024 by default).
In addition to installing HAProxy, this setup requires the following ownership changes on all nodes that will run HAProxy:
sudo mkdir /var/run/haproxy
sudo chown conductr-agent:conductr-agent /var/run/haproxy
sudo chown conductr-agent:conductr-agent /etc/haproxy/haproxy.cfg
We now need to create a custom HAProxy configuration bundle. Starting with the default /tmp/custom-haproxy-conf/haproxy-override.cfg
shown above, remove the chroot
, user
and group
lines, such that the top of the file looks as follows:
global
log /dev/log local0
log /dev/log local1 notice
maxconn 1024
daemon
# We like statistics
stats socket {{haproxyHost}}:8999 level user
stats timeout 2m #Wait up to 2 minutes for input
#The remaining HAProxy configuration below remains unchanged from the default
# ...
A new script will be required to reload HAProxy without root access. We’ll create it as follows:
touch /tmp/custom-haproxy-conf/reload-haproxy-nosudo.sh
Populate /tmp/custom-haproxy-conf/reload-haproxy-nosudo.sh
with the following:
#!/bin/bash
haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy/haproxy.pid -sf $(cat /var/run/haproxy/haproxy.pid)
Populate /tmp/custom-haproxy-conf/runtime-config.sh
with the following to ensure that the reload-haproxy-nosudo.sh is used for reloading haproxy:
#!/bin/bash
CONFIG_DIR=$( cd $( dirname "${BASH_SOURCE[0]}" ) && pwd )
export CONDUCTR_HAPROXY_CONFIG_OVERRIDE="$CONFIG_DIR/haproxy-override.cfg"
#Update Conductr-Agent Configuration to call the custom reload HAProxy script
CONFIG_FILE="conductr-haproxy/conf/application.ini"
echo -Dconductr-haproxy.configurator.command.0="/bin/bash" | tee -a "${CONFIG_FILE}"
echo -Dconductr-haproxy.configurator.command.1="$CONFIG_DIR/reload-haproxy-nosudo.sh" | tee -a "${CONFIG_FILE}"
Finally, you can use the following command to load conductr-haproxy
with a custom configuration:
conduct load conductr-haproxy /tmp/custom-haproxy-conf
Should you wish to save the configuration bundle for use later, the following command will create a bundle file for you instead:
bndl /tmp/custom-haproxy-conf -f configuration -o ~/my-conductr-haproxy-config.zip
§Troubleshooting
Here are the common troubleshooting steps when there’s a problem exposing bundle endpoints.
§Check if bundle is running
The proxy will only be configured for endpoints from the bundles which are running. Use the command conduct info
to ensure that your bundle is in the running state.
§Ensure ACL is exposed
Review the ACL configuration of the endpoints deployed in the ConductR cluster. Use the command conduct acls http
and conduct acls tcp
to display request ACLs for HTTP and TCP traffic respectively.
The command will provide a warning message if duplicate endpoints are detected. Refer to Requirement section for more details on duplicate endpoints.
§Check the logs from ConductR HAProxy bundle
Check the logs from ConductR HAProxy bundle for possible errors.
If there is a problem with the supplied custom HAProxy configuration template, they will be logged as part of ConductR HAProxy bundle.
If consolidated logging with Elasticsearch is enabled, the command conduct logs conductr-haproxy
will display logs from the ConductR HAProxy bundle.
Refer to Logging for further details on Elasticsearch bundle installation and setup.
§Check the generated HAProxy configuration
The ConductR HAProxy bundle will generate HAProxy configuration in /etc/haproxy/haproxy.cfg
- check if the generated HAProxy configuration is correct.
Next: Log Files