Multiple SmartOS Zones on One IPv4 Address

Purpose

This document explains how to share one external IPv4 address between multiple SmartOS zones using an Etherstub and NAT.

Requirements

This how-to requires that your SmartOS server has two public IP addresses. The first is used for the global zone. The second is used by a zone which functions as a router. The router provides NAT between the public address and a private subnet.

Process

Create an Etherstub

Configure an Ethernet stub by adding this line to your /usbkey/config file:

etherstub=private0

If you want multiple Ethernet stubs, you can separate them with commas:

etherstub=stub0,stub1

Next, you can reboot to prove that the Ethernet stub is created at boot time. If you prefer not to reboot, you can create the same Ethernet stub manually with nictagadm:

[root@smartos ~]# nictagadm add -l private0

You can see the existing NICs in a system:

[root@smartos ~]# nictagadm list
NAME           MACADDRESS         LINK           TYPE
admin          c8:60:00:12:34:56  rge0           normal
private0       -                  -              etherstub
waldo0         -                  -              etherstub

You can also use the dladm command to manipulate Ethernet stubs.

Create a Router Zone

Our SmartOS server is hosted at Hetzner. Hetzner requires that our MAC address match one of the assigned values or packets will be dropped, so we have assigned the MAC address given by Hetzner to the admin net_tag. Also, if you are not using IPv6, you should leave the IPv6 resolvers and addresses out of your configuration. Here, we create a JSON file describing our router zone:

[root@smartos ~]# cat > router.json <<EOF
{
  "brand": "joyent",
  "image_uuid": "b2d3c018-e6e5-11e6-9815-f7d92aad9089",
  "max_physical_memory": 256,
  "hostname": "oregano",
  "resolvers": [
    "8.8.8.8",
    "8.8.4.4",
    "2001:4860:4860::8888",
    "2001:4860:4860::8844"
  ],
  "alias": "oregano",
  "nics": [
    {
      "mac": "00:50:56:12:34:56",
      "nic_tag": "admin",
      "gateways": [
        "144.76.51.193"
      ],
      "netmask": "255.255.255.224",
      "ips": [
        "144.76.51.203/27",
        "2a01:4f8:190:9293::1:1/64"
      ],
      "allow_ip_spoofing": true,
      "primary": true
    },
    {
      "nic_tag": "private0",
      "gateways": [
        "192.168.2.1"
      ],
      "netmask": "255.255.255.0",
      "ips": [
        "192.168.2.1/24"
      ],
      "allow_ip_spoofing": true
    }
  ],
  "firewall_enabled": true,
  "quota": 1,
  "tags": {
    "role": "router"
  }
}
EOF

Note

We only assigned an IPv6 address to the external interface. This is because we have no need for NAT with IPv6. While we could assign an IPv6 address to the internal interface, this would require some manual configuration of the routing tables and provide no additional benefit.

Then, create the router zone with vmadm:

[root@smartos ~]# vmadm create -f router.json
Successfully created VM 6c753865-c478-cb2b-de22-fd1acbdc57b6

Next, use zlogin to check connectivity. As we explained in IPv6 on SmartOS on Hetzner, until issue 548 is fixed you, must add the default gateway (fe80::1) before IPv6 connectivity will work:

[root@smartos ~]# zlogin 6c753865-c478-cb2b-de22-fd1acbdc57b6
[Connected to zone '6c753865-c478-cb2b-de22-fd1acbdc57b6' pts/2]
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (minimal-64-lts 16.4.1)
                   `-'  https://docs.joyent.com/images/smartos/minimal

[root@oregano ~]# route -p add -inet6 default fe80::1
add net default: gateway fe80::1
add persistent net default: gateway fe80::1
[root@oregano ~]# ping -a google.com
google.com (2a00:1450:4001:818::200e) is alive
google.com (172.217.21.206) is alive
[root@oregano ~]# ifconfig
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
net0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 144.76.51.203 netmask ffffffe0 broadcast 144.76.51.223
        ether 0:50:56:12:34:56
net1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 192.168.2.1 netmask ffffff00 broadcast 192.168.2.255
        ether 72:e3:2f:1f:7f:f1
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128
net0: flags=2004841<UP,RUNNING,MULTICAST,DHCP,IPv6> mtu 1500 index 2
        inet6 fe80::250:56ff:fe12:3456/10
        ether 0:50:56:12:34:56
net0:1: flags=2000841<UP,RUNNING,MULTICAST,IPv6> mtu 1500 index 2
        inet6 2a01:4f8:190:9293::1:1/64

Create a Web Server Zone

To test our router zone, create a web server zone:

[root@smartos ~]# cat > httpd.json <<EOF
{
  "brand": "joyent",
  "image_uuid": "b2d3c018-e6e5-11e6-9815-f7d92aad9089",
  "max_physical_memory": 256,
  "hostname": "thyme",
  "resolvers": [
    "8.8.8.8",
    "8.8.4.4",
    "2001:4860:4860::8888",
    "2001:4860:4860::8844"
  ],
  "alias": "thyme",
  "nics": [
    {
      "nic_tag": "admin",
      "ips": [
        "2a01:4f8:190:9293::2:2/64"
      ]
    },
    {
      "nic_tag": "private0",
      "gateways": [
        "192.168.2.1"
      ],
      "ips": [
        "192.168.2.2/24"
      ],
      "primary": true
    }
  ],
  "firewall_enabled": true,
  "quota": 1,
  "tags": {
    "ipv4_nat": true,
    "role": "web"
  }
}
EOF
[root@smartos ~]# vmadm create -f httpd.json
Successfully created VM ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2
[root@smartos ~]# zlogin ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2
[Connected to zone 'ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2' pts/4]
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (minimal-64-lts 16.4.1)
                   `-'  https://docs.joyent.com/images/smartos/minimal

[root@thyme ~]# route -p add -inet6 default fe80::1
add net default: gateway fe80::1
add persistent net default: gateway fe80::1
[root@thyme ~]# ping -n 2001:4860:4860::8888
2001:4860:4860::8888 is alive
[root@thyme ~]# ping -n 192.168.2.1
192.168.2.1 is alive
[root@thyme ~]# ping -A inet6 -a google.com
google.com (2a00:1450:4001:819::200e) is alive
[root@thyme ~]# ping -A inet -a google.com
ping: sendto No route to host

You can see that we have IPv6 connectivity, but only limited IPv4 connectivity. This is because we have not enabled NAT in the router zone. Letʼs do that now.

Configure NAT

Create a NAT configuration file in the router zone at /etc/ipf/ipnat.conf. Weʼll allow NAT for all outbound and certain inbound connections:

[root@oregano ~]# cat > /etc/ipf/ipnat.conf <<EOF
# Inbound port forwarding
# HTTP and HTTPS
rdr net0 from any to 144.76.51.203 port = 80 -> 192.168.2.2 port 80 tcp
rdr net0 from any to 144.76.51.203 port = 443 -> 192.168.2.2 port 443 tcp

# Outbound connections
# Use the Dynamic Ports range for port mapping
# https://tools.ietf.org/html/rfc6335#section-6
map net0 from 192.168.2.0/24 to any -> 0/32 portmap tcp/udp 49152:65535
map net0 from 192.168.2.0/24 to any -> 0/32
EOF

Next, enable packet forwarding and ipfilter:

[root@oregano ~]# routeadm -u -e ipv4-forwarding
[root@oregano ~]# svcadm enable ipfilter

You can check the NAT configuration with ipnat -l:

[root@oregano ~]# ipnat -l
List of active MAP/Redirect filters:
rdr net0 from any to 144.76.51.203/32 port = www -> 192.168.2.2 port 80 tcp
rdr net0 from any to 144.76.51.203/32 port = https -> 192.168.2.2 port 443 tcp
map net0 from 192.168.2.0/24 to any -> 0.0.0.0/32 portmap tcp/udp 49152:65535
map net0 from 192.168.2.0/24 to any -> 0.0.0.0/32

List of active sessions:

Configure the Firewall

Weʼre starting with an almost empty slate. You can see your existing firewall rules with fwadm list:

[root@smartos ~]# fwadm list
UUID                                 ENABLED RULE
689ffd36-f250-4872-aeed-ddec87154c23 true    FROM any TO all vms ALLOW icmp6 TYPE all
a3312dbe-a49c-4196-9287-ef0c5937a471 true    FROM any TO all vms ALLOW icmp TYPE all

When creating a new zone which uses the router zone for NAT, you need to update the firewall rules. We have two goals:

  1. Connections from the Internet should reach all web server zones on ports 80 and 443.
  2. The zones must be able to reach pkgsrc.joyent.com on port 443 so we can install packages.

For the first goal, the router zone must accept and forward IPv4 packets on port 80 and 443. Additionally, any zone which functions as a web server must accept IPv4 packets on port 80 and 443 from the router zone and IPv6 packets on ports 80 and 443 from anywhere. This will require multiple firewall rules.

Firewall rules take the form:

FROM <source> TO <destination> <action> <protocol> <ports_or_types>

For the first rule, the source is any and the destination is the router zone vm 6c753865-c478-cb2b-de22-fd1acbdc57b6 [1]. The action is ALLOW (or BLOCK if you want to block the packets). The protocol is tcp and the ports are ports 80,443, so our rule looks like this:

FROM any TO vm 6c753865-c478-cb2b-de22-fd1acbdc57b6 ALLOW tcp ports 80,443

Note

You can read the rules for creating firewall rules on the fwrule man page:

[root@smartos ~]# man fwrule

For the second rule, the source is any [2] and the destination is the web server zone vm ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2. The action is ALLOW, the protocol is tcp and the ports are ports 80,443. The rule looks like this:

FROM any to vm ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2 ALLOW tcp ports 80,443

Note, however, that we used a special tag when we created the router zone:

"tags":
  {
    "role": "router"
  }

The web server also had to role "web", so we can write a more general rule which covers both routers and web servers:

FROM any TO (tag role = router OR tag role = web) ALLOW tcp ports 80,443

Note

This rule does double duty: it accepts inbound HTTP and HTTPS requests from the external network and it allows outbound HTTP and HTTPS requests from zones behind the NAT. If you want to limit outbound requests from the zones behind the NAT, you will need an additional rule, or to use a more specific rule using ipf.

For the second goal, we could design a set of rules that allow only the web server zone to connect to pkgsrc.joyent.com through the router zone, but that is unnecessarily restrictive. Other zones may want to download packages, so we can add a temporary [3] rule that allows all zones to connect to pkgsrc.joyent.com [4] on port 443:

FROM all vms TO ip 37.153.97.16 ALLOW tcp port 443

For general use, we need to allow DNS lookups, which communicate with the Google Public DNS servers over UDP and TCP port 53:

FROM all vms TO (ip 8.8.8.8 OR ip 8.8.4.4 OR ip 2001:4860:4860::8888 or ip 2001:4860:4860::8844) ALLOW udp port 53
FROM all vms TO (ip 8.8.8.8 OR ip 8.8.4.4 OR ip 2001:4860:4860::8888 or ip 2001:4860:4860::8844) ALLOW tcp port 53

Finally, we need the router zone to accept outbound DNS queries from the zones behind NAT:

FROM tag "ipv4_nat" TO tag "role" = "router" ALLOW udp ports 53
FROM tag "ipv4_nat" TO tag "role" = "router" ALLOW tcp ports 53

We need to package these rules in JSON so the fwadm command understand it. Then we can add them:

[root@smartos ~]# cat > firewall-rules.json <<EOF
{
  "rules": [
    {
      "rule": "FROM any TO (tag role = router OR tag role = web) ALLOW tcp ports 80,443",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    },
    {
      "rule": "FROM all vms TO ip 37.153.97.16 ALLOW tcp port 443",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    },
    {
      "rule": "FROM all vms TO (ip 8.8.8.8 OR ip 8.8.4.4 OR ip 2001:4860:4860::8888 or ip 2001:4860:4860::8844) ALLOW udp port 53",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    },
    {
      "rule": "FROM all vms TO (ip 8.8.8.8 OR ip 8.8.4.4 OR ip 2001:4860:4860::8888 or ip 2001:4860:4860::8844) ALLOW tcp port 53",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    },
    {
      "rule": "FROM tag ipv4_nat TO tag role = router ALLOW udp port 53",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    },
    {
      "rule": "FROM tag ipv4_nat TO tag role = router ALLOW tcp port 53",
      "enabled": true,
      "owner_uuid": "00000000-0000-0000-0000-000000000000"
    }
  ]
}
EOF
[root@smartos ~]# fwadm add -f firewall-rules.json
Added rules:
92ee8dda-09d1-4f9a-b666-c7b09c480425 true    FROM any TO (tag "role" = "router" OR tag "role" = "web") ALLOW tcp (PORT 443 AND PORT 80)
10c26e0c-ba09-4fd6-8e82-56d9ef797544 true    FROM all vms TO ip 37.153.97.16 ALLOW tcp PORT 443
92d84254-aa78-480e-8a73-0fe11f6a8d48 true    FROM all vms TO (ip 2001:4860:4860::8844 OR ip 2001:4860:4860::8888 OR ip 8.8.4.4 OR ip 8.8.8.8) ALLOW udp PORT 53
3dc0bde4-c800-4da6-8476-258ba3aeca9e true    FROM all vms TO (ip 2001:4860:4860::8844 OR ip 2001:4860:4860::8888 OR ip 8.8.4.4 OR ip 8.8.8.8) ALLOW tcp PORT 53
ac371d7e-f513-4b43-bf82-0c05d93594c1 true    FROM tag "ipv4_nat" TO tag "role" = "router" ALLOW udp PORT 53
cb0b34e2-f250-448c-a9d9-14cb5482a4fd true    FROM tag "ipv4_nat" TO tag "role" = "router" ALLOW tcp PORT 53

Note

owner_uuid

The owner_uuid for a firewall rule must match the zoneʼs owner_uuid. You can get the owner_uuid for your zone from vmadm:

[root@smartos ~]# vmadm get ae5be3e9-e504-4e0d-9234-ba7aa7a2cad2 | json owner_uuid
00000000-0000-0000-0000-000000000000

owner_uuid exists to allow multiple users to control zones and firewall rules on a system. For servers with a single user, this is not important. For larger systems, like Joyent Triton and Project FiFo, this is important.

[1]If you need to find your zoneʼs UUID by hostname, you can use a command like vmadm lookup hostname=oregano. Read the vmadm man page for more details.
[2]We could try to block IPv4 requests coming from hosts other than the router zone, but there is no need if the intent is for the web site to be public.
[3]Temporary because we will remove the rule after we have installed the necessary software.
[4]dig +short pkgsrc.joyent.com

Install a Web Server

Now that the web server zone has IPv4 connectivity, we can install nginx:

[root@thyme ~]# pkgin install nginx
reading local summary...
processing local summary...
calculating dependencies... done.

nothing to upgrade.
2 packages to be installed (1883K to download, 5167K to install):

pcre-8.39 nginx-1.11.4

proceed ? [Y/n] Y
downloading packages...
pcre-8.39.tgz                                                                                                                                                         100% 1246KB 622.9KB/s 769.8KB/s   00:02
nginx-1.11.4.tgz                                                                                                                                                      100%  637KB 637.1KB/s 177.1KB/s   00:01
installing packages...
installing pcre-8.39...
installing nginx-1.11.4...
nginx-1.11.4: Creating group ``www''
nginx-1.11.4: Creating user ``www''
passwd: password information changed for www
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/fastcgi.conf to /opt/local/etc/nginx/fastcgi.conf
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/fastcgi_params to /opt/local/etc/nginx/fastcgi_params
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/koi-utf to /opt/local/etc/nginx/koi-utf
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/koi-win to /opt/local/etc/nginx/koi-win
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/mime.types to /opt/local/etc/nginx/mime.types
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/naxsi_core.rules to /opt/local/etc/nginx/naxsi_core.rules
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/nginx.conf to /opt/local/etc/nginx/nginx.conf
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/uwsgi_params to /opt/local/etc/nginx/uwsgi_params
nginx-1.11.4: copying /opt/local/share/examples/nginx/conf/win-utf to /opt/local/etc/nginx/win-utf
===========================================================================
This package has SMF support.  You may use svcadm(1M) to 'enable', 'disable'
or 'restart' services.  To enable the instance(s) for this package, run:

        /usr/sbin/svcadm enable -r svc:/pkgsrc/nginx:default

Use svcs(1) to check on service status.  See smf(5) for more information.
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1 2013/02/22 17:06:54 imil Exp $

Consider adding something like following lines to /etc/newsyslog.conf:

/var/log/nginx/access.log www:www 640 7 * 24 Z  /var/db/nginx/nginx.pid SIGUSR1
/var/log/nginx/error.log  www:www 640 7 * 24 Z  /var/db/nginx/nginx.pid SIGUSR1

===========================================================================
pkg_install warnings: 0, errors: 0
reading local summary...
processing local summary...
marking nginx-1.11.4 as non auto-removable

Next, we can enable it:

[root@thyme ~]# svcadm enable nginx

Finally, we can test it:

[root@thyme ~]# curl -I http://localhost/
HTTP/1.1 200 OK
Server: nginx/1.11.4
Date: Mon, 20 Feb 2017 06:01:00 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Fri, 27 Jan 2017 00:35:30 GMT
Connection: keep-alive
ETag: "588a95d2-264"
Accept-Ranges: bytes

Knowing that it works from inside the zone, we can test it from outside the zone, sending the request to the router zoneʼs external address:

[root@smartos ~]# curl -I http://144.76.51.203/
HTTP/1.1 200 OK
Server: nginx/1.11.4
Date: Mon, 20 Feb 2017 06:04:42 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Fri, 27 Jan 2017 00:35:30 GMT
Connection: keep-alive
ETag: "588a95d2-264"
Accept-Ranges: bytes

If that works, you can verify it from your browser.

Summary

In this how-to, we have seen how to create a SmartOS zone which controls an external IPv4 address. We have configured that zone to provide NAT to other zones and configured a firewall to limit access to the zones. Finally, we verified connectivity by downloading and installing a web server and by connecting to the web server from our computers.