Post coming soon!

Have fun!

Reading the freeradius documentation there is a standard fix to this issue:

It may happen that the server does not respond with the right source ip address. Thus the client will likely ignore the server’s response. On Linux and FreeBSD you can solve this by specifying ./configure –with-udpfromto during compilation.

Unfortunately I was not in the position to recompile the code, since I use a specific version provided by a commercial company.

In the attempt to solve this issues I have tried many approach including:

• iptables on the radius server – Trying to NAT the physical IP address to the VIP. It works fine for all the other protocols, but not with Radius. I am suspicious that Radius interface the protocol stack below the IP tables
• creating an extra loopback interface on the server and routing the frame to it. It works, but again Radius replies with the physical address

If someone have more ideas please send them to me.

Eventually I discovered that the Mikrotik Routers that I use support defining a secondary radius server, that is used only if the query to the primary radius servers timesouts, something like:

> radius print
Flags: X - disabled
 #   SERVICE  CALLED-ID  DOMAIN  ADDRESS         SECRET                      
 0   ppp                         10.49.49.205    xxxyyy                   
 1   ppp                         10.49.49.206    xxxyyy

Where 10.49.49.205 is the primary server and 206 is the backup server.

Sometime the simpler solutions are the more effective !

Have fun!

Warning: I am going to upgrade drivers/kernel module. This can make your system unusable. You have been warned, now. No responsibility is assumed for anything at all.

First you want to be sure of what is installed in your system, Type:

lspci -vv

From there you get the driver name. To have more info on the driver (e.g. assuming driver name is igb):

modinfo igb

I started to browse around and discovered ethtool. It was installed by default on my CentOS system, but you can also download it from here.

with it you can have a closer look at how your card is configured:

ethtool eth0

at the errors

ethtool -S eth0

and see what driver version it runs

ethtool -i eth0

If you have a very old driver version, it is probably a good idea to upgrade it. I have an Intel card and I started here.

The process is in four steps

1. download the kit and unpack it (e.g. wget, untar)
2. build it (e.g. make install)
3. remove the old driver (e.g. rmmod e1000e)
4. install the new driver (e.g. modprobe e1000e)

Read the README file that is contained in your driver kit. When you are done reboot and check the result with:

ethtool -i eth0

If the problem has not disappeared, as in my case , you may want to start force some parameters with ethtool. A good guide is here.

My issue was that the switch port was set to full-duplex and the server port was set to autonegotiate. This is not a valid configuration. I decided to  set everything full duplex at 1 Gb/s with this command:

ethtool -s eth3 speed 100 autoneg off duplex full

That solved the issue.

Have fun!

There are several link-aggregation techniques that are possible, probably the best known being Bonding on Unix and NIC-teaming on Windows.

Following the recommendation of my friend Piero I went a different path, I used bridging.

DISCLAIMER: don’t attempt this on a production network! Folks who have seen a Spanning Tree loop don’t want to see a second one! Test on a small separate testing environment and be ready for packet storms, network crashes, server crashes, etc.! You have been warned, now. No responsibility is assumed for anything at all.

1) Bridging with STP

Piero sent me this synthetic, but complete description of what to do on a generic server:

ifconfig eth0 0.0.0.0
ifconfig eth3 0.0.0.0
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth3
brctl stp br0 on
ifconfig br0  up
ifconfig br0 10.49.48.205 netmask 255.255.255.0

There are some excellent tutorials on bridging in Linux, for example:

• http://www.tldp.org/HOWTO/Ethernet-Bridge-netfilter-HOWTO-3.html#ss3.1
• http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge

The first thing is to draw the network diagram of what you are trying to achieve:This is an example of a config file for server-2

#!/bin/bash
PATH="/sbin:/usr/sbin:/usr/local/sbin";
cmd="$1";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
  start)
    brctl addbr br0;
    brctl stp br0 on;
    brctl setbridgeprio br0 40960; ### We don't want to be route
    brctl addif br0 eth0;
    brctl addif br0 eth3;
    brctl setportprio br0 eth3 100; ### by defaukt we want to bock eth3
    ifconfig eth0 down;
    ifconfig eth3 down;
    ifconfig eth0 0.0.0.0 up;
    ifconfig eth3 0.0.0.0 up;
    ifconfig br0 10.49.48.206 netmask 255.255.255.0 up
    route add default gw 10.49.48.209;
    echo "nameserver 10.49.48.210" > /etc/resolv.conf
    echo "nameserver 10.49.48.211" >> /etc/resolv.conf
    ;;
  stop)
    brctl delif br0 eth0;
    brctl delif br0 eth3;
    ifconfig br0 down;
    brctl delbr br0;
    ifup eth0;
    echo "nameserver 10.49.48.210" > /etc/resolv.conf
    echo "nameserver 10.49.48.211" >> /etc/resolv.conf
    ;;
  restart)
    $0 stop;
    sleep 3;
    $0 start;
    ;;
  status)
        route
        ;;
  *)
	echo "Usage: bridge-rstp {start|stop|restart|status}"
	exit 1
	;;
esac;

A cleaner way to do this can be achieved in two steps. First create using the GUI a network interface named br0 and assign there the IP address, netmask, the default gateway, and nameservers. The file in CentOS will be called /etc/sysconfig/network-scripts/ifcfg-br0.These files are read by a service called NetworkManager.

At this point you can simplify the previous script to something like

#!/bin/bash
PATH="/sbin:/usr/sbin:/usr/local/sbin";
cmd="$1";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
  start)
    brctl addbr br0;
    brctl stp br0 on;
    brctl setbridgeprio br0 40960; ### We don't want to be route
    brctl addif br0 eth0;
    brctl addif br0 eth3;
    brctl setportprio br0 eth3 100; ### by default we want to bock eth3
    ifconfig eth0 down;
    ifconfig eth3 down;
    ifconfig eth0 0.0.0.0 up;
    ifconfig eth3 0.0.0.0 up;
    ifuo br0;
    ;;
  stop)
    brctl delif br0 eth0;
    brctl delif br0 eth3;
    ifconfig br0 down;
    brctl delbr br0;
    ifup eth0;
    ;;
  restart,reload)
    $0 stop;
    sleep 3;
    $0 start;
    ;;
  status)
        route
        ;;
  *)
	echo "Usage: bridge-rstp {start|stop|restart|status}"
	exit 1
	;;
esac;

You can further simplify the script and at the end get rid of it if you use bridge-utils.  I haven’t tested it, but my understanding is that bridge-utils move all the bridge configuration inside the file /etc/sysconfig/network-scripts/ifcfg-br0.

I haven’t tested it! and as a matter of fact I went the opposite direction and removed deinstalled NetworkManager

# yum delete NetworkManager

This works, it is robust and it has a convergence time of around 30 seconds, to which you will need to add the convergence time of VRRP (see note at the end of this post) for a total of approximately 1 minute. Probably good enough in most cases.

2) Bridging with RSTP

At this point I decided to do something that I rarely do: improve Piero’s solution. It is the same, but with RSTP (Rapid STP) instead of STP. The goal is to reduce the convergence time to few seconds.

Warning: don’t attempt this if not all your bridges are configured with RSTP. Use wireshark to verify that there are no classical STP BPDUs on the network.

I searched extensively in March 2012 and found reference to the possibility that in the future RSTP will be supported in the kernel, but I was not able to find any kernel implementation. So I decided to use a user space implementation available in the Fedora repository and I installed it on CentOS.

If you run Fedora, it should be pretty simple, just do:

yum install rstp

For your reference this is the Fedora repository.

There are two alternative way you can do this in CentOS, since CentOS uses a slightly older version of glibc and I don’t think it is wise trying to upgrade glibc that is a crucial library.

a) either download an earlier version of rstp

http://rpm.pbone.net/index.php3/stat/4/idpl/15479839/dir/fedora_14/com/rstp-04012009git-4.fc14.i686.rpm.html

and install the rpm with:

rpm -i rstp-04012009git-4.fc14.i686.rpm.html

b) or download the source code and recompile

The source code is at:
http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/source/SRPMS/r/rstp-04012009git-7.fc17.src.rpm

You will need to apply the two patches contained in the root directory before recompiling.

patch < rstp-type-punning.patch
patch < rstp-unused-fix.patch
make
make install

After installing you will need to understand how to enable RSTP instead of STP.

Unfortunately the software only documentation is reading this email thread starting at: http://www.mail-archive.com/bridge@lists.linux-foundation.org/msg00432.html

The key point in understanding the role of the file /sbin/bridge-stp

From the previous email thread:

When you do "brctl stp <bridge_name> on", the kernel runs
"/sbin/bridge-stp <bridge_name> start" and waits for the exit code. If
the exit code is 0, the kernel sets stp_state to 2, and assumes that
userspace will take care of STP and does not do any STP actions. (This
state is different from STP disabled, since during STP disabled, the
kernel still brings up ports from learning to forwarding using a
timeout, and also, the bridge forwards BPDUs too in that state.) If
the exit code is non-zero, stp_state is set to 1 and kernel STP is
activated.

Depending on how you installed the software /sbin/bridge-stp may be present or not, but I highly recommend to replace it with this simple file.

  #!/bin/bash
  exit 0

Remember

chmod 755 /sbin/bridge-stp

At this point you have disabled STP in the kernel and created the condition to use RSTP in user space. Now you need to start things in the correct order, I used this script (name it bridge-rstp)

#!/bin/bash
#set -x
#
### BEGIN INIT INFO
# Provides: RSTP
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $network $syslog d
# Default-Stop: 0 1 6
# Short-Description: Reconfigure the ports to support bridging and RSTP
# Description: RSTP has faster convergence than STP
### END INIT INFO
#
PATH="/sbin:/usr/sbin:/usr/local/sbin:/usr/bin:/bin";
cmd="$1";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
  start)
    brctl addbr br0;
    brctl addif br0 eth0;
    brctl addif br0 eth3;
    brctl stp br0 on;
    ifconfig eth0 down;
    ifconfig eth3 down;
    ifconfig eth0 0.0.0.0 up;
    ifconfig eth3 0.0.0.0 up;
    ifconfig br0 10.49.48.206 netmask 255.255.255.0 up;
    route add default gw 10.49.48.209;
    rstpd
    rstpctl rstp br0 on
    rstpctl setbridgeprio br0 40960
    rstpctl setportpathcost br0 eth3 40000
    echo "nameserver 10.49.48.209" > /etc/resolv.conf
    ;;
  stop)
    route del default
    ifconfig br0 down
    rstpctl rstp br0 off
    brctl delif br0 eth0;
    brctl delif br0 eth3;
    brctl delbr br0;
    killall -9 rstpd
    ifconfig eth0 10.49.48.206 netmask 255.255.255.0 up;
    route add default gw 10.49.48.209;
    echo "nameserver 10.49.48.209" > /etc/resolv.conf
    ;;
  restart,reload)
    $0 stop;
    sleep 3;
    $0 start;
    ;;
  status)
	ps ax | grep rstpd
        rstpctl showport br0
        route
        ;;
  *)
	echo "Usage: bridge-rstp {start|stop|restart|status}"
	exit 1
	;;
esac;

Try multiple time

./bridge-rstp start
./bridge-rstp stp

After starting rstp check it with commands like

rstpctl showport br0

Be careful that NetworkManager is running in the backgroud and may try to manipulate interfaces that you are using for bridging. NetworkManager reads the files in /ets/sysconfig/network-scripts, you may want to erase the one that you don’t need at boot or at least edit them and set boot=off. As I told you before I removed NetworkManager

If you are satisfied by the result achieved, you can make it permanent:

cp bridge-stp /etc/init.d/bridge-rstp
chown root:root /etc/init.d/bridge-rstp
chmod 755 /etc/init.d/bridge-rstp
chkconfig --add bridge-rstp
chkconfig bridge-rstp on

From this point on it will automatically start at boot. You can start and stop it manually with:

service bridge-rstp start
service bridge-rstp stop

Since reliability is very important for me, I decided to run a 5 hours RSTP stress test on my servers by bringing up and down interfaces continuously. I used this perl script.

#!/usr/bin/perl -w
use strict;
use warnings;
#
# This script brings up and down the interfaces in a bridge group
# to stress test RSTP
#
# Set these parameters correctly
#
my @interfaces = ("eth0", "eth3");
my $maxpostdelay = 10; # in seconds
my $maxdowndelay = 10; # in seconds
my $interactions = 1800;
#
# variables used by the script
#
my $interface;
my $downdelay;
my $postdelay;
#
 open LOG, ">", "rstp.log" or die $!;
 print LOG "RSTP stress test started at: ";
 print LOG localtime() ."\n";
 while ($interactions) {
   $downdelay=int(rand($maxdowndelay));
   $postdelay=int(rand($maxpostdelay));
   $interface=$interfaces[int(rand(@interfaces))];
   print LOG localtime() ." - ";
   print LOG "Interface = $interface - Down = $downdelay - Post = $postdelay\n";
   print LOG "$interactions interacions to go - " . localtime() ." - Interface $interface down\n";
   print "\n---------------------------------------------\n";
   print "$interactions interacions to go - " . localtime() ." - Interface $interface down\n\n";
   system("ifconfig $interface down");
   system("ifconfig $interface");
   sleep ($downdelay);
   system("rstpctl showport br0");
   print "\n";
   print LOG localtime() ." - Interface $interface up\n";
   system("ifconfig $interface up");
   system("ifconfig $interface");
   sleep ($postdelay);
   system("rstpctl showport br0");
   $interactions--;
 }
close LOG;

During this test and other tests done manually I noticed that the worst case was a 5 seconds convergence time, with many cases in which the convergence time was subsecond. This implies that in most cases not a single ping was lost!

3) VRRP

The next step is to get a VIP (Virtual IP) address, using VRRP (Virtual Router Redundancy Protocol): don’t get confused, even if VRRP has “Router” in its name, it can be used on hosts.

Be careful, firewalls on the hosts like to drop VRRP frames, be sure that it is not happening.

Unfortunately VRRP does not run on TCP or UDP, but directly over IP, so you cannot simply open a TCP/UDP port. Test with the firewall disable, when it works close the firewall and add to your IPTABLES something like:

 iptables -I INPUT -p vrrp -j ACCEPT

You can do it in bridge-rstp immediately before starting VRRP.

You can see the result by typing:

iptables --list -v

There are also discussion on how to do this at:

• http://archive.linuxvirtualserver.org/html/lvs-users/2006-10/msg00021.html
• http://www.cyberciti.biz/faq/linux-unix-verify-keepalived-working-or-not/

There are implementations of VRRP available at sourceforce and at http://off.net/~jme/vrrpd/. I was able to use  the sourceforce implementation over an ethernet interface with a command like:

vrrpd -i eth0 -v 2 -p 200 10.49.48.2

which start vrrpd over eth0, on group 2, with priority 200 and VIP = 10.49.48.2.

Unfortunately, I was not able to make it work on a bridge interface like br0. So i decided to go another path and installed keepalived that comes precompiled for CentOS.

# yum install keepalived

Keepalived is a very powerful package that contains many things, I just enabled only VRRP following the instruction contained here.

I edited the file /etc/keepalived/keepalived.conf to look like:

! Configuration File for keepalived

vrrp_instance VI_1 {
    state MASTER
    interface br0
    virtual_router_id 10
    priority 200
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1234
    }
    virtual_ipaddress {
        10.49.48.2/24
    }
}

On the backup server use state BACKUP and priority 100.

At the end I put all together in /etc/init.d/bridge-rstp

#!/bin/bash
#set -x
#
### BEGIN INIT INFO
# Provides: RSTP
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $network $syslog d
# Default-Stop: 0 1 6
# Short-Description: Reconfigure the ports to support bridging and RSTP
# Description: RSTP has faster convergence than STP
### END INIT INFO
#
PATH="/sbin:/usr/sbin:/usr/local/sbin:/usr/bin:/bin";
cmd="$1";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
  start)
    brctl addbr br0;
    brctl addif br0 eth0;
    brctl addif br0 eth3;
    brctl stp br0 on;
    ifconfig eth0 down;
    ifconfig eth3 down;
    ifconfig eth0 0.0.0.0 up;
    ifconfig eth3 0.0.0.0 up;
    ifconfig br0 10.49.48.205 netmask 255.255.255.0 up;
    rstpd
    rstpctl rstp br0 on
    rstpctl setbridgeprio br0 40960
    rstpctl setportpathcost br0 eth3 40000
    route add default gw 10.49.48.208;
    iptables -I INPUT -p vrrp -j ACCEPT
    service keepalived start
    echo "nameserver 10.49.48.208" > /etc/resolv.conf
    ;;
  stop)
    service keepalived stop
    route del default
    ifconfig br0 down;
    rstpctl rstp br0 off
    brctl delif br0 eth1;
    brctl delif br0 eth3;
    brctl delbr br0;
    killall -9 rstpd
    ifconfig eth0 10.49.48.205 netmask 255.255.255.0 up;
    route add default gw 10.49.48.208;
    echo "nameserver 10.49.48.208" > /etc/resolv.conf
     ;;
  restart)
    $0 stop;
    sleep 3;
    $0 start;
    ;;
  status)
	ps ax | grep rstpd
        rstpctl showport br0
        route
        ;;
  *)
	echo "Usage: bridge-rstp {start|stop|restart|status}"
	exit 1
	;;
esac;

Piero also warned me that I need to slow down the convergence of VRRP to let the STP (Spanning Tree Protocol) converge first (otherwise both servers can become VRRP masters in the transient ;-( ). not so much an issue if you use RSTP,  but classical STP takes 50 seconds to converge! So you will need to increase the convergence time of VRRP to let STP converge first. With RSTP the convergence was fast enough that I used the default timers of VRRP.

This is left for you to experiment

Have fun!

A close second is NTOP. You probably haven’t heard about this tool developed by my friend Luca Deri: it is a gem in the rough! It is a traffic analysis tool which provides you a more high level view of what is going on in your networks, which are the most used protocols, who consumes more bandwidth, is it there any BitTorent  traffic? I cannot recommend more this tools. Luca also develops other network tools that you will find on his website.

In third place i have fping. fping is a program to ping host in parallel. It is very useful when you want to understand which hosts are active on a particular subnet. You can specify any number of hosts on the command line, or specify a file containing the lists of hosts to ping. Instead of trying one host until it timeouts or replies, fping will send out a ping packet and move on to the next host in a round-robin fashion.

In fourth place I rank DNSBench. One of the reasons of slow internet surfing is that we pick the wrong DNS server. DNSBench runs a survey of all the DNS servers and help you pick the most responsive one. Program them into your home router and all your hosts will benefit from this.

In fifth place I have snmpwalk. snmpwalk is an SNMP application that uses SNMP GETNEXT requests to query a network entity for a tree of information. For example you can type

 snmpwalk 192.168.0.231 -v 1 -c public .1

and discover which OIDs are supported by the device 192.168.0.231.

When you have discoverted the OIDs, you can use my sixth place application MIB Browser from iReasoning. If you work with SNMP, MIB Browser is a must. It will help you walk any MIB and read/set any parameter. It is very usefull for studying which parameters are worth monitoring through an SNMP tool like Nagios.

In the final place a tool that my friend Piero recently recommended: GFTP. I haven’t yet tested it, but there is no guy I trust more than Piero when it comes to troubleshooting computer networks. I must admit that FileZilla is a valid alternative.

Have fun!

I am happy, I trust my friend Dinesh, but then I ask the question: “so if my boot disk fails I can restore it from the backup?”

Well to turn a long story short the answer is NO! Dinesh explains me that if I want to do that I need a cloning software, also called “bare metal backup”.Backup software like Déjà Dup are good for backing up user data (very important application), not system files.

The original cloning software in Linux was “dd“. dd can be used to copy regions of raw device files, for example backing up the boot sector of a hard disk. The disadvantage of dd is that it copies also unused space. Partclone on the other hand understand the file system structure and it is capable of cloning only used blocks.

It turns out that there are mini linux system that you can install on a USB key and that contain partclone and other tools.

• SystemRescueCD
• RedoBackup
• CloneZilla
• RIP

after some playing around I decided for RedoBackup and I have been very happy I have used it to backup Linux and Windows XP servers. It has all the tools I need.

Have fun!

If you need to enable routing, I recommend using OSPF, only the backbone area. There is an example on how to do this here. The full description on how to program OSPF can be found here.

If you need VRRP to have a stable default gateway for the host, it is pretty simple to program it following this instructions. Just be very careful that if you have a bridge interface, for example, because you want to run spanning tree, you need to enable VRRP on the bridge interface. Let’s assume you have created bridge1 over ether2-master-local. You need to enable VRRP on bridge1 and not on ether2-master-local. If you make a mistake you will get an I – Invalid Flag. In this case, the best thing is to remove the interfaces and recreate them correctly.

In general it is good to create two VRRP groups and have one active on a router and one active on the other, for example on router 1

/int vrrp print                                       
Flags: X - disabled, I - invalid, R - running, M - master, B - backup
 #     NAME                INTERFACE                MAC-ADDRESS       VRID PRIORITY INTERVAL             VERSION V3-PROTOCOL
 0  RM vrrp208             bridge1                  00:00:5E:00:01:D0  208      200 1s                   3       ipv4       
 1   B vrrp209             bridge1                  00:00:5E:00:01:D1  209      100 1s                   3       ipv4

and on router 2

/int vrrp print                                      
Flags: X - disabled, I - invalid, R - running, M - master, B - backup
 #     NAME              INTERFACE             MAC-ADDRESS       VRID PRIORITY INTERVAL             VERSION V3-PROTOCOL
 0   B vrrp208           bridge1               00:00:5E:00:01:D0  208      100 1s                   3       ipv4       
 1  RM vrrp209           bridge1               00:00:5E:00:01:D1  209      200 1s                   3       ipv4

Another thing I had to do was to monitor my NAT. I used:

# ip firewall nat print all stats

The next thing is t check which packages are installed and active and the software release. You can do that with:

/ system package print
Flags: X - disabled
 #   NAME                                           VERSION                                          SCHEDULED              
 0   system                                         5.14                                                                    
 1   gps                                            5.14                                                                    
 2   advanced-tools                                 5.14                                                                    
 3   security                                       5.14                                                                    
 4 X ipv6                                           5.14                                                                    
 5   ntp                                            5.14                                                                    
 6   dhcp                                           5.14                                                                    
 7   calea                                          5.14                                                                    
 8   hotspot                                        5.14                                                                    
 9   lcd                                            5.14                                                                    
10 X mpls                                           5.14                                                                    
11   ppp                                            5.14                                                                    
12   user-manager                                   5.14                                                                    
13   routing                                        5.14                                                                    
14   multicast                                      5.14                                                                    
15   ups                                            5.14                                                                    
16   routerboard                                    5.14                                                                    
17 X wireless                                       5.14

The content of the package is discussed here.

You can enable and disable packages using:

/system package enable
/system package disable

You may also want to check if your router is eligible for a free upgrade, by typing:

/system license print

the upgradable-to: field will tell you the highest release you can upgrade to.  Read more here.

If you decide to upgrade follow one of these methods. I successfully used ftp.

Have fun !

Then I started to use it seriously and I understood that it has on operating system that is very complete and that supports all the network feature that I need, from Radius to PPPoE, from conntrack, to NAT, OSPF, DHCP/client/relay/server, etc. I have a previous post that explain how to program many features.

Still I was stuck with the four port switch and not being able to have a router, for example, with 3 or 4 ports. Then I read this post. As you can see the RB750/GL uses the Atheros 8327 which allows to program the ports as bridged ports or routed ports.

The four port switch that comes standard with the RB750/GL is just a default configuration, not the only one, other are possible.

I started to play with my RB750/GL trying to transform port 5 from a bridged port to a routed port.

I started exploring with commands like:

/interface ethernet switch print
/interface ethernet print

then I took the courage and did

/interface ethernet set master-port=none name=ether5-routed                  
/interface ethernet  print                                
Flags: X - disabled, R - running, S - slave
 #    NAME                                              MTU MAC-ADDRESS       ARP        MASTER-PORT                                          SWITCH                                         
 0 R  ether1-gateway                                   1500 00:0C:42:E6:90:A6 enabled    none                                                 switch1                                        
 1    ether2-master-local                              1500 00:0C:42:E6:90:A7 enabled    none                                                 switch1                                        
 2  S ether3-slave-local                               1500 00:0C:42:E6:90:A8 enabled    ether2-master-local                                  switch1                                        
 3  S ether4-slave-local                               1500 00:0C:42:E6:90:A9 enabled    ether2-master-local                                  switch1                                        
 4    ether5-routed                                    1500 00:0C:42:E6:90:AA enabled    none                                                 switch1

and then decided to assign ether5-routed an IP address

/ip address add address=10.49.47.210/24 interface=ether5-routed
/ip add print                                                   
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE                                                                                                                                            
 0   10.49.49.211/24    10.49.49.0      ether2-master-local                                                                                                                                  
 1   12.0.64.211/24     12.0.64.0       ether1-gateway                                                                                                                                       
 2   12.0.64.205/32     12.0.64.205     ether1-gateway                                                                                                                                       
 3   12.0.64.206/32     12.0.64.206     ether1-gateway                                                                                                                                       
 4   10.255.255.1/32    10.255.255.1    loopback                                                                                                                                             
 5   10.49.47.210/24    10.49.47.0      ether5-routed

Now, if you want, you can repeat the process for ether4-slave-local and ether3-slave-local.

Good Luck and Have fun !

 PHP Warning: It is not safe to rely on the system’s timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected ‘America/Los_Angeles’ for ‘PDT/-7.0/DST

To fix this I edited the file /etc/php.ini and changed the following line, fixing also my coordinates at the same time:

[Date]
; Defines the default timezone used by the date functions
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = "America/Los_Angeles"

; http://www.php.net/manual/en/datetime.configuration.php#ini.date.default-latitude
date.default_latitude = 37.8388

; http://www.php.net/manual/en/datetime.configuration.php#ini.date.default-longitude
date.default_longitude = -120.2308