Recently, a client had the need to run two instances of OHS on the same host. This typically wouldn’t be a problem except for the fact that they wanted to use the same port number for both instances. Why did they want to do this you ask? Well, this was for one of their development environments, and they wanted to simulate the setup of their high-availability staging and production environments without the additional investment in a second OHS server. To achieve this we needed to bind a second IP address to the network card then associate the instances to their respective IP. Read on for the details.
Binding the second IP
Run the command ip addr list on the Linux host and you’ll probably see two entries, one for the loopback (127.0.0.1) virtual interface and the other for the actual network interface. It’ll look something like this:
[ec2-user@ip-10-0-254-251 ~]$ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
link/ether 0a:f1:a7:ff:67:c9 brd ff:ff:ff:ff:ff:ff
inet 10.0.254.251/24 brd 10.0.254.255 scope global eth0
inet6 fe80::8f1:a7ff:feff:67c9/64 scope link
valid_lft forever preferred_lft forever
[ec2-user@ip-10-0-254-251 ~]$
Issue the command sudo ip addr add 10.0.254.153/24 dev eth0 to add the second IP to your network adapter (of course, substitute your IP and adapter for the one in this example). Re-run ip addr list again and you’ll see the second IP attached (highlighted in red).
[ec2-user@ip-10-0-254-251 ~]$ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
link/ether 0a:f1:a7:ff:67:c9 brd ff:ff:ff:ff:ff:ff
inet 10.0.254.251/24 brd 10.0.254.255 scope global eth0
inet 10.0.254.153/24 scope global secondary eth0
inet6 fe80::8f1:a7ff:feff:67c9/64 scope link
valid_lft forever preferred_lft forever
[ec2-user@ip-10-0-254-251 ~]$
With OHS1 already installed and running, execute the command to list the network connections, sudo lsof -i –P. You’ll see something like this for those entries related to the web tier utilities:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd.wor 1981 ofm 4u IPv6 313416 0t0 TCP *:7777 (LISTEN)
httpd.wor 1981 ofm 6u IPv6 313420 0t0 TCP *:4443 (LISTEN)
httpd.wor 1981 ofm 8u IPv6 313425 0t0 TCP *:7779 (LISTEN)
httpd.wor 1991 ofm 4u IPv6 313416 0t0 TCP *:7777 (LISTEN)
httpd.wor 1991 ofm 6u IPv6 313420 0t0 TCP *:4443 (LISTEN)
httpd.wor 1991 ofm 8u IPv6 313425 0t0 TCP *:7779 (LISTEN)
httpd.wor 1991 ofm 16u IPv6 314264 0t0 TCP localhost:49034->localhost:1880 (ESTABLISHED)
httpd.wor 1996 ofm 4u IPv6 313416 0t0 TCP *:7777 (LISTEN)
httpd.wor 1996 ofm 6u IPv6 313420 0t0 TCP *:4443 (LISTEN)
httpd.wor 1996 ofm 8u IPv6 313425 0t0 TCP *:7779 (LISTEN)
httpd.wor 1997 ofm 4u IPv6 313416 0t0 TCP *:7777 (LISTEN)
httpd.wor 1997 ofm 6u IPv6 313420 0t0 TCP *:4443 (LISTEN)
httpd.wor 1997 ofm 8u IPv6 313425 0t0 TCP *:7779 (LISTEN)
opmn 2108 ofm 6u IPv6 314256 0t0 TCP localhost:1880 (LISTEN)
opmn 2108 ofm 7u IPv4 314257 0t0 TCP localhost:1880 (LISTEN)
opmn 2108 ofm 8u IPv6 314258 0t0 TCP *:6701 (LISTEN)
opmn 2108 ofm 9u IPv6 314265 0t0 TCP localhost:1880->localhost:49034 (ESTABLISHED)
opmn 2108 ofm 11u IPv6 314480 0t0 TCP localhost:1880->localhost:49038 (ESTABLISHED)
opmn 2108 ofm 12u IPv6 314487 0t0 TCP localhost:1880->localhost:49039 (ESTABLISHED)
If you’ve got the second OHS instance installed with it’s default configuration, and tried to execute it, you’ll find that it won’t run. Note the *:7777, *4443, *.6701, etc… in the list above. You can see that the first instance is listening on all IPs, which explains why the second won’t run. We’ll fix this for OPMN and OHS (shown above as httpd.wor) by associating each instance with its own IP.
Associating the instances with their own IP
For instance 1, where I have XX.XX.XX.XX you’ll add your primary IP, which in my example is 10.0.254.251.
To affect httpd.wor, do the following:
- In httpd.conf, find the directive listen 7777 and change it to listen XX.XX.XX.XX:7777
- In admin.conf, find the directive listen 7779 and change it to listen XX.XX.XX.XX:7779
- In ssl.conf, find the directive listen 4443 and change it to listen XX.XX.XX.XX:4443
To affect opmn, go to $INSTANCEHOME/config/OPMN/opmn and edit the file opmn.xml. Add the tag shown in red below to the block as shown.
<notification-server interface=”any”>
<ipaddr remote=”XX.XX.XX.XX” request=”XX.XX.XX.XX”/>
<port local=”1880″ remote=”6701″/>
<ssl enabled=”true” wallet-file=”/opt/ora/fmw/Oracle_WT1/admin/instance1…
</notification-server>
Stop OPMN and all its managed processes. Start it again, then start just OHS1.
[ofm@ip-10-0-254-251 bin]$ ./opmnctl stopall
opmnctl stopall: stopping opmn and all managed processes…
[ofm@ip-10-0-254-251 bin]$ ./opmnctl start
[ofm@ip-10-0-254-251 bin]$ ./opmnctl startproc ias-component=ohs1
opmnctl startproc: starting opmn managed processes…
[ofm@ip-10-0-254-251 bin]$
Re-run the sudo lsof -i –P command again and you should see the asterisks replaced with your IP, hostname, or DNS entry. That’s what you want!
[ofm@ip-10-0-254-251 bin]$ lsof -i -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
opmn 12003 ofm 6u IPv6 373589 0t0 TCP localhost:1880 (LISTEN)
opmn 12003 ofm 7u IPv4 373590 0t0 TCP localhost:1880 (LISTEN)
opmn 12003 ofm 8u IPv4 373591 0t0 TCP ip-10-0-254-251.ec2.internal:6701 (LISTEN)
opmn 12003 ofm 10u IPv6 373688 0t0 TCP localhost:1880->localhost:54782 (ESTABLISHED)
httpd.wor 12029 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12029 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12029 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12038 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12038 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12038 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12038 ofm 13u IPv6 373687 0t0 TCP localhost:54782->localhost:1880 (ESTABLISHED)
httpd.wor 12043 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12043 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12043 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12044 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12044 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12044 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
Next, stop OPMN and all its managed processes again, then begin the install of instance2 (if not installed already). When the Oracle installer utility performs its verification it checks all IPs bound to OPMN, OHS, and WebCache ports. For that reason, you’ll need to stop the first instance when performing the install of instance2.
Then, for this new instance, make the same adjustments to httpd.conf, admin.conf, ssl.conf, opmn.xml, but using the secondary IP address. IMPORTANT: You’ll have to make one additional change to the opmn.xml file for the second instance. You’ll have to change the local port to a different number (shown below in red). OPMN local “always” uses localhost, and I don’t know if there’s any way around it. That said, you’ll have to change this port to get the second instance of OPMN to even start.
<notification-server interface=”any”>
<ipaddr remote=”XX.XX.XX.XX” request=”XX.XX.XX.XX”/>
<port local=”1881” remote=”6701″/>
<ssl enabled=”true” wallet-file=”/opt/ora/fmw/Oracle_WT1/admin/instance2…
</notification-server>
Finally, start the second instance of OPMN then start the second instance of OHS. You shouldn’t have a problem with the startup.
[ofm@ip-10-0-254-251 bin]$ ./opmnctl start
[ofm@ip-10-0-254-251 bin]$ ./opmnctl startproc ias-component=ohs2
opmnctl startproc: starting opmn managed processes…
[ofm@ip-10-0-254-251 bin]$
When you run lsof –I -P again you’ll see two sets of opmn and httpd.wor, one for each IP.
[ofm@ip-10-0-254-251 bin]$ lsof -i -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
opmn 12003 ofm 6u IPv6 373589 0t0 TCP localhost:1880 (LISTEN)
opmn 12003 ofm 7u IPv4 373590 0t0 TCP localhost:1880 (LISTEN)
opmn 12003 ofm 8u IPv4 373591 0t0 TCP ip-10-0-254-153.ec2.internal:6701 (LISTEN)
opmn 12003 ofm 10u IPv6 373688 0t0 TCP localhost:1880->localhost:54782 (ESTABLISHED)
httpd.wor 12029 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-153.ec2.internal:7777 (LISTEN)
httpd.wor 12029 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-153.ec2.internal:4443 (LISTEN)
httpd.wor 12029 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-153.ec2.internal:7779 (LISTEN)
httpd.wor 12038 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-153.ec2.internal:7777 (LISTEN)
httpd.wor 12038 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-153.ec2.internal:4443 (LISTEN)
httpd.wor 12038 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-153.ec2.internal:7779 (LISTEN)
httpd.wor 12038 ofm 13u IPv6 373687 0t0 TCP localhost:54782->localhost:1880 (ESTABLISHED)
httpd.wor 12043 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-153.ec2.internal:7777 (LISTEN)
httpd.wor 12043 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-153.ec2.internal:4443 (LISTEN)
httpd.wor 12043 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-153.ec2.internal:7779 (LISTEN)
httpd.wor 12044 ofm 3u IPv4 373630 0t0 TCP ip-10-0-254-153.ec2.internal:7777 (LISTEN)
httpd.wor 12044 ofm 4u IPv4 373633 0t0 TCP ip-10-0-254-153.ec2.internal:4443 (LISTEN)
httpd.wor 12044 ofm 5u IPv4 373636 0t0 TCP ip-10-0-254-153.ec2.internal:7779 (LISTEN)
opmn 12421 ofm 6u IPv6 374877 0t0 TCP localhost:1881 (LISTEN)
opmn 12421 ofm 7u IPv4 374878 0t0 TCP localhost:1881 (LISTEN)
opmn 12421 ofm 8u IPv4 374879 0t0 TCP ip-10-0-254-251.ec2.internal:6701 (LISTEN)
opmn 12421 ofm 10u IPv6 374984 0t0 TCP localhost:1881->localhost:58037 (ESTABLISHED)
httpd.wor 12454 ofm 3u IPv4 374934 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12454 ofm 4u IPv4 374936 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12454 ofm 5u IPv4 374939 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12463 ofm 3u IPv4 374934 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12463 ofm 4u IPv4 374936 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12463 ofm 5u IPv4 374939 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12463 ofm 13u IPv6 374983 0t0 TCP localhost:58037->localhost:1881 (ESTABLISHED)
httpd.wor 12468 ofm 3u IPv4 374934 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12468 ofm 4u IPv4 374936 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12468 ofm 5u IPv4 374939 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
httpd.wor 12469 ofm 3u IPv4 374934 0t0 TCP ip-10-0-254-251.ec2.internal:7777 (LISTEN)
httpd.wor 12469 ofm 4u IPv4 374936 0t0 TCP ip-10-0-254-251.ec2.internal:4443 (LISTEN)
httpd.wor 12469 ofm 5u IPv4 374939 0t0 TCP ip-10-0-254-251.ec2.internal:7779 (LISTEN)
[ofm@ip-10-0-254-251 bin]$
This indicates that both instances are running and listening on the appropriate IP address.
