## Installing the eeE8 DDI8 Driver for UnixWare* 7

Installation Configuration, Teaming, and VLAN Technical Notes and Troubleshooting

NOTE: The instructions and documentation below reflect the process using the console mode, rather than a graphical interface. X Windows* commands may be worded somewhat differently, but the process for installation and configuration is fundamentally the same.

When installing and configuring network adapters, you may need to refer to SCO UnixWare* 7 documentation. Please have this documentation available during the installation process.

For maximum system stability, it is recommended that all network adapters be configured with the same driver type (eeE8 (DDI8)). To use the teaming and VLAN features of your Intel adapter, you will need to install DDI8 on all adapters that will use iANS.

### Installation

#### To ensure predictable behavior when installing network adapters, deconfigure all adapters and reboot the system prior to configuring new adapters or reconfiguring installed adapters.

1. Copy the eeE8.pkg file into any directory on the UnixWare system, such as in the /tmp directory.

2. Make sure no other users are logged on and all user applications are closed.

3. If there is an older version of either eeE or eeE8 driver on the system, first run 'netcfg' and remove any configuration using these drivers. Exit 'netcfg'. Reboot the system using "init 6".

4. Install the new driver using 'pkgadd'. For example: # pkgadd -d /tmp/eeE8.pkg

5. If you intend to use advanced options such as teaming/VLAN, repeat steps 1 and 4 above to add the iANS.pkg file to your system.

6. If you install the iANS.pkg, reboot your system after installation, before beginning to configure your adapters.

7. Add and configure the NICs. See Configuration and Teaming below to add and configure adapters, configure iANS, and set up teaming and VLANs.

#### Notes:

- Hot Plug enabled systems require the eeE8 (DDI8) drivers and (if you're not using iANS) must only be configured using the Hot Plug utility.
  # scoadmin hot

> If you are using iANS, see the section Using Hot Plug and iANS.

- Hot Plug PCI capabilities also require a Hot Plug Controller driver and the appropriate Hot Plug PCI ptfs.

- Systems not requiring Hot Plug PCI capabilities but wishing to use eeE8 (DDI8) drivers should use netcfg.

- Use netcfg to configure eeE8 (DDI8) drivers not # scoadmin network

### Configuration, Teaming, and VLAN

Creating a team with iANS in UnixWare* 7 is a two-step process. The first step involves adding members to the team, during which process the team's configuration is established. During the second step, you will add a virtual adapter (or several virtual adapters in VLAN mode) and choose the networking protocol you wish to use on the virtual adapter(s).

1. Copy the iANS.pkg file to any directory on the UnixWare system.

2. Install iANS using 'pkgadd'. For example: #pkgadd -d /tmp/iANS.pkg. After installing iANS, reboot your system.

3. If the base driver (DDI eeE8) has not yet been installed, follow the instructions in Installation to install the driver. Make sure that all network adapters in the system are configured with the same driver type, and that no protocols are configured on the adapters.

4. Start netcfg. # netcfg

5. Netcfg will present the available adapters and drivers. Select the first adapter to configure, then select Continue. NOTE: If you also have the DDI7 (eeE) driver on your system, it will appear as a configuration option. Make sure you select only the adapters displaying the eeE8 driver.

6. Configure Advanced Options if you wish to set a default duplex speed (Half Duplex_10Mbps, Full Duplex_10Mbps, Half Duplex_10Mbps, or Full Duplex_100Mbps). For automatic duplexing, select OK. (NOTE: "Auto_Auto" is the default for this setting. Select Auto_Auto, not Cancel, to leave this screen without making changes. Cancel is nonfunctional in this screen, and may leave Netcfg in an unusable state if it is selected.)

7. Netcfg will install the driver. Select the initial protocol for this adapter. For teaming or VLAN operation, select iANS team "0" (where "0" is the team ID given to the first team), then Add. NOTE: You must select iANS before you select any other protocol in order to use this adapter as part of a virtual adapter. You will be able to configure TCP/IP or IPX/SPX on the virtual adapter after you have successfully configured iANS on your adapters.

8. Choose the priority of the member just added, then select OK. (Priority will determine the manner in which the different NICs will handle traffic or errors.) The adapter will be configured with the chosen settings.

9. The ANS Configuration Utility window will prompt you to choose the role of the member in the team. The user will determine the team configuration only once, during the addition of the first team member. NOTE: Teaming configuration read the elow

10. You may now configure a second adapter from the Hardware menu by selecting "Add New LAN Adapter". Repeat steps 5-9. If you are configuring more than one team, you will need to select another team ID for subsequent teams, and then select the same protocol for all adapters you add to those team(s).

#### Teaming

1. After adding the first member of the team, the ANS Configuration Utility screen will appear. Identify the team, and select VLAN mode if you wish to include VLANs in your team. (If you commit the team in "no VLAN" mode, you will still need to add a virtual adapter (Ethernet-iANS Virtual Adapter without Vlan-PCI Slot 0 Bus 0) and select a protocol to be installed over it. Instructions for adding a virtual adapter are included below.)

2. Enter the number of VLANs that will be added.

3. Select the teaming mode for the team being configured (none, ALB, AFT, FEC, GEC, 3AD). NOTE: Adaptive Load Balancing (ALB) encompasses Adapter Fault Tolerance (AFT) as well.

4. Some users may wish to configure Advanced Options for a team. Summary of configuration options (NOTE: one tick = 50 milliseconds):

- Probe mode (on/off) - Determines whether ANS probes are used or not.

- Probes addr mode (Broadcast / Multicast) - Determines whether ANS probes will be sent as broadcast or multicast packets.

- Check timeout (ticks) - After sending a probe, n ticks will expire before the system verifies that the probe was sent to the other members.

- Send time (ticks) - Interval time between probes.

- Max retry count - Number of probe retry bursts to transmit if the first probe was not received by team members.

- Receive timeout (ticks) - After sending a probe retry burst, n ticks will expire before checking to see if the retry probe was transmitted to other team members.

- Rx back cycles - Number of probe back cycles to check (determines if probes from previous cycles are valid).

- Balance interval (ticks) - Refresh time of load balancing tables (ALB, FEC).

- Join Individuals (yes/no) - In 802.3ad mode (3AD), determines whether members should be joined together into an aggregator (when those members do not receive answers from the link partner; for example, when connected to a switch that doesn't support 802.3ad).

- Aggregator Selection (bandwidth/count) - In 802.3ad mode (3AD), determines whether the active aggregator is selected according to the total bandwidth of the candidate aggregator links, or according to the number of links.

NOTE: Most users will not need to configure advanced options. Unless necessary, do not modify these settings.

#### Adding virtual adapters

Follow the steps below to add a virtual adapter.

1. In netcfg, select "Add New LAN Adapter" from the Hardware menu.

2. The virtual adapter created in the previous steps will be highlighted. For example:

Ethernet-iANS Virtual Adapter with VLAN - PCI Slot 0 Bus 1

If several virtual adapters were added in the previous installation stage, they will appear with sequential bus numbers.

3. Choose the networking protocol to be installed on the virtual adapter. NOTE: Although iANS is presented by netcfg as an available protocol, iANS cannot be installed and configured over itself on a virtual adapter.

#### Examples

These examples summarize the actions needed to build two typical topologies:

For a team without VLAN and with several members:

1. Add the first member using "Add New LAN Adapter".

2. Choose teaming modes as desired, and set VLAN mode to "off".

3. Add all the other members in the same manner.

4. Add the virtual adapter without vlan using "Add New LAN Adapter".

For a team with VLAN and with several members:

1. Add the first member using "Add New LAN Adapter".

2. Choose teaming modes as desired, and set VLAN mode to "on".

3. Enter the number of VLAN IDs you will use.

4. Add all the other members in the same manner.

5. Add the virtual adapters using "Add New LAN Adapter" from the Hardware menu. For each adapter added you will be prompted for the virtual adapter's VLAN ID. Do not give two virtual adapters in the same team the same VLAN ID.

##### Remarks

- Viewing a team's configuration can be done when selecting the team's protocol, and choosing Protocol > View protocol configuration from the menu.

- Modifying a team's configuration can be done when selecting the team's protocol, and by choosing Protocol > Modify protocol configuration from the menu. Some modifications require a reboot; others do not. See Modifying a Team's Configuration for further information about dynamic configuration options.

- Viewing a virtual adapter can be done when selecting the virtual adapter hardware, and choosing Hardware > View hardware configuration.

- Modifying a virtual adapter with VLAN can be done when selecting the virtual adapter hardware, and choosing Hardware > Modify hardware configuration. This will allow you to change the virtual adapter's VLAN ID.

### Troubleshooting/Technical Notes

Installation ANS usage guidelines Modifying a Team's Configuration Known Issues

### Installation issues

- ANS should only be installed over Intel’s DDI8 driver:
  Add an ANS team as a protocol over Intel’s DDI8 driver (PRO/100+ (eeE8)). Trying to add it over a different driver is possible in netcfg, but ANS will fail to work.

- Don’t add a protocol to an adapter that already has ANS configured above it:
  After adding an ANS team as a protocol over an adapter, netcfg will allow adding other protocols over that adapter. This should be avoided as unexpected results may occur (the system will not “know” which of the 2 protocols to use).

- It is recommended that you set configurations using the graphic interface (in an X environment). There are some limitations when configuring using the console-mode.

- It is recommended that you do not select "Cancel" when in the process of adding an adapter. If you do so, the system may require a reboot.

- Tunable parameters:
  When the ANS package is installed, two system tunable parameters are altered. This is needed in order to support the maximum number of Vlans that can be configured in ANS (64 vlans).
  Parameter Default value Value set when ANS is installed

  FMOD_RESERVE 50 114
  SFNOLIM 64 172
  After removing the ANS package, these parameters retain the values set by ANS. This shouldn’t affect the user, but the parameters can nevertheless be restored to their default values or to any other value using the SCOAdmin System Tuner or using the “idtune” command (use “man idtune” for more info).

### Using Hot Plug and iANS

#### Requirements:

- You will need a Hot Plug controller driver (such as intelhped for Intel Hot Plug Controller) and the latest Unixware 7.1.x Hot Plug patch. (For 7.1.1, use ptf7604c, and for 7.1.0, use ptf7428d or later).

- If you already have the above packages installed, remove them before installing new drivers.

- Install the necessary drivers (DDI8 drivers for adapters and iANS).

- Reboot the system.

- Configure your initial network configuration.

- Install the Hot Plug controller driver and the Unixware Hot Plug patch.

- Reboot to rebuild the kernel.

#### Using Hot Plug with ANS Team Members:

- Set normal team configurations using netcfg.

- To replace a member adapter: remove it from the team using netcfg, power off the slot using the SCOAdmin Hot Plug manager, remove the adapter, insert the new adapter, power on the slot using the SCOAdmin Hot Plug manager, and add the adapter to the team using netcfg.

### Modifying a Team's Configuration

Most changes to an active team can be accomplished dynamically, without rebooting after the change. Other topology changes will require a reboot. View the following table to determine whether your change will require that your system be rebooted before the change will take effect.

Changes that do not require reboot:

- Create a new team

- Add or remove a member to an existing team ("Hot Add")

- Add a virtual adapter to an existing team

- Remove a virtual adapter from an existing team

- Replace a member in an existing team

- Change VLAN ID

- Probe settings (on/off, or other probe parameters)

- 802.3ad advanced settings

- Balance interval (in ticks)

Changes that do require reboot:

- Change teaming mode (ALB, AFT, FEC, GEC, 3AD)

- VLAN on/off

- Probe address mode (broadcast/multicast)

Limitations of dynamic configurations in Hot Plug computers:

- When ANS is used on a computer in which Hot Plug is enabled, the following limitations apply (due to the way netcfg works in these cases):

> 1. Building a team and adding virtual adapters require a reboot of the computer.

> 2. In order to configure the IPX/SPX protocol over a virtual adapter, the user should first configure TCP/IP over it, reboot the system, and only then add the IPX/SPX protocol (the TCP/IP protocol can then be removed).

- These limitations can be avoided by temporarily disabling Hot Plug before doing anything that will create virtual adapter resources in netcfg (i.e. creating a new team or adding vlans when modifying a team).

> Hot Plug can be temporarily disabled using "modadmin -U <HPCD_MODULE>" where HPCD_MODULE is hpcd/intelhpcd or any other module name used for your Hot Plug Controller Driver.

> Hot Plug can be enabled again after configuring all virtual adapters by using "modadmin -l <HPCD_MODULE>.

### ANS Usage Guidelines/recommendations and Limitations

Switching init-states:
When the system is in initstate 1, do not go directly to initstate 3 (using “init 3”). Instead reboot (using “init 6”) and bring the system up to initstate 3. Going directly from initstate 1 to initstate 3 causes some adapters not to open when adding them to netcfg, and therefore hot-add failures might occur.

Throughput limitations:
There is a SCO limitation regarding the maximum TCP/IP throughput of an adapter. Therefore, when using ALB or FEC mode for TCP/IP traffic, we recommend configuring no more than 4 members in a team PER virtual adapter. This is because the virtual adapter’s throughput will not exceed the mentioned SCO limitation so more members will not improve the total throughput.
If the configuration has more than one virtual adapter in a team (in VLAN mode), the throughput of each virtual adapter will vary according to the distribution of the traffic between the virtual adapters. As a rule of thumb, traffic that is equally distributed between all virtual adapters allows better average throughput per virtual adapter.
Examples:
- If you configure a non-VLAN ALB/FEC team (only one virtual adapter), the team should have a maximum of 4 members.
- If you configure an ALB/FEC team with 2 VLANs, a maximum of 8 members should be configured, but their utilization depends on the distribution of traffic between the VAdapters.

Hot-remove in FEC or GEC mode:
When hot-removing a member in FEC or GEC mode, you should also disconnect the member’s link from the switch. If you fail to do this, the switch will keep transmitting to that port (the switch isn’t aware of the member’s removal from the team).

Hot-removing a team’s original primary adapter:
A team’s MAC address is taken from the MAC address of its original primary physical adapter (the adapter that was chosen as primary at boot-time). If this adapter is hot-removed from the team, both the team and the adapter will have the same MAC address and the following warning will be printed to the system log:
“WARNING: the MAC Address ------ is still in use by Team _”.
In such a case, the removed adapter should not be used anywhere in the network until the system is rebooted, or unexpected results may occur.

Running /etc/nd stop on a member:
/etc/nd stop should NOT be executed on a member’s net. If it is executed, the member will become non-operational and you will have to remove it from the team and add it back, or reboot, for the member to become operational again. The following warning will be printed to the system log in this case:
“WARNING: Closed member [PCI Slot -, (---)] which is still attached to a team! It must be removed to become operational again.”

### Known issues

“Error in handshake with adapter’s driver: adapter wasn't opened” error message at boot:
Symptom: At boot time, a member fails to be added to a team, and the following error messages are seen on the screen:
Error in handshake with adapter’s driver: adapter wasn't opened.
[FAILED]: anscfg cmd=add_lower team=… lower=… net=… low_attr=…
Cause: This is a known SCO bug, in which the adapter’s driver isn’t opened.
Possible workarounds: The problem usually disappears after a reboot. If it doesn’t you can try the following: Try to add a different member and reboot if necessary. Try removing some members from netcfg and then adding other members. Reboot if necessary. If none of the above helps, try to remove everything from netcfg, reboot and reconfigure everything.
NOTE: This error message can also be seen when hot adding a member.

Hot-add failures with no apparent reason:
Symptom: Hot adding a member to a team fails. The member will not be configured in netcfg and the user will see the following message:
“Hot add of member ___ failed”.
Look at the system log. It is possible that the hot-add failed because of a legitimate reason (no server adapter in team, adapter that doesn’t support vlan in a vlan-team, etc). In such cases, a relevant error message will be seen in the log. If there is no message in the log, or if the following message is seen: “Error in handshake with adapter’s driver: adapter wasn't opened”, then this is an anomaly which is a known issue. It happens without ANS as well (in that case, the adapter will be configured in netcfg but will not function).
Possible workarounds:

- Try to hot add the same member again (try several times).

- Try to hot add a different member.

- Try removing some members from netcfg and then adding other members.

- Reboot.

> If none of the above helps, try to remove everything from netcfg, reboot and reconfigure everything.

Repeated messages of members being deactivated and rejoined in VLAN mode:
Symptom: the following messages are printed to the system log repeatedly while in VLAN mode:

“Secondary adapter ___ deactivated / isolated from other team members.”

"Secondary adapter ___ rejoined."

(Note: messages of fail-overs might also appear)

Cause: VLAN IDs are configured in an ANS team, but some of these IDs are not configured in the switch ports to which the team’s members are connected. This causes ANS probes transmitted on these VLAN IDs to be lost, which causes adapters to be deactivated.

Solution: Configure all the switch ports to which the team’s members are connected to be 802.3ac/802.1q VLAN tagged, with all the VLAN IDs corresponding to the ones configured in the team.

Periodic messages of members being deactivated and rejoined in high-stress traffic:
Symptom: the following messages are printed to the system log from time to time when transmitting/receiving high-stress traffic:

“Secondary adapter ___ deactivated / isolated from other team members.”

"Secondary adapter ___ rejoined."

(Note: messages of fail-overs might also appear)

Cause: ANS probes are getting lost.

Possible workaround: Changing the team’s probe-settings may solve the problem or at least limit it:

In netcfg, modify the team’s advanced options (choose team, protocol->Modify protocol configuration, OK, Advanced options) as follows:

Change “Max retry count” to 20 (causes more probe retries).

Change “Receive timeout” to 1 (less time between each retry).

All of the ANS configuration commands fail, “Could not open ctrl device” message printed to the ANS log. (Very rare):
Symptom: At boot time, ANS fails to be configured. The console is filled with [FAIL] messages describing the commands that ANS failed to run. In the ANS log (etc/*ans/data/log), the following messages are also printed:
"[***] The last operation failed. Exited with status =…” and “Could not open ctrl device…”
Possible workaround:

Remove everything from netcfg.

Re-install the ANS package using “pkgadd –d” (no need to remove the package).

Reboot.

Error messages concerning a “wrong state” of an adapter’s driver, followed by “OS Configuration error: message received on the wrong queue. Try to remove the network configuration and reconfigure.” Could also cause “Member is not ready for destruction” messages (very rare):

Symptoms: When booting the system, or when hot-adding a new member, the following messages are printed to the system log:

“eeE8open: wrong state! “

“OS Configuration error: message received on the wrong queue. Try to remove the network configuration and reconfigure.

The “problematic” member will not function correctly. Other system abnormalities might also occur.

If the user doesn’t reconfigure the network, another problem might occur at shutdown or when removing the last virtual adapter from netcfg. The system can enter an infinite loop, printing the message: “Member is not ready for destruction”.
Cause: a known SCO bug preventing an adapter’s driver to be opened correctly, a thing that later causes problems in ANS.

Workaround: The user should remove the entire network configuration from netcfg and then reconfigure it (as the error message says).

If the user doesn’t reconfigure netcfg and enters an infinite loop, he should do so after resetting the system.

#### SCO Contact Information (for base driver support)

SCO Unix
Santa Cruz Operation
425 Encinal St.
Santa Cruz, CA 95060

(800) 726-8649 (408) 425-7222 FAX (408) 458-4227

Go to the SCO website: http://www.sco.com/
Go to Support.
Go to "Search the SCO Support Library Technical Articles".
Search for "Intel" in search box.

For Teaming ANS/VLAN support use the following email address:

unixteam.nics@intel.com