| 1 | |
| 2 |
|
| 3 | ==========================================
|
| 4 | EQL Driver: Serial IP Load Balancing HOWTO
|
| 5 | ==========================================
|
| 6 |
|
| 7 | Simon "Guru Aleph-Null" Janes, [email protected]
|
| 8 |
|
| 9 | v1.1, February 27, 1995
|
| 10 |
|
| 11 | This is the manual for the EQL device driver. EQL is a software device
|
| 12 | that lets you load-balance IP serial links (SLIP or uncompressed PPP)
|
| 13 | to increase your bandwidth. It will not reduce your latency (i.e. ping
|
| 14 | times) except in the case where you already have lots of traffic on
|
| 15 | your link, in which it will help them out. This driver has been tested
|
| 16 | with the 1.1.75 kernel, and is known to have patched cleanly with
|
| 17 | 1.1.86. Some testing with 1.1.92 has been done with the v1.1 patch
|
| 18 | which was only created to patch cleanly in the very latest kernel
|
| 19 | source trees. (Yes, it worked fine.)
|
| 20 |
|
| 21 | 1. Introduction
|
| 22 | ===============
|
| 23 |
|
| 24 | Which is worse? A huge fee for a 56K leased line or two phone lines?
|
| 25 | It's probably the former. If you find yourself craving more bandwidth,
|
| 26 | and have a ISP that is flexible, it is now possible to bind modems
|
| 27 | together to work as one point-to-point link to increase your
|
| 28 | bandwidth. All without having to have a special black box on either
|
| 29 | side.
|
| 30 |
|
| 31 |
|
| 32 | The eql driver has only been tested with the Livingston PortMaster-2e
|
| 33 | terminal server. I do not know if other terminal servers support load-
|
| 34 | balancing, but I do know that the PortMaster does it, and does it
|
| 35 | almost as well as the eql driver seems to do it (-- Unfortunately, in
|
| 36 | my testing so far, the Livingston PortMaster 2e's load-balancing is a
|
| 37 | good 1 to 2 KB/s slower than the test machine working with a 28.8 Kbps
|
| 38 | and 14.4 Kbps connection. However, I am not sure that it really is
|
| 39 | the PortMaster, or if it's Linux's TCP drivers. I'm told that Linux's
|
| 40 | TCP implementation is pretty fast though.--)
|
| 41 |
|
| 42 |
|
| 43 | I suggest to ISPs out there that it would probably be fair to charge
|
| 44 | a load-balancing client 75% of the cost of the second line and 50% of
|
| 45 | the cost of the third line etc...
|
| 46 |
|
| 47 |
|
| 48 | Hey, we can all dream you know...
|
| 49 |
|
| 50 |
|
| 51 | 2. Kernel Configuration
|
| 52 | =======================
|
| 53 |
|
| 54 | Here I describe the general steps of getting a kernel up and working
|
| 55 | with the eql driver. From patching, building, to installing.
|
| 56 |
|
| 57 |
|
| 58 | 2.1. Patching The Kernel
|
| 59 | ------------------------
|
| 60 |
|
| 61 | If you do not have or cannot get a copy of the kernel with the eql
|
| 62 | driver folded into it, get your copy of the driver from
|
| 63 | ftp://slaughter.ncm.com/pub/Linux/LOAD_BALANCING/eql-1.1.tar.gz.
|
| 64 | Unpack this archive someplace obvious like /usr/local/src/. It will
|
| 65 | create the following files::
|
| 66 |
|
| 67 | -rw-r--r-- guru/ncm 198 Jan 19 18:53 1995 eql-1.1/NO-WARRANTY
|
| 68 | -rw-r--r-- guru/ncm 30620 Feb 27 21:40 1995 eql-1.1/eql-1.1.patch
|
| 69 | -rwxr-xr-x guru/ncm 16111 Jan 12 22:29 1995 eql-1.1/eql_enslave
|
| 70 | -rw-r--r-- guru/ncm 2195 Jan 10 21:48 1995 eql-1.1/eql_enslave.c
|
| 71 |
|
| 72 | Unpack a recent kernel (something after 1.1.92) someplace convenient
|
| 73 | like say /usr/src/linux-1.1.92.eql. Use symbolic links to point
|
| 74 | /usr/src/linux to this development directory.
|
| 75 |
|
| 76 |
|
| 77 | Apply the patch by running the commands::
|
| 78 |
|
| 79 | cd /usr/src
|
| 80 | patch </usr/local/src/eql-1.1/eql-1.1.patch
|
| 81 |
|
| 82 |
|
| 83 | 2.2. Building The Kernel
|
| 84 | ------------------------
|
| 85 |
|
| 86 | After patching the kernel, run make config and configure the kernel
|
| 87 | for your hardware.
|
| 88 |
|
| 89 |
|
| 90 | After configuration, make and install according to your habit.
|
| 91 |
|
| 92 |
|
| 93 | 3. Network Configuration
|
| 94 | ========================
|
| 95 |
|
| 96 | So far, I have only used the eql device with the DSLIP SLIP connection
|
| 97 | manager by Matt Dillon (-- "The man who sold his soul to code so much
|
| 98 | so quickly."--) . How you configure it for other "connection"
|
| 99 | managers is up to you. Most other connection managers that I've seen
|
| 100 | don't do a very good job when it comes to handling more than one
|
| 101 | connection.
|
| 102 |
|
| 103 |
|
| 104 | 3.1. /etc/rc.d/rc.inet1
|
| 105 | -----------------------
|
| 106 |
|
| 107 | In rc.inet1, ifconfig the eql device to the IP address you usually use
|
| 108 | for your machine, and the MTU you prefer for your SLIP lines. One
|
| 109 | could argue that MTU should be roughly half the usual size for two
|
| 110 | modems, one-third for three, one-fourth for four, etc... But going
|
| 111 | too far below 296 is probably overkill. Here is an example ifconfig
|
| 112 | command that sets up the eql device::
|
| 113 |
|
| 114 | ifconfig eql 198.67.33.239 mtu 1006
|
| 115 |
|
| 116 | Once the eql device is up and running, add a static default route to
|
| 117 | it in the routing table using the cool new route syntax that makes
|
| 118 | life so much easier::
|
| 119 |
|
| 120 | route add default eql
|
| 121 |
|
| 122 |
|
| 123 | 3.2. Enslaving Devices By Hand
|
| 124 | ------------------------------
|
| 125 |
|
| 126 | Enslaving devices by hand requires two utility programs: eql_enslave
|
| 127 | and eql_emancipate (-- eql_emancipate hasn't been written because when
|
| 128 | an enslaved device "dies", it is automatically taken out of the queue.
|
| 129 | I haven't found a good reason to write it yet... other than for
|
| 130 | completeness, but that isn't a good motivator is it?--)
|
| 131 |
|
| 132 |
|
| 133 | The syntax for enslaving a device is "eql_enslave <master-name>
|
| 134 | <slave-name> <estimated-bps>". Here are some example enslavings::
|
| 135 |
|
| 136 | eql_enslave eql sl0 28800
|
| 137 | eql_enslave eql ppp0 14400
|
| 138 | eql_enslave eql sl1 57600
|
| 139 |
|
| 140 | When you want to free a device from its life of slavery, you can
|
| 141 | either down the device with ifconfig (eql will automatically bury the
|
| 142 | dead slave and remove it from its queue) or use eql_emancipate to free
|
| 143 | it. (-- Or just ifconfig it down, and the eql driver will take it out
|
| 144 | for you.--)::
|
| 145 |
|
| 146 | eql_emancipate eql sl0
|
| 147 | eql_emancipate eql ppp0
|
| 148 | eql_emancipate eql sl1
|
| 149 |
|
| 150 |
|
| 151 | 3.3. DSLIP Configuration for the eql Device
|
| 152 | -------------------------------------------
|
| 153 |
|
| 154 | The general idea is to bring up and keep up as many SLIP connections
|
| 155 | as you need, automatically.
|
| 156 |
|
| 157 |
|
| 158 | 3.3.1. /etc/slip/runslip.conf
|
| 159 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 160 |
|
| 161 | Here is an example runslip.conf::
|
| 162 |
|
| 163 | name sl-line-1
|
| 164 | enabled
|
| 165 | baud 38400
|
| 166 | mtu 576
|
| 167 | ducmd -e /etc/slip/dialout/cua2-288.xp -t 9
|
| 168 | command eql_enslave eql $interface 28800
|
| 169 | address 198.67.33.239
|
| 170 | line /dev/cua2
|
| 171 |
|
| 172 | name sl-line-2
|
| 173 | enabled
|
| 174 | baud 38400
|
| 175 | mtu 576
|
| 176 | ducmd -e /etc/slip/dialout/cua3-288.xp -t 9
|
| 177 | command eql_enslave eql $interface 28800
|
| 178 | address 198.67.33.239
|
| 179 | line /dev/cua3
|
| 180 |
|
| 181 |
|
| 182 | 3.4. Using PPP and the eql Device
|
| 183 | ---------------------------------
|
| 184 |
|
| 185 | I have not yet done any load-balancing testing for PPP devices, mainly
|
| 186 | because I don't have a PPP-connection manager like SLIP has with
|
| 187 | DSLIP. I did find a good tip from LinuxNET:Billy for PPP performance:
|
| 188 | make sure you have asyncmap set to something so that control
|
| 189 | characters are not escaped.
|
| 190 |
|
| 191 |
|
| 192 | I tried to fix up a PPP script/system for redialing lost PPP
|
| 193 | connections for use with the eql driver the weekend of Feb 25-26 '95
|
| 194 | (Hereafter known as the 8-hour PPP Hate Festival). Perhaps later this
|
| 195 | year.
|
| 196 |
|
| 197 |
|
| 198 | 4. About the Slave Scheduler Algorithm
|
| 199 | ======================================
|
| 200 |
|
| 201 | The slave scheduler probably could be replaced with a dozen other
|
| 202 | things and push traffic much faster. The formula in the current set
|
| 203 | up of the driver was tuned to handle slaves with wildly different
|
| 204 | bits-per-second "priorities".
|
| 205 |
|
| 206 |
|
| 207 | All testing I have done was with two 28.8 V.FC modems, one connecting
|
| 208 | at 28800 bps or slower, and the other connecting at 14400 bps all the
|
| 209 | time.
|
| 210 |
|
| 211 |
|
| 212 | One version of the scheduler was able to push 5.3 K/s through the
|
| 213 | 28800 and 14400 connections, but when the priorities on the links were
|
| 214 | very wide apart (57600 vs. 14400) the "faster" modem received all
|
| 215 | traffic and the "slower" modem starved.
|
| 216 |
|
| 217 |
|
| 218 | 5. Testers' Reports
|
| 219 | ===================
|
| 220 |
|
| 221 | Some people have experimented with the eql device with newer
|
| 222 | kernels (than 1.1.75). I have since updated the driver to patch
|
| 223 | cleanly in newer kernels because of the removal of the old "slave-
|
| 224 | balancing" driver config option.
|
| 225 |
|
| 226 |
|
| 227 | - icee from LinuxNET patched 1.1.86 without any rejects and was able
|
| 228 | to boot the kernel and enslave a couple of ISDN PPP links.
|
| 229 |
|
| 230 | 5.1. Randolph Bentson's Test Report
|
| 231 | -----------------------------------
|
| 232 |
|
| 233 | ::
|
| 234 |
|
| 235 | From [email protected] Wed Feb 8 19:08:09 1995
|
| 236 | Date: Tue, 7 Feb 95 22:57 PST
|
| 237 | From: Randolph Bentson <[email protected]>
|
| 238 | To: [email protected]
|
| 239 | Subject: EQL driver tests
|
| 240 |
|
| 241 |
|
| 242 | I have been checking out your eql driver. (Nice work, that!)
|
| 243 | Although you may already done this performance testing, here
|
| 244 | are some data I've discovered.
|
| 245 |
|
| 246 | Randolph Bentson
|
| 247 | [email protected]
|
| 248 |
|
| 249 | ------------------------------------------------------------------
|
| 250 |
|
| 251 |
|
| 252 | A pseudo-device driver, EQL, written by Simon Janes, can be used
|
| 253 | to bundle multiple SLIP connections into what appears to be a
|
| 254 | single connection. This allows one to improve dial-up network
|
| 255 | connectivity gradually, without having to buy expensive DSU/CSU
|
| 256 | hardware and services.
|
| 257 |
|
| 258 | I have done some testing of this software, with two goals in
|
| 259 | mind: first, to ensure it actually works as described and
|
| 260 | second, as a method of exercising my device driver.
|
| 261 |
|
| 262 | The following performance measurements were derived from a set
|
| 263 | of SLIP connections run between two Linux systems (1.1.84) using
|
| 264 | a 486DX2/66 with a Cyclom-8Ys and a 486SLC/40 with a Cyclom-16Y.
|
| 265 | (Ports 0,1,2,3 were used. A later configuration will distribute
|
| 266 | port selection across the different Cirrus chips on the boards.)
|
| 267 | Once a link was established, I timed a binary ftp transfer of
|
| 268 | 289284 bytes of data. If there were no overhead (packet headers,
|
| 269 | inter-character and inter-packet delays, etc.) the transfers
|
| 270 | would take the following times::
|
| 271 |
|
| 272 | bits/sec seconds
|
| 273 | 345600 8.3
|
| 274 | 234600 12.3
|
| 275 | 172800 16.7
|
| 276 | 153600 18.8
|
| 277 | 76800 37.6
|
| 278 | 57600 50.2
|
| 279 | 38400 75.3
|
| 280 | 28800 100.4
|
| 281 | 19200 150.6
|
| 282 | 9600 301.3
|
| 283 |
|
| 284 | A single line running at the lower speeds and with large packets
|
| 285 | comes to within 2% of this. Performance is limited for the higher
|
| 286 | speeds (as predicted by the Cirrus databook) to an aggregate of
|
| 287 | about 160 kbits/sec. The next round of testing will distribute
|
| 288 | the load across two or more Cirrus chips.
|
| 289 |
|
| 290 | The good news is that one gets nearly the full advantage of the
|
| 291 | second, third, and fourth line's bandwidth. (The bad news is
|
| 292 | that the connection establishment seemed fragile for the higher
|
| 293 | speeds. Once established, the connection seemed robust enough.)
|
| 294 |
|
| 295 | ====== ======== === ======== ======= ======= ===
|
| 296 | #lines speed mtu seconds theory actual %of
|
| 297 | kbit/sec duration speed speed max
|
| 298 | ====== ======== === ======== ======= ======= ===
|
| 299 | 3 115200 900 _ 345600
|
| 300 | 3 115200 400 18.1 345600 159825 46
|
| 301 | 2 115200 900 _ 230400
|
| 302 | 2 115200 600 18.1 230400 159825 69
|
| 303 | 2 115200 400 19.3 230400 149888 65
|
| 304 | 4 57600 900 _ 234600
|
| 305 | 4 57600 600 _ 234600
|
| 306 | 4 57600 400 _ 234600
|
| 307 | 3 57600 600 20.9 172800 138413 80
|
| 308 | 3 57600 900 21.2 172800 136455 78
|
| 309 | 3 115200 600 21.7 345600 133311 38
|
| 310 | 3 57600 400 22.5 172800 128571 74
|
| 311 | 4 38400 900 25.2 153600 114795 74
|
| 312 | 4 38400 600 26.4 153600 109577 71
|
| 313 | 4 38400 400 27.3 153600 105965 68
|
| 314 | 2 57600 900 29.1 115200 99410.3 86
|
| 315 | 1 115200 900 30.7 115200 94229.3 81
|
| 316 | 2 57600 600 30.2 115200 95789.4 83
|
| 317 | 3 38400 900 30.3 115200 95473.3 82
|
| 318 | 3 38400 600 31.2 115200 92719.2 80
|
| 319 | 1 115200 600 31.3 115200 92423 80
|
| 320 | 2 57600 400 32.3 115200 89561.6 77
|
| 321 | 1 115200 400 32.8 115200 88196.3 76
|
| 322 | 3 38400 400 33.5 115200 86353.4 74
|
| 323 | 2 38400 900 43.7 76800 66197.7 86
|
| 324 | 2 38400 600 44 76800 65746.4 85
|
| 325 | 2 38400 400 47.2 76800 61289 79
|
| 326 | 4 19200 900 50.8 76800 56945.7 74
|
| 327 | 4 19200 400 53.2 76800 54376.7 70
|
| 328 | 4 19200 600 53.7 76800 53870.4 70
|
| 329 | 1 57600 900 54.6 57600 52982.4 91
|
| 330 | 1 57600 600 56.2 57600 51474 89
|
| 331 | 3 19200 900 60.5 57600 47815.5 83
|
| 332 | 1 57600 400 60.2 57600 48053.8 83
|
| 333 | 3 19200 600 62 57600 46658.7 81
|
| 334 | 3 19200 400 64.7 57600 44711.6 77
|
| 335 | 1 38400 900 79.4 38400 36433.8 94
|
| 336 | 1 38400 600 82.4 38400 35107.3 91
|
| 337 | 2 19200 900 84.4 38400 34275.4 89
|
| 338 | 1 38400 400 86.8 38400 33327.6 86
|
| 339 | 2 19200 600 87.6 38400 33023.3 85
|
| 340 | 2 19200 400 91.2 38400 31719.7 82
|
| 341 | 4 9600 900 94.7 38400 30547.4 79
|
| 342 | 4 9600 400 106 38400 27290.9 71
|
| 343 | 4 9600 600 110 38400 26298.5 68
|
| 344 | 3 9600 900 118 28800 24515.6 85
|
| 345 | 3 9600 600 120 28800 24107 83
|
| 346 | 3 9600 400 131 28800 22082.7 76
|
| 347 | 1 19200 900 155 19200 18663.5 97
|
| 348 | 1 19200 600 161 19200 17968 93
|
| 349 | 1 19200 400 170 19200 17016.7 88
|
| 350 | 2 9600 600 176 19200 16436.6 85
|
| 351 | 2 9600 900 180 19200 16071.3 83
|
| 352 | 2 9600 400 181 19200 15982.5 83
|
| 353 | 1 9600 900 305 9600 9484.72 98
|
| 354 | 1 9600 600 314 9600 9212.87 95
|
| 355 | 1 9600 400 332 9600 8713.37 90
|
| 356 | ====== ======== === ======== ======= ======= ===
|
| 357 |
|
| 358 | 5.2. Anthony Healy's Report
|
| 359 | ---------------------------
|
| 360 |
|
| 361 | ::
|
| 362 |
|
| 363 | Date: Mon, 13 Feb 1995 16:17:29 +1100 (EST)
|
| 364 | From: Antony Healey <[email protected]>
|
| 365 | To: Simon Janes <[email protected]>
|
| 366 | Subject: Re: Load Balancing
|
| 367 |
|
| 368 | Hi Simon,
|
| 369 | I've installed your patch and it works great. I have trialed
|
| 370 | it over twin SL/IP lines, just over null modems, but I was
|
| 371 | able to data at over 48Kb/s [ISDN link -Simon]. I managed a
|
| 372 | transfer of up to 7.5 Kbyte/s on one go, but averaged around
|
| 373 | 6.4 Kbyte/s, which I think is pretty cool. :)
|