I hate OSPF, but I hate BGP more.... added two routers to the network last week. Have three Internet gateways two via BGP and one with OSPF. The BGP gateways have evaporated a dozen times since I added the two new routers and that means that ALL traffic is going through the smallest of our gateways... usually right at peak times too :-( The only time to work on it is 2AM to 3AM, because it's disruptive, but so far no amount of lost sleep has managed to fix it.
Gawd I hate technology :-))
Monday, 28 May 2012
Saturday, 19 May 2012
The O word...
What's the O word ? It's a mysterious thing that no one ever wants to disclose. It happens all the time everywhere but good luck getting anyone to tell you about it. It's a fact of life in our business.
Oversubscription is a fact of life for bandwidth providers everywhere. The economics of what we do dictate that it is a requirement.
I had this conversation with a prospective customer yesterday. It was like a revelation that a provider would admit to over-subbing... but we ALL do it.
Biggest to Smallest we ALL oversub. When you're a small ISP you oversub because the Incumbents charge ludicrous amounts of money for small amounts of bandwidth. Oversubbing when you're small is difficult but you MUST do it to make money. Big ISP's oversub as much because of the cost of equipment as bandwidth. Oversubbing is 'easier' with a larger customer base because probability starts to work in the provider's favor.
When we started in 2005 1Mbps of bandwidth cost us $240. Yes two hundred and forty dollars.
Well, originally it was $160, but when the incumbent realized that we were trying to start something that might in some small way impact their business the price went up by 50% for no apparent reason..... anyways we started at $240 per Megabit per second per month. We sold 1/4 Mbps for $39/month. So we bought it for $240 and sold it for ~$160. Not much of a business plan eh ? Enter Oversubbing. We decided on an Oversub of ~20:1 (which looking back was pretty ugly) So for every $240 we paid out we potentially took in $800. Hey profit... lots of it too!! Wahhoo!!! well, not really. Tower rent, office expense, trucks, gas insurance, EI, WCB, equipment, supplies, maintenance etc etc... what looks like it should be huge gobs of pure money turns out to be not so much in reality.
But as we grew and found new sources for bandwidth the cost per Mbps dropped from $240/Mbps to $160 to under $80.... and now with some strong partnerships we're lower than that - in fact we're close to Urban-center rates in the heart of the Valley... and in turn our customer's speeds increased all along the way.
We still follow the oversub model but we dropped from 20:1 to 18:1 to 12:1 to 10:1. But why, if our cost is so much better do we still oversub ? - the opposite evil of high Feed costs it the Cost of the equipment required to move that amount of data through our network. Where we could get away with a $1,200 radio ($2,400 per point to point link) to move 30Mbps we now need links above 100 Mbps and that $1,200 radio now must be replaced with a $8,000 model :-( And that $8,000 radio will only go 1/2 the distance of the cheaper one (for a bunch of different reasons) so you need twice as many of them.... So that $2,400 x 30Mbps link is now $32,000 x 300Mbps - bandwidth goes up x10 but cost of moving that bandwidth goes up by x15..... add on the fact that you need twice as many towers, and than mean twice as much rent (from $50 to $1,200 depending per tower per month) and you see where the cheap transit cost disappears quickly.
So long story short - we all oversub to some degree because we have to to make the business finacially viable - if you're being told that is not true, and that you get 'always dedicated, not shared' bandwidth at residential prices, then I'd like to talk to your ISP :-))
Oversubscription is a fact of life for bandwidth providers everywhere. The economics of what we do dictate that it is a requirement.
I had this conversation with a prospective customer yesterday. It was like a revelation that a provider would admit to over-subbing... but we ALL do it.
Biggest to Smallest we ALL oversub. When you're a small ISP you oversub because the Incumbents charge ludicrous amounts of money for small amounts of bandwidth. Oversubbing when you're small is difficult but you MUST do it to make money. Big ISP's oversub as much because of the cost of equipment as bandwidth. Oversubbing is 'easier' with a larger customer base because probability starts to work in the provider's favor.
When we started in 2005 1Mbps of bandwidth cost us $240. Yes two hundred and forty dollars.
Well, originally it was $160, but when the incumbent realized that we were trying to start something that might in some small way impact their business the price went up by 50% for no apparent reason..... anyways we started at $240 per Megabit per second per month. We sold 1/4 Mbps for $39/month. So we bought it for $240 and sold it for ~$160. Not much of a business plan eh ? Enter Oversubbing. We decided on an Oversub of ~20:1 (which looking back was pretty ugly) So for every $240 we paid out we potentially took in $800. Hey profit... lots of it too!! Wahhoo!!! well, not really. Tower rent, office expense, trucks, gas insurance, EI, WCB, equipment, supplies, maintenance etc etc... what looks like it should be huge gobs of pure money turns out to be not so much in reality.
But as we grew and found new sources for bandwidth the cost per Mbps dropped from $240/Mbps to $160 to under $80.... and now with some strong partnerships we're lower than that - in fact we're close to Urban-center rates in the heart of the Valley... and in turn our customer's speeds increased all along the way.
We still follow the oversub model but we dropped from 20:1 to 18:1 to 12:1 to 10:1. But why, if our cost is so much better do we still oversub ? - the opposite evil of high Feed costs it the Cost of the equipment required to move that amount of data through our network. Where we could get away with a $1,200 radio ($2,400 per point to point link) to move 30Mbps we now need links above 100 Mbps and that $1,200 radio now must be replaced with a $8,000 model :-( And that $8,000 radio will only go 1/2 the distance of the cheaper one (for a bunch of different reasons) so you need twice as many of them.... So that $2,400 x 30Mbps link is now $32,000 x 300Mbps - bandwidth goes up x10 but cost of moving that bandwidth goes up by x15..... add on the fact that you need twice as many towers, and than mean twice as much rent (from $50 to $1,200 depending per tower per month) and you see where the cheap transit cost disappears quickly.
So long story short - we all oversub to some degree because we have to to make the business finacially viable - if you're being told that is not true, and that you get 'always dedicated, not shared' bandwidth at residential prices, then I'd like to talk to your ISP :-))
Wednesday, 16 May 2012
T'is the Season
Bringing three new towers and 6 PtP links into the network this week. Monday we hung two new radios in Bristol QC. Tuesday (yesterday) replaced two backhauls at McNab. Up at 4 on Monday, spent 6 hours 100' off the ground. Up at 4 on Tuesday getting the gear configured before we started rigging.... Tuesday it was another 6 hours at 210'. Up at three this AM changing OSPF and bringing one of the new backhauls online (300 Mbps at 3 ms - sweet). Today have a miniX to trench 200 feet to a new tower in Horton. 2 guys on that and 2 rigging the tower. I'll only be 50' up this time... but it's going to rain so that'll be truly precious... Tomorrow (Thursday) building out at Marathon... another 8 or 10 hours 110' up. And Friday morning it'll be 3 AM again bringing Marathon and Horton into the OSPF area. At least the blackflies haven't shown up yet.
Sunday, 13 May 2012
So sorry you missed that
To the kind fellow in the White SUV who passed me 5 times on the 417 between Palladium Drive and Arnprior:
I know that, even today, some vehicles don't have Cruise control. I can only surmise that your White Cadillac SUV did not come with that option since you passed me 5 times and then pulled in immediately in front of me only to slow right back down to a speed below what MY cruise control was set for (about 112 km/h) prompting me, naturally, to pass you in turn.
I also realize that passing the same vehicle time and again is frustrating. So I acknowledge how you were motivated to not allow me to pass you that one last time as the divided highway was coming to an end.
I am So sorry that, as you overtook me on the RIGHT at what I can only assume was at least 125 km/h based on how fast you flew by, that you didn't realize that the speed limit had dropped from 100km/h to 90km/h... that you failed to use your turn signal as you cut in front of me (from the right hand lane) and screamed forward well in excess of the speed limit, it's a gosh darn shame that you failed to notice the vehicle directly behind me was an OPP SUV.
I do appreciate that you pulled over nice and far to the Right side of the road so that I could get by as you and your new best friend had a little chat about the technicalities of the highway traffic act.
So sorry you missed that....
I know that, even today, some vehicles don't have Cruise control. I can only surmise that your White Cadillac SUV did not come with that option since you passed me 5 times and then pulled in immediately in front of me only to slow right back down to a speed below what MY cruise control was set for (about 112 km/h) prompting me, naturally, to pass you in turn.
I also realize that passing the same vehicle time and again is frustrating. So I acknowledge how you were motivated to not allow me to pass you that one last time as the divided highway was coming to an end.
I am So sorry that, as you overtook me on the RIGHT at what I can only assume was at least 125 km/h based on how fast you flew by, that you didn't realize that the speed limit had dropped from 100km/h to 90km/h... that you failed to use your turn signal as you cut in front of me (from the right hand lane) and screamed forward well in excess of the speed limit, it's a gosh darn shame that you failed to notice the vehicle directly behind me was an OPP SUV.
I do appreciate that you pulled over nice and far to the Right side of the road so that I could get by as you and your new best friend had a little chat about the technicalities of the highway traffic act.
So sorry you missed that....
Saturday, 12 May 2012
Reality Check
After resisting for many years it looks like I'm going to try to write a blog. I don't talk much - so it should be brief :-)) But I do get into some interesting situations, and I spend a LOT of time on the road and see far too many things, and I do shifts as a CSR for our Highspeed Company so this might be brief, disturbing and funny. The first two are guaranteed - the third one is up in the air.
Anyways ...... Hi my name is Ian I'm a long time listener and a first time ... err... wait that's my support group. Sorry. K so first topic. A degree does not ensure common sense...
Call on the help desk - the customer starts the conversation by informing the CSR (Customer Service Representative) that they are an Electrical Engineer and they previously designed telephone systems. Uh oh - you know it's going to be an interesting time when the caller starts by restating their credentials. After listening to the Caller's c.v. the CSR determines that the Caller's DSL Service is not functioning. Conversation goes like this:
CSR: "Can you tell me if the ADSL Light on your modem is on"
Client: "No it's not"
CSR: "OK is it flashing?"
Client: "No"
CSR: "hmm... ok is the power LED on"
Client: "No"
CSR: "OK it looks like your DSL modem has no power... can you check that the Power Adapter is connected to the back of the modem ?"
Client: "Yes it is"
CSR: "OK - and it's plugged in firmly ? ... can you check that the other end is plugged into the wall ?"
Client: "Yes it's plugged in ... I can't check the wall plug though"
CSR: "Ummm - no problem - is it behind the desk or ..."
Client: "No - I can't see under the desk - the power is off so the lights are out"
CSR: "Uhhhh. You mean the lights are off or the power is off ?"
Client: "The power is out - I blew a fuse and don't have another one... I need to go to town but I want to check to see what their hours are so I need to get online before I drive all that way" (It's about 15km from the customer to the nearest hardware store)
Client: "Should I get a flash light?"
CSR: "Uhhhhhhh...."
I kid you not. I heard the conversation myself - I am the CSR....
Anyways ...... Hi my name is Ian I'm a long time listener and a first time ... err... wait that's my support group. Sorry. K so first topic. A degree does not ensure common sense...
Call on the help desk - the customer starts the conversation by informing the CSR (Customer Service Representative) that they are an Electrical Engineer and they previously designed telephone systems. Uh oh - you know it's going to be an interesting time when the caller starts by restating their credentials. After listening to the Caller's c.v. the CSR determines that the Caller's DSL Service is not functioning. Conversation goes like this:
CSR: "Can you tell me if the ADSL Light on your modem is on"
Client: "No it's not"
CSR: "OK is it flashing?"
Client: "No"
CSR: "hmm... ok is the power LED on"
Client: "No"
CSR: "OK it looks like your DSL modem has no power... can you check that the Power Adapter is connected to the back of the modem ?"
Client: "Yes it is"
CSR: "OK - and it's plugged in firmly ? ... can you check that the other end is plugged into the wall ?"
Client: "Yes it's plugged in ... I can't check the wall plug though"
CSR: "Ummm - no problem - is it behind the desk or ..."
Client: "No - I can't see under the desk - the power is off so the lights are out"
CSR: "Uhhhh. You mean the lights are off or the power is off ?"
Client: "The power is out - I blew a fuse and don't have another one... I need to go to town but I want to check to see what their hours are so I need to get online before I drive all that way" (It's about 15km from the customer to the nearest hardware store)
Client: "Should I get a flash light?"
CSR: "Uhhhhhhh...."
I kid you not. I heard the conversation myself - I am the CSR....
Subscribe to:
Posts (Atom)