What is going through all those seacables?

iljitsch · May 4, 2024

I saw a video recently about whether the internet is in danger of being physically sabotaged. On land that seems extremely difficult as there’s so much fiber everywhere so you can’t cut it all and/or suspicious activities in unpopulated areas will likely be obvious.

The number of seacables is much more limited and their positions are pretty well known to avoid accidental cutting by fishing boats (which still happens regularly). Still, on www.submarinecablemap.com it shows some 20 different cables between North America and Europe, and North America and Asia. Obviously cutting all of those using a surface vessel would get obvious very fast, and doing it with a submarine would take a good amount of time.

And I actually think the site undercounts the actual number of cables.

But the question this brings up is: why so many of these seacables? Each fiber pair can carry 20+ terabits per second. I seem to remember that the relatively old FLAG cable had two pairs, but newer ones have up to 16 pairs. So that’s something like 1600 terabits per second across the Atlantic and the Pacific.

What’s all that bandwidth used for???

It’s not like we Europeans are watching Netflix from servers in the US or anything like that. I’m sure syncing up all the user generated content on the big platforms uses a lot of bandwidth, but 1600 tbps is several megabits per inhabitant of North America and/or Europe. That can’t be right. So what gives?

koala · May 4, 2024

Redundancy?

hestermofet · May 4, 2024

iljitsch said:
I saw a video recently about whether the internet is in danger of being physically sabotaged. On land that seems extremely difficult as there’s so much fiber everywhere so you can’t cut it all and/or suspicious activities in unpopulated areas will likely be obvious.

Not true. I work for one of the three big telecoms in Canada (i.e. a physically huge country with lots of unpopulated regions). I kid you not, our number one reason for service outages is people stealing cables for copper to sell. We are the leaders nationwide for deploying fiber, so the future looks brighter, but the problem is so bad that we've had to deploy creative solutions like rapid response drones that take photos of the thieves and harass them until the police arrive.

I work in the CISO's directorate, and you don't really think of physical security as being a part of cybersecurity, but I guess your career sometimes takes you places you normally wouldn't expect.

Paladin · May 4, 2024

Plus people are just idiots and destructive. Get a couple beers in them, or just a bad day, and they'll chop things down, shoot stuff or pull on things just to see what they can break. I would not have too much of a bad feeling to run a high voltage, low amperage cable along with fiber when it is anywhere reachable. Use it to power some cameras and to zap idiots that mess with stuff they should not touch.

As for the need for more cables under sea, well they are not just there for anyone to use. Some companies/organizations/governments definitely prefer to run their own rather than lease access from someone else in perpituity. Just because there is more total capacity than we need, that does not mean any single cable run is enough for all the people who would need it if 50% of the others went away. Plus the cost for access would go way up if supply were suddenly restricted.

iljitsch · May 5, 2024

(Sorry, 16 fibers = 8 pairs, not 16 pairs. Although, who knows?)

Obviously redundancy is important, as the cables tend to be pretty vulnerable in shallow water. Not sure about the middle of the ocean. Although if they break there dredging up both ends from a depth of 4 km and then reconnecting them isn’t all that easy!

Obviously redundancy is important. AFAIK in the early days fibers tended to be laid down in a loop so traffic could be rerouted in the opposite direction after a break using SONET/SDH protection. I believe these days individual paths are used and rerouting is done higher up in the protocol stack so it’s also possible to use multiple paths in parallel. In this situation, a single cable can be beneficial, and I believe some relatively recent Google ones are singles without their own redundancy.

Performance is another important consideration. There used to be a cable between Europe, Iceland and Canada which provided good ping times from Iceland to both Europe and North America, but it predated erbium doped amplifiers, so the capacity couldn’t be upgraded from the original 2.5 Gbps so it got replaced by modern cables, but these only go to Europe. So now Iceland has horrible ping times towards North America...

With the advent of high frequency trading it actually became attractive to spend hundreds of millions pulling new shorter cables to shave a few milliseconds off of transatlantic transaction speeds...

But none of this explains 20 different cables across the north atlantic as well as the north pacific. You don’t keep so many parallel cables in operation unless you need the bandwidth.

I also don’t believe that anyone cares enough about having their own cable to spend this kind of money. A government might, but these don’t tend to rule across oceans, if we ignore relatively minor overseas dependencies.

So still wondering what all that traffic is that’s flowing across all those fibers.

As for losing some: in my experience, you don’t even notice a reduction in performance until you lose at least 50% of your bandwidth. So then you notice, and depending on the application, you can lose a good deal more bandwidth before you can no longer reasonably do what you intended to do. And even more before you can’t do at least the critical things anymore.

So I’m thinking the countermeasures against wide scale cable sabotage will be underway long before the impact becomes critical. An exception could be the red sea where so many cables pass through narrow bottlenecks at a fairly shallow depth. Here a surface vessel could basically cut through all the Europe - Asia cables in one go and then have the Houthi’s target the repair ships.

koala · May 5, 2024

I don't know, losing significant capacity from North America to Europe would be massively disruptive. Plus, repairing such cables might require a significant amount of time. I remember reading a few years ago some issues derived by the loss of undersea cable in Asia, but when googling for that I found a lot of news from two months ago:

Bloomberg - Are you a robot?

Loss of four cables was a huge deal, suddenly having 20 doesn't sound so much to me.

Paladin · May 5, 2024

iljitsch said:
(Sorry, 16 fibers = 8 pairs, not 16 pairs. Although, who knows?)

Obviously redundancy is important, as the cables tend to be pretty vulnerable in shallow water. Not sure about the middle of the ocean. Although if they break there dredging up both ends from a depth of 4 km and then reconnecting them isn’t all that easy!

Obviously redundancy is important. AFAIK in the early days fibers tended to be laid down in a loop so traffic could be rerouted in the opposite direction after a break using SONET/SDH protection. I believe these days individual paths are used and rerouting is done higher up in the protocol stack so it’s also possible to use multiple paths in parallel. In this situation, a single cable can be beneficial, and I believe some relatively recent Google ones are singles without their own redundancy.

Performance is another important consideration. There used to be a cable between Europe, Iceland and Canada which provided good ping times from Iceland to both Europe and North America, but it predated erbium doped amplifiers, so the capacity couldn’t be upgraded from the original 2.5 Gbps so it got replaced by modern cables, but these only go to Europe. So now Iceland has horrible ping times towards North America...

With the advent of high frequency trading it actually became attractive to spend hundreds of millions pulling new shorter cables to shave a few milliseconds off of transatlantic transaction speeds...

But none of this explains 20 different cables across the north atlantic as well as the north pacific. You don’t keep so many parallel cables in operation unless you need the bandwidth.

I also don’t believe that anyone cares enough about having their own cable to spend this kind of money. A government might, but these don’t tend to rule across oceans, if we ignore relatively minor overseas dependencies.

Why not? A few are old and in need of replacement or repair in the near future, some are owned by organizations that are reluctant to lease service or dark pairs, some are owned by governments that can't share access, some are just priced high enough that it makes sense to install a new one and start selling access to it.

It's not like a road where you have to buy land from someone to build it. You surely have to get all kinds of permissions and file a ton of paperwork and other stuff but the vast majority of the fixed cost of install is simply parts and labor plus the ongoing maintenance after install. If there is a maket (and presumably there is or they would not exist) then there will be people willing to spend the money to get into that market.
Basically the fact that they exist tells you that they are needed for some reason.

iljitsch said:
So still wondering what all that traffic is that’s flowing across all those fibers.

Unless you get get a passive optical tap on them, you always will.

A lot of it is your average internet traffic but a lot is probably also completely private or they are simply dark for future expansion or redundancy or maintenance window failover and upgrade windows. Basically you can assume that they exist for reasons of money and security. They make money or they need unusual security regardless of making money. That's about it.

iljitsch said:
As for losing some: in my experience, you don’t even notice a reduction in performance until you lose at least 50% of your bandwidth. So then you notice, and depending on the application, you can lose a good deal more bandwidth before you can no longer reasonably do what you intended to do. And even more before you can’t do at least the critical things anymore.

So I’m thinking the countermeasures against wide scale cable sabotage will be underway long before the impact becomes critical. An exception could be the red sea where so many cables pass through narrow bottlenecks at a fairly shallow depth. Here a surface vessel could basically cut through all the Europe - Asia cables in one go and then have the Houthi’s target the repair ships.

I think you might be thinking in terms of internet service when at least some of the fibers are not for general packet traffic delivery. In more specific terms, you can't simply upgrade the gear at either end to shove more traffic through it for more customers because it's not capacity that drives its existence. There will be at least a significant number that are not internet traffic at all so the 20 you find too high or unreasonable, is probably more like 14 or less for general internet traffic. Government owned ones likely do not carry internet stuff at all. Some privately owned ones will be similar in that they don't carry general internet traffic at all, though there may be some of that on it. Their primary purpose is internal traffic for whatever their needs are. So cut it down to 10 or less that are just 'internet stuff' and you need N+1 or better for reasonable reliability so now it's more like 5-7, depending on who owns them, if you factor worst case scenarios and overlapping maintenance outages etc.

That's all speculation but it seems reasonable to me.

Xelas · May 5, 2024

As newer, faster and higher capacity, cables get developed and deployed the old ones aren't simply abandoned immediately, so there are several generations of fiber cabling in use. Cables can break due to anchors being dragged, sabotage, but many break due to undersea landslides, earthquakes, and other such events. At some point, when an older cable breaks it is no longer cost competitive with newer ones to make a repair worthwhile.

DrWebster · May 6, 2024

iljitsch said:
But the question this brings up is: why so many of these seacables? Each fiber pair can carry 20+ terabits per second. I seem to remember that the relatively old FLAG cable had two pairs, but newer ones have up to 16 pairs. So that’s something like 1600 terabits per second across the Atlantic and the Pacific.

This assumes that all of the fiber is for general Internet traffic -- this is not true. Companies will lease fiber for direct or dedicated connections (think HFT, where latency is king, or large corporate clients who want a non-Internet path for the traffic between their campuses).

iljitsch · May 6, 2024

Well, whether the traffic flows over the public internet or over private wavelengths, private "dark" fibers or even completely private cables, there’s still the question of what needs so much bandwidth.

Sure, stuff like me in Europe visiting Ars and typing on this very forum generates traffic across the atlantic, and that adds up. But relative to all the in-continent traffic this has to be a tiny fraction of bandwidth use. Don’t forget that virtually all big web destinations are behind content delivery networks these days.

I don’t know the details of high frequency trading, but I’m pretty sure these transactions are relatively small and it’s not like hundreds of millions of people are doing this. The biggest bandwidth hog in general is streaming video, but that is heavily cached very close to the users, and definitely not being done between continents. (Of course content replication requires non-negligible bandwidth, but again, as far as I can see, not hundreds of terabits.)

Governments? I don’t see it. It’s not like one half of a country is on one continent and the other half on another. And from what I’ve seen with my own eyes, government traffic is very limited relative to ISP traffic in the first place.

Now obviously seacable sabotage would be a very bad day for a good number of people. But IMO, unless adversaries of the free world have a dozen or more cable cutting submarines working together, any sea cable sabotage between Europe and North America or North America and Asia isn’t going to seriously impact anything critical.

This is very different for Africa, South America and Australia, and even more so for remote islands like Iceland, as they depend on only a few seacables, and unlike Europe, NA and Asia don’t have all the critical internet infrastructure such as DNS servers for the DNS root and important top-level domains and financial stuff reachable over land.

It’s interesting to see how more modern cables use more fibers than older cables. With the advent of erbium-doped amplifiers in the late 1990s the amplifiers that are necessary every 100 km or so became signal-agnostic, so you could upgrade the equipment at the ends to more wavelengths and/or higher bitrates per wavelength over existing cables. So assuming the fibers themselves are not a bottleneck, and AFAIK they aren’t, you can upgrade the bandwidth per fiber to the state of the art.

Now obviously the costs of maintaining and when needed, repairing a cable are pretty much the same regardless of the number of fibers, so at some point it’s no longer economical to maintain old cables with few fibers. Still, if you didn’t need the extra bandwidth, keeping a cable with fewer fibers running would generally make more sense than laying a new one with more fibers.

So bandwidth requirements must come in at some point, although in and of themselves they don’t explain the number of cables directly.

Paladin · May 6, 2024

Think of it this way: If you have to string a fiber that costs $100 million whether you string 4 pair or 96 pair, which would you do? If your fiber supports 10 gbit or 1 tbit, and the only cost is a different set of electronics on the two ends, which would you choose? If your fiber you layed 20 years ago is now costing $20 million a year in maintenance, repairs and outage losses, and a new one costs $100 million and will be good for another 20 years of much lower upkeep costs, what do you do? If it takes 5 years or more to actually plan and install such a cable, do you wait until your others are near failure to start that process or get more installed before you really need another? If you need one cable run that is 99.9999% reliable, do you get it by running 2 cables right next to each other or in diverse paths to reduce the chance of a single boat anchor taking out both at the same time? If you can bill customers $15 million a year for traffic/services on the fibers and another $5-10 million a year for dark fiber leases, is that $100 million worth it for the install cost?

For government ones, you're talking about a fixed install cost (my $100 million is completely made up, I have no idea) that is a negligible fraction of a mix of budgets for state security, defense, intelligence, and international relations. Imagine just one scenario: you need to have reliable, high speed and low latency communications from your country to a country around the other side of the world to run military operations and that military deployment is literally hundreds of billions in materials, equipment, and human power and training investments. A few hundred million in cable installs and equipment is nothing to keep that kind of deployment functional and effective. That's not even addressing the other normal day to day stuff that will be going over the fiber aside from the specific military stuff for a given deployment.

Again though, the fact that they exist is a very plain indication that there is a valid use for them. They are likely not all at full use, I should hope not, but they are in use. Otherwise why would people put them there? Did the US build interstate highways to nowhere? No, they built them where they were needed to reach places people needed/wanted to go for various reasons. Even if some of them may seem empty at times or even redundant when you look at a map of them, taking one away has an immediate and measurable effect on the others and the people who depend on being able to use them.

koala · May 6, 2024

iljitsch said:
Now obviously seacable sabotage would be a very bad day for a good number of people. But IMO, unless adversaries of the free world have a dozen or more cable cutting submarines working together, any sea cable sabotage between Europe and North America or North America and Asia isn’t going to seriously impact anything critical.

This is very different for Africa, South America and Australia, and even more so for remote islands like Iceland, as they depend on only a few seacables, and unlike Europe, NA and Asia don’t have all the critical internet infrastructure such as DNS servers for the DNS root and important top-level domains and financial stuff reachable over land.

I dunno, if 4 cables can disrupt America <-> Africa, 20 does not sound like a huge number of cables for America <-> Europe.

When a single AWS AZ has issues, the effects are quite impactful; cables between America <-> Europe would have a far greater impact, and would require much more time to fix. If some of those 20 cables are "unmovable" for private traffic I don't think there's excessive redundancy.

Of course, there could be excessive bandwidth, but what was said above: given the cost of placing a cable, you're going to have the greatest capacity that you can afford, because it's a small part of the cost and, well, if you can delay a few years the next big investment in increase bandwidth, it makes sense.

TL;DR: maybe there's a lot of unused capacity. That is good for future proofing and likely a big must for the required reliability.

DrWebster · May 6, 2024

iljitsch said:
So bandwidth requirements must come in at some point, although in and of themselves they don’t explain the number of cables directly.

You're focused too much on raw bandwidth. The reasons why there are "so many" undersea cables is because of everything other than bandwidth, including what Paladin explained.

In HFT, the companies involved want nothing but the fastest connection, in latency. Bandwidth isn't huge, but they need the link all to themselves for consistency's sake. They cannot use the public Internet because that would involve 1) equipment they can't control, and 2) their traffic would get lumped in with everyone else's, which is also a scenario they can't control. At a bare minimum this means they need a wave of their own, but more often they want entire fiber stands. In at least one case, an entire undersea cable was laid exclusively for HFT.

Another factor: Those cables are all owned by different companies, who each want to make money. Think of it like this: Why do we have UPS, FedEx, etc when the US Postal Service can also deliver packages?

gfunkdave · May 6, 2024

I'll also add that each strand on these subsea cables is split into several logical strands using DWDM (dense wavelength division multiplexing) where the optics on each side send multiple laser signals at different wavelengths so they don't interfere with each other, then pull them apart at the other end. It's sort of like a VLAN for fiber. Each strand can carry dozens of individual wavelengths. Many organizations that buy IRUs to the cable will require their own wavelength for bandwidth or security purposes (in addition to encrypting all traffic that goes through their strand).

The demand for bandwidth increases every year, so never fear that the available capacity is getting used.

Also, the cables can last 20-30 years or more. To upgrade the speed you generally just need to replace the optics on either end.

Xelas · May 7, 2024

gfunkdave said:
Also, the cables can last 20-30 years or more. To upgrade the speed you generally just need to replace the optics on either end.

Is this really true for undersea cables, though? These cables can be thousands of miles long. There are amplifiers laid down with the cable to boost the signal strength, and these amplifiers can run out of capacity as well.
I'm also not sure if the amplifiers allow for DWDM. They use erbium-doped fiber amplifiers which very specifically amplify 1550nm wavelength, which would mean that a dwdwm scheme would not work.

Tobold · May 7, 2024

https://ieeexplore.ieee.org/abstract/document/10116183

I only read the abstract, but it appears they are moving on from DWDM. I know earlier cables definitely supported it from a documentary I watched - they were using 88 frequencies at that point, each at 100Gbps. Many of them were sold as private lines for multinational companies to have their own private network links. Others were contracted to carry Internet traffic.

What is going through all those seacables?

Ars Tribunus Angusticlavius

Ars Tribunus Angusticlavius

Ars Legatus Legionis

Ars Legatus Legionis

Ars Tribunus Angusticlavius

Ars Tribunus Angusticlavius

Ars Legatus Legionis

Ars Praefectus

Ars Praefectus

Ars Tribunus Angusticlavius

Ars Legatus Legionis

Ars Tribunus Angusticlavius

Ars Praefectus

Wise, Aged Ars Veteran

Ars Praefectus

Ars Scholae Palatinae