Andree Toonk

Yandex still at 33% only, this is going to take while. Yandex’s status page now mentions it will waive all of its fees for existing customers till end of the month

State of Yandex Cloud: half its main availability zones are down. ru-central1-b (Sasovo) since 8 Oct, ru-central1-a (Vladimir) since yesterday What’s left runs on zones D and E. Gotta be hard to absorb: Yandex froze new capacity and is telling customers to move to other providers

Yandex Cloud is still largely down: ~30% of its responding IPs remain after two waves of drone strikes. Yandex's own crawlers have gone quiet too, and its status page still lists two zones down, with the platform in emergency mode. watchthenet.com/e/asn/AS200350

This week for Yandex in one video: two waves of drone strikes took down two of Yandex Cloud's availability zones. ~70% of its responding IPs are gone; Yandex confirms the UAV attack and says the platform is in emergency mode. watchthenet.com/e/asn/AS200350

Yandex confirms: "As a result of the UAV attack, the infrastructure of Yandex's data center in Vladimir was damaged. The data center's operation has been completely suspended." Yandex Cloud is now in "emergency mode"; ~70% of its responding IPs are gone. status.yandex.cloud/en/incidents...

Huge outage: two Yandex Cloud availability zones are down after drone strikes on its Sasovo and Vladimir data centers. Yandex's status page confirms the attack and says it's in emergency mode. ~70% of Yandex Cloud's responding IPs are gone. watchthenet.com/e/asn/AS200350

A major outage hit Yandex following a Ukrainian Defense Forces attack on its data center in Sasovo. watchthenet.com/e/org/RU/ORG...

Wondering what's happening with the Internet right now? Watch the Net measures reachability of every routed IPv4 address, 24/7. When a country, city or network drops off, you'll see it, often minutes before the news does. watchthenet.com

Pretty crazy, this video was a one shot Claude prompt (regular Claude not Claude code) from my Claude app on my phone Waited 10min, this was the result. 🤯 All while eating a Sunday lunch in the park

Internet in #Ukraine in September, as seen by watchthenet.com 🇺🇦 Russia went after the network itself: Kyiv data centres, an internet exchange, telecom HQs & the power grid The country as a whole barely moved. Individual networks went dark, and some still aren't back.

toonk.ioWireGuard at 40 Mpps with DPDK, in GoOver the last few months, I’ve been spending a lot of time experimenting with high-performance packet processing in Go. A lot of that work turned into packetio, one Go API for sending and receiving pa...

This weekend’s project: DPDK + WireGuard 🤓 I wired wireguard-go up to my packetio library and got a single tunnel to ~30M pps clean, peaking 40M pps. That’s 70× what kernel WireGuard can do! Wrote it all up here if you’re into that kind of thing: toonk.io/wireguard-dp...

Reports today of a strike on the Cosmonova data center in Kyiv (AS34867). Also visible on WatchTheNet.com: it went fully dark at 07:26 UTC for ~1.5h, with reachability dropping to zero. Back online since ~09:00 UTC 👇

Beautiful Saturday afternoon in downtown Vancouver

New on WatchTheNet: BGP data + replay, overlaid on our active probe data 🎥 Here's a demo video of today's Iraq country wide outage due to exam shutdown visualized. Pretty neat to watch BGP path hunting before it runs out of options and the networks drop offline.

You can see it here watchthenet.com/e/country/IQ watchthenet.com/bgplay.html?...

I'm as social media influencer now 😆

Tailscale@tailscale.com

If short videos, product explainers, or homelab how to’s are your thing, you might consider following us on TikTok (@ tailscale). Our social media manager (who is not an intern or unpaid, btw) says it’s worth a shot. So here goes our attempt at being TikTok people or whatever they say over there.

Russian drones hit Kyiv this morning. Several regional ISPs went offline, and five hours later some still haven't come back. watchthenet.com/e/city/UA-30...

Reviewing outages from overnight, stumbled on the O2 outage in Czech Republic AS5610. Affecting about 1,5M IP addreses watchthenet.com/e/asn/CZ/AS5...

irozhlas.czNěkteré služby byly nedostupné. O2 se od rána potýkalo s výpadky kvůli poškozené sítiPotíže od pondělního rána zaznamenaly stovky uživatelů, vyplývá z dat monitorovacích služeb Výpadky24 a Downdetector.

Also see www.irozhlas.cz/zpravy-domov...

Fun hobby project: "Watch the Net" 🌎 To detect Internet outages. Finally got around to making it available to everyone. It measures the "pulse of the Internet" with active probes and shows outages/reachability across countries, cities and ASNs. watchthenet.com

toonk.ioWatch the Net: measuring the pulse of the InternetToday I'm releasing a fun new project I've been working on called Watch the Net. It's a live view of what's happening on the Internet, based entirely on active measurements. You can drill down by coun...

And if curious, little write up on my blog on how it works and why i built it. toonk.io/blog-watch-t...

Beautiful sunny Sunday in Vancouver today! Also, someone noticed the @tailscale.com logo on my sweater, turns out he was a big fan 🤩

19 Sept

Rough night in Ukraine seen from my probes Odesa: reachability dropped ~30% at 01:06 UTC and hasn't come back. Donetsk: -90% for three hours over night , and it came back ~25% lower than before.

18 Sept

Pretty major Internet disruption in Donetsk Oblast, Ukraine right now. Reachability dropped sharply, likely due to widespread power outages.

18 Sept

Last two days for context. The disruption is clearly visible in both Donetsk and Komsomol's'ke.

17 Sept

for a second there, i was led to believe all of Google was down. Turns out they just blocked my probe IP 🙁 For what it's worth AWS did that within a few hours (weeks ago) hehe well at least my new telegram notifier bot is working 🤓

17 Sept

poking at AS1103. SURF, the Dutch university network, one of the first ASes I ever worked on. you can see the campuses wake up: responding IPs go up in the morning, every weekday. weekends, nothing. and since Sep 1 the peaks went up a lot more. students are back 😄

17 Sept

zooming in on one city: Enschede, home of the University of Twente. same AS1103. every weekday morning the number of responding IPs quadruples, and on weekends it barely moves. you could set a clock by it

15 Sept

My new morning routine: breakfast + coffee, then checking my Internet Weather map to see what broke overnight. ☕️ Really happy with how this is coming together. It’s become a pretty powerful tool, and a lot of fun to use 🤓 Might make this publicly available soon.

15 Sept

in this mornings good news: the Internet in Portugal has been stable today 🎉 This is the view of the last 2 days

14 Sept

Wow, what just happened to Internet in Portugal? AS3243 (MEO), Portugal’s main residential broadband ASN, took a huge hit and briefly dropped to essentially zero from my vantage point. Country-wide reachability fell significantly. Still ongoing. 👀

14 Sept

Yah that was pretty big! Took some of Ghana down with it as well! All back now

toonk.ioHow fast can you actually go on AWS? Pushing packets on the biggest network optimized EC2 instancesIf you've been following along, you know I've spent the last few months making packets go fast in Go. We looked at AF_XDP, DPDK, and most recently Mellanox Direct Verbs, all behind one library, packet...

I rented some of the biggest network-optimized EC2 instances AWS has, including the 600 Gbps c8gn.48xlarge, and pointed a DPDK packet generator at them. Then did the same for the rest of the c8gn family. Some interesting findings 🤓 learnings and link in the thread 🧵

5/ Full write-up, with the tables and how I tested: toonk.io/how-fast-can...

github.comGitHub - atoonk/packetioContribute to atoonk/packetio development by creating an account on GitHub.

6/ Tooling I used is the blast and drop examples in packetio: github.com/atoonk/packe...

3/ The bandwidth numbers are honest, with a catch. Every instance hit its published number, but the 24xlarge only gets to 300 Gbps with jumbo frames. At 1500 bytes it stops at 282. And the 48xlarge's 600 Gbps is two network cards, 300 each. One ENI = 300G. Want 600? Two ENIs.

4/ One core buys you about 2 million packets per second on ENA. On a ConnectX it's ~60M. That's a 30x gap per core, and it's not my code, plain DPDK lands in the same place. You need 24 cores just to push enough packets to find the instance limit.

1/ AWS publishes bandwidth for every instance type. It publishes nothing about packets per second. So I measured it. From the c8gn.12xlarge up, one network interface tops out at ~36 million packets per second. 150G, 300G, 600G, doesn't matter. Same 36M pps, that's the max!

2/ For context: a single 100G ConnectX does 148M pps. So the 300 Gbps instance does about a quarter of the packets of one 100G NIC. If you do networking in software, pps is the number that matters, and it's not on any AWS spec sheet.

One lesson learned the hard way in my ec2 network testing: Turns out the "600 Gbps" on a c8gn.48xlarge deserves an asterisk. One NIC tops out at 304 Gbps, bw_out_allowance_exceeded ticking. AWS shaping me at 300G on a 600G instance?! Turns out, you need two network cards, 300G each!

ooh wow just got a phone call from AWS regarding suspicious usage on my account. Actually really good! I guess that's what you get if you play with these instance types 🤓 c8gn.8xlarge 100 Gbps c8gn.12xlarge 150 c8gn.16xlarge 200 c8gn.24xlarge 300 c8gn.48xlarge 600

anyways, fun is over. Time to shut this down before i bankrupt myself...

apparently getting a c8gn.24xlarge ec2 instances in us-east-1 is impossible :( Giving up after 25min of terraform loops moving my network testing to ca-central-1

12 Sept

TIL while continuing to work with DPDK on EC2: sending a packet through the ENA driver costs ~1,450 CPU cycles. On a ConnectX (mlx5) it's ~50. Same DPDK, ~30x the cost per packet. One core: 2.2 Mpps on ENA vs ~60 Mpps on mlx5. Pretty wild.

10 Sept

Pretty cool to see the start of the school year show up in my Internet measurement project. The University of British Columbia (AS393249) is back this week, and you can clearly see it in responding IPs on campus. Quiet over the long weekend, a bump yesterday, and a peak today. 🎓

10 Sept

leads to an interesting question, do all these students get public ipv4 addresses?