Migrating 200,000 Network Ports: How to Swap SDNs in a Live Public Cloud

In my previous post, I provided an analysis on why legacy Software-Defined Networking (SDN) architectures like OpenStack Neutron break at hyperscale and why we need to replace them. However, architecting and deploying a modern SDN is only half the solution. The second half is the migration itself. How do you take a live public cloud with over 160,000 running VMs, 3,000 bare metal hypervisors and roughly 200,000 virtual ports, and replace its underlying networking without causing catastrophic downtime? This post details the technical methodology, the operational realities required to execute a hyperscale infrastructure migration. This is a real challenge that was also faced by AWS, GCP, Azure and me at VK CLOUD.

To be clear, migrating an environment of this size is a massive, cross-functional effort involving Product Managers, Software Developers, L1/L2 Tech Support, Site Reliability Engineers (SREs), and Solution Architects. As a Lead Solution Architect, I led the urgent migration of our top clients responsible for the majority of the cloud’s revenue. I developed Public API-based migration tools and methodology for these customers. Using the Public API is the safest way to migrate stateful infrastructure. It respects the cloud’s built-in safety checks and quotas, unlike unstable backend database modifications that bypass user validation and risk silent corruption. These tools and the edge-cases discovered during their use then served as the core requirements and blueprint for the internal, automated pipelines used by the SRE team to migrate the remaining thousands of smaller clients seamlessly.

While we will look specifically at the migration from OpenStack Neutron to VK Cloud’s proprietary Sprut SDN, the core methodologies discussed here are not exclusive to OpenStack. The principles of dependency graphing, state reconciliation, and blast-radius reduction can be applied to any large-scale cloud provider or bare-metal migration.

We will also explore why swapping an SDN is not just an IaaS (Infrastructure-as-a-Service) problem. Higher-level PaaS offerings-like Database-as-a-Service (DBaaS) and Kubernetes-as-a-Service (KaaS) - are built directly on top of this networking layer and require specialized care during the transition. PaaS environments often involve complex, dynamically scaling resources and shared multi-tenant networks (like load balancers and ingress controllers) that easily break if the underlying network engine changes unexpectedly.

This post details experience gained in this massive project. We will break it down step-by-step, how we approached the problem, from laying out all network dependencies and fixing Terraform state files, to the politics of convincing enterprise CTOs and CEO to accept scheduled maintenance windows.

Table of Contents

• I. Reality of Major Public Cloud Migrations
• II. The Philosophy of Migration: State Machines vs. Database Intervention
• III. The Server-Level Migration: Hot-Swapping the Dataplane
• IV. The Client-Level Migration: Safe Tooling for VIPs
• V. The Human Element: Selling Downtime to an Angry CTO
• VI. Conclusion & Retrospective
• VII. References & Further Reading

I. Reality of Major Public Cloud Migrations

Replacing the underlying network or control plane of a live cloud is a monumental engineering challenge. It is a high-risk maneuver that vendors avoid for as long as possible, yet the physics of hyperscale growth eventually force everyone’s hand. To understand the operational reality of our migration, we first need to look at why these migrations happen and how the rest of the industry handles them.

Why Hyperscalers Replace Foundational Architectures

Replacing the foundational architecture of a live cloud is rare, nevertheless, almost every major hyperscaler has faced it. When cloud providers decide to execute migrations of this scale, the decision is driven by unavoidable architectural limitations:

1. Hitting the “Flat” Scalability Ceiling: Early clouds started with “flat” networks. Flat architectures are fast to build but fail at scale because broadcast domains become too large, IP exhaustion occurs, and routing tables become too complex to compute. Migrations are required to introduce hierarchical, subnet-based VPC architectures.
2. Distributed State & Reliability: Early control planes relied on centralized databases. As the cloud grows in server and datacenter count, a single database lock causes a cascading failure. Control planes must be rewritten to use highly distributed, eventually-consistent state storage (like Paxos or Raft consensus algorithms) to ensure regional failures don’t take down the entire cloud.
3. Reducing Computational Overhead: Software-based control planes running on the host server steal valuable CPU cycles from the customer. Migrations are driven by the need to push network and storage processing down to the hardware level (via SmartNICs or Data Processing Units), dramatically lowering latency.
4. The Feature-Parity Trap: Architectural debt prevents revenue. If an old SDN cannot natively support IPv6, overlapping IP spaces, or VPC Peering, the vendor loses enterprise clients. Rebuilding the SDN unblocks the product roadmap.
5. Operational Manageability & Blast Radius: At hyperscale, hardware statistically malfunctions or dies every minute. Modern SDNs are redesigned to limit the “blast radius.” If a switch dies, the new architecture ensures it only impacts a strictly isolated micro-segment of the network.This principle of blast-radius reduction applies equally to modern Software-Defined Compute (SDC) and Software-Defined Storage (SDS) architectures.
6. Tenant Isolation and Security: Early clouds were multi-tenant but heavily relied on software firewalls. As cyber threats evolved, rebuilding the control plane was necessary to implement Zero Trust architectures natively at the packet level.

Why VK Cloud Replaced Its Foundational Architecture

As discussed in my previous article, the catalyst for VK Cloud was the “200k port” Neutron disaster. Our cloud had grown to a scale where the central RabbitMQ message bus could no longer synchronize the dataplane state across 3,000 hypervisors, resulting in catastrophic “Full Sync” timeout loops during datacenter recoveries.

The engineering solution was VK Cloud’s proprietary SDN, “Sprut.” By abandoning the fragile message queue in favor of a declarative HTTP REST architecture, the performance gains were massive. Our benchmarks showed a 65% speedup in full network creation, an 87% speedup in network removal, and an 84% speedup in mass deletion (a critical metric for clients running automated Terraform CI/CD environments).

The Hyperscaler Playbook: Externalizing the Risk

When global hyperscalers execute these foundational upgrades, their standard playbook is to externalize the labor and risk. To be fair, at their scale, there is rarely another option. They set a date, provide a script, and tell the customer: “Figure it out, or lose support.” Here are historical cases of hyperscalers handling major migrations similar to VK Cloud’s:

1. AWS: The Retirement of EC2-Classic: In August 2021, AWS announced the deprecation of their original flat network, EC2-Classic. While they provided an automation runbook, AWS explicitly required customers to hunt down their legacy resources, run the scripts, rewrite security groups, and absorb the downtime themselves. There was no managed tool for smoothly replacing a VM’s underlying network without causing that VM to become unavailable for an extended period.
2. Google Cloud: Legacy Networks to VPCs: GCP’s deprecation of their subnet-less “Legacy Networks” is perhaps the starkest contrast to a managed migration. Because legacy networks spanned multiple regions by default, Google’s official documentation literally states: “There is no automated solution to convert multiple regions… Recreate your resources with the same configurations… Delete your old resources.” Enterprise customers were told to build a new network from scratch. For a large enterprise, this is incredibly problematic because different departments typically have varying subnetwork architectures. If the legacy network isn’t perfectly documented (which is almost always the case), it is almost impossible to execute this manually without errors.
3. Azure: ASM to ARM Control Plane Shift: Azure’s shift from Azure Service Management (Classic) to Azure Resource Manager (ARM) forced a massive architectural change. Customers were given hard deadlines to migrate via PowerShell workflows. Crucially, any legacy automation scripts written for the ASM API became useless and had to be completely rewritten into ARM JSON templates.
4. AWS: Xen to Nitro Hypervisor Swap: Moving from legacy Xen to the hardware-offloaded Nitro system required deep OS-level changes. The AWS migration runbook details brutal prerequisites: updating ENA network drivers, editing /etc/fstab to use UUIDs, and modifying GRUB boot parameters. If a client’s custom Linux kernel wasn’t prepared, the migration simply failed.

The consistent thread across all of these cases is the transfer of operational burden. The downtime, the troubleshooting of edge cases, and the overall migration strategy development were dumped squarely onto the clients’ Operations teams.

The Architectural Evolution: Flat Networks to Modern SDNs

A common thread across the hyperscaler cases described above is the replacement of flat networks and aging control plane components. (VK Cloud also used to have a flat network before Neutron, though that transition occurred before I joined the team).

It is worth noting that the reason all hyperscalers started with flat networks was not a lack of experience. Cloud is a capital-intensive business requiring massive hardware investments that become obsolete in just a few years. In this environment, time-to-market and simplicity were paramount. In the early days of the cloud, engineers simply replicated what they knew: physical data centers were mostly flat. SDN was still an emerging paradigm. Overlay networks (like L2 over L3) were not prevalent, and encapsulation was slow and required heavy CPU usage. Running a flat network on physical switches, however, was hardware-accelerated and lightning-fast.

Building a system that can dynamically create thousands of isolated virtual routers and subnets across thousands of physical hosts is an incredibly hard distributed systems problem. In the “Move Fast and Break Things” era, a flat network was the path of least resistance.

There were distinct advantages to these older networking solutions. They offered ultra-low latency with no encapsulation overhead, and debugging was simple: if a packet wasn’t moving, it was a standard L2 issue. However, the disadvantages ultimately proved fatal at scale:

• The “ARP Storm”: As a cloud hits 100,000+ VMs, the amount of broadcast traffic becomes so dense that the network spends all its time processing background noise instead of actual data.An ARP storm occurs when a massive number of devices simultaneously broadcast Address Resolution Protocol requests across a flat L2 network, saturating the switches and crippling overall network throughput.
• Security Risks: If a tenant managed to spoof a MAC address or ARP response, they could potentially intercept traffic from other tenants on the same flat segment.
• Geographic Limits: You cannot stretch a flat L2 network across the globe without it breaking.

Modern SDNs (like OVN and Sprut) were designed to mitigate these exact issues at hyperscale. The advantages are clear: infinite scale allowing millions of tenants with overlapping IP ranges, and deep programmability allowing entire network topologies to change via a single API call without touching a physical wire. Live migration also became seamless; because the network is software, you can move a VM from one physical rack to another and it keeps its IP address because the overlay simply updates its tunnel map.

However, modern SDNs also introduced new complexities:

• MTU Issues: Because you are wrapping a packet inside another packet for the overlay tunnel, the inner packet has to be smaller, causing MTU (Maximum Transmission Unit) fragmentation overhead.We once faced a severe MTU-related bug with a direct L2 channel established between a VK Cloud datacenter in Moscow and an AWS datacenter in Stockholm, which I will detail in a future article.
• Control Plane Reliance: If the SDN control plane hangs, the entire network can go “blind,” even if the physical wires are perfectly fine.

The VK Cloud Difference: A Partnership Approach

As noted earlier, international hyperscalers operate at a magnitude that makes personalized migrations nearly impossible. When AWS shut down EC2-Classic, they were coordinating the migration of 15 years of technical debt across millions of active servers. At that scale, automated self-service is the only viable path.

However, for a regional hyperscaler like VK Cloud (managing several data centers across Russia and Kazakhstan), telling a massive enterprise client to “recreate your resources on your own” is unacceptable.

We chose a partnership approach. We built automated backend pipelines to migrate smaller tenants invisibly. But for our VIP clients-which included global giants like Philip Morris International and top-tier EdTech platforms like Uchi.ru-we engineered surgical, transparent migration tools.

My team and I sat directly with their engineers. We developed the Public API-based tooling, hosted production webinars, and even built Terraform sandbox environments so clients could safely test the migration on dummy infrastructure before touching production. We gathered their feedback, refined the tools in real-time, and fed that data directly back to our internal engineering teams to perfect the mass-migration tool for the rest of the cloud.

This migration required late-night maintenance windows, manipulating Terraform state files, and convincing CTOs that a brief 20-second network disconnect for their VMs was a necessary trade-off for the long-term survival and speed of their infrastructure.

II. The Philosophy of Migration: State Machines vs. Database Intervention

Before writing a single script or scheduling a maintenance window, we had to determine the architectural philosophy of the migration. When moving 200,000 virtual ports, the decision of how to interact with the cloud-via public APIs or through direct database manipulation-dictates the entire risk profile of the project.

The Cloud as a State Machine

At its core, a cloud environment can be viewed as a massive state machine. As shown in the simplified model above, external users interact with a Public API to declare their desired infrastructure state (e.g., “Create a network with this CIDR block”). The Cloud Control Plane accepts these requests, writes the “Target State” to the central database, and continuously works to synchronize the physical hardware with that database.

The safest way to change the state of physical infrastructure is strictly through the Public API. When a request comes through, the Control Plane executes a list of safety checks: it verifies quotas, checks for physical resource availability, validates parameter formatting, and prevents security exploits. This rigorous validation is what guarantees the reliability of the cloud and ensures the database remains uncorrupted.

However, safety comes at a cost. At hyperscale, executing millions of API calls and waiting for validation checks takes significant time. To speed up operations, engineers are often tempted to bypass the Public API and interact directly with the physical infrastructure or private, unrestricted administrative APIs. While faster, this direct intervention removes the safety net, significantly increasing the risk of state corruption.

Mapping Migration to the SDN Layers

This state machine philosophy maps directly onto the architecture of a Software-Defined Network. As established in the previous article, SDN separates the Control Plane from the Data Plane. Crucially, it introduces different levels of access:

• Northbound Interfaces (The Safe Path): This is the Neutron REST API. It is the highest level of abstraction, used by users and network applications. It is slow but mathematically safe.
• Southbound Interfaces (The Hardware Path): This is the protocol (like OpenFlow or OVSDB) used by the controller to configure the physical hardware, such as the Open vSwitch (OVS) on bare-metal hypervisors. Users and administrators are generally not supposed to send requests directly to the Southbound interface.Bypassing the Northbound API to manually inject rules into a Southbound interface creates an immediate desync between the physical reality of the hypervisor and the logical “Target State” stored in the central database.

When deciding how to migrate the network, we had to choose which interface level to target.

The Universal Mechanics of an SDN Migration

Before detailing the specific execution paths, it is crucial to understand what “migrating” virtual machines to a new network actually means.

Despite existing in the same cloud, Neutron and Sprut do not share a common backend. Each SDN is an entirely isolated service with its own database, its own resource space, and its own operational principles. Therefore, you cannot simply update a database flag to transition a network from Neutron to Sprut.

Regardless of whether you use the Northbound API or Southbound database hacks, the fundamental physical reality of the migration requires a three-step cloning process:

1. The Original State: The virtual machine is actively connected to the legacy Neutron network, serving traffic.
2. Cloning the Target State: A parallel, isolated duplicate of the network infrastructure is created in the Sprut SDN. This includes matching subnets, security groups, virtual routers and routing rules.
3. The Port Swap & Cleanup: The VM’s virtual network interface is disconnected from Neutron and forcefully reattached to the newly created Sprut network. This is the only moment during the entire process where network connectivity is lost. Once the VM is successfully operating on Sprut, the legacy Neutron resources are deleted.

While this three-step physical sequence remains the same, the method used to execute it changes drastically depending on which SDN interface layer we target.

The Three Paths of Migration

Based on the access levels described earlier, We defined three internal paths for migrating OpenStack tenants to the Sprut SDN.In OpenStack, a “Tenant” (or Project) is an isolated set of cloud resources, identical to an AWS Account or a GCP Project. Migration was executed strictly on a tenant-by-tenant basis. It is important to note the “-level” postfix here: we are not migrating servers; we are migrating networks at different operational levels.

1. Client-Level Migration: A set of actions performed exclusively through the standard, existing Northbound API (e.g., recreating a virtual router, requesting a new port, reconnecting a VM interface). This approach is the most natural, predictable, and secure. However, because new entities are created via the API, the OpenStack UUIDs of those network resources will change. The guest operating systems must adapt to the switch (e.g., renewing DHCP leases), which often requires coordination with the client.
2. Server-Level Migration: A process performed entirely on the backend by operations engineers. It utilizes service-level access, direct queries to private administrative APIs, and low-level Southbound configurations. The primary goal is for the guest VMs to “notice nothing”-UUIDs remain the same, and no client involvement is required. In theory, this is the ultimate seamless migration. In practice, achieving this safely requires monumental, customized engineering effort that often exceeds the budget and timeline of a critical migration.
3. Hybrid-Level Migration: A blend of the two, utilizing Client-level API scripts for entity creation, but assisted by specific Server-level tools to bypass certain API limitations.

The Decision Matrix and Execution

With a massive cloud footprint and limited engineering resources, we had to prioritize ruthlessly. We developed a strict decision matrix (illustrated above) to determine exactly how and when an OpenStack tenant would be migrated.

Step 1: Analysis and Prioritization We started by listing and ranking all existing OpenStack tenants. Our primary goal was to reduce the “blast radius” of a potential Neutron Full Sync disaster as quickly as possible. The fewer ports left on legacy Neutron, the faster the recovery time for everyone. Therefore, we prioritized the largest projects and our highest-paying VIP clients.

Step 2: Checking for Specialized Entities We systematically analyzed each tenant for complex services. VIP clients rarely use simple IaaS (Infrastructure as a Service) VMs. They utilize complex PaaS environments, DBaaS, shared networks, VPNaaS, Floating IPs, and highly customized solutions. The presence of these entities immediately dictated the technical limitations of the migration.

Step 3: Choosing Priority and Migration Level Based on the analysis, we assigned the tenant to a priority queue (first, second, third) and selected the migration branch. The presence of complex services forced the routing between Server-Level and Client-Level migration.

For the vast majority of our VIP clients, Client-Level Migration was the only viable choice, offering distinct advantages:

• Total Feature Support: It seamlessly handles DBaaS, PaaS, shared networks, and custom solutions because it relies on native API creation.
• Architectural Control: The process was entirely guided by a Solution Architect working directly with the customer’s engineering team.
• The Disadvantage: It introduces friction. It requires convincing a CTO to allocate their engineering resources for an infrastructure upgrade. Additionally, changing UUIDs requires clients to update their Infrastructure-as-Code (Terraform) states.

Conversely, Server-Level Migration was utilized primarily for smaller, simpler tenants.

• The Advantage: It is incredibly fast, preserves all OpenStack UUIDs (saving Terraform configurations), and requires zero involvement from the end-user.
• The Disadvantage: It explicitly breaks if the tenant uses PaaS or shared networks, making it unsuitable for complex enterprise environments.

Step 4: Migration Preparation Preparation diverged entirely depending on the chosen path.

• Server-Level Preparation: We allocated an internal L2/L3 support engineer, scheduled a downtime window, and notified the Solution Architect if the tenant belonged to a VIP client.
• Client-Level Preparation: Preparation was highly collaborative. The customer’s InfoSec team audited our open-source migration scripts. We then trained the customer’s engineers on how to execute the tools, allocated a Solution Architect to oversee the process, and agreed on a strict downtime window. To minimize that window, our scripts pre-created all necessary Sprut subnets, security groups, load balancers, and IPsec tunnels days in advance without impacting the live environment.

Step 5: Migration Execution and Troubleshooting

• Server-Level Execution: L2/L3 engineers autonomously executed the migration using backend utilities, bypassing the Northbound API to preserve OpenStack UUIDs. If problems arose, they followed strict internal runbooks. Because this method manipulates the database directly, rolling back to Neutron in the event of a failure was strictly impossible.
• Client-Level Execution: The customer’s engineer executed the migration steps. During the maintenance window, the only action required was the physical port swap-disconnecting the Neutron port and attaching the pre-created Sprut port. This method’s greatest strength was its rollback capability. If an obscure application-level bug appeared after moving to Sprut, we could instantly roll the VM’s port back to Neutron using the same scripts.This saved us during a rare edge case where Sprut initially mishandled the dual-port connection requirements (control vs. file ports) of legacy FTP traffic. We safely rolled the affected VMs back to Neutron while our SDN developers patched Sprut.

While clients could theoretically run these API scripts in parallel to migrate dozens of VMs in a few seconds, most preferred a sequential, one-VM-at-a-time approach, taking about 20 seconds per server to allow for immediate health checks. Interestingly, as the script repository matured with comprehensive documentation, a shift occurred. Large enterprise clients managing dozens of OpenStack tenants-such as Philip Morris International-became increasingly independent. Once they grew comfortable with the methodology, they began executing migrations entirely on their own, and in some cases, built specialized internal tools based on our scripts. Eventually, simply handing over the repository link was enough for many to self-serve, though our Solution Architects always remained on standby to assist if needed.

Ultimately, every successful migration-regardless of the level chosen-accelerated our goal. It moved critical workloads onto a much faster, declarative SDN, while simultaneously reducing the port-count burden on the legacy Neutron database, making the cloud safer for everyone.

III. The Server-Level Migration: Hot-Swapping the Dataplane

While the Client-level migration utilizing the Northbound API was the safest route for our VIPs, it required active coordination and client engineering effort. For the thousands of smaller tenants remaining in the cloud, we needed a completely invisible backend solution. This is the Server-level migration.

The core concept of a Server-level migration is based on cloning the client’s network infrastructure from the source SDN (Neutron) to the destination SDN (Sprut) in a 1-to-1 ratio. We temporarily create duplicates while preserving the original values of all significant parameters, most importantly the OpenStack UUIDs. By bypassing the standard public APIs, we ensure the guest virtual machines-and the clients managing them-notice absolutely nothing other than a brief network disconnect.

Mapping Migration to the Two Jobs of SDN

In the previous article, we established that a modern cloud SDN performs two primary jobs: Entity Management (Control Plane) and Network Function Virtualization (Data Plane). The Server-level migration is cleanly split into two corresponding phases:

1. Logical Migration: This handles the Entity Management. It involves cloning the cloud entities (Networks, Subnets, Ports, Security Groups) directly in the databases.
2. Physical Migration: This handles the Data Plane. It involves mechanically rewiring the virtual machine’s network interface on the bare-metal hypervisor.

As shown in Figure 4, executing this requires Admin-level access and direct database manipulation (e.g., via SQL CLI), completely bypassing the standard Northbound API. One of the major advantages of this approach is speed; by interacting directly with the databases and network nodes, we bypass the slow RabbitMQ RPC queues that bottleneck legacy Neutron.Avoiding the RPC queue provides a massive speedup, but it is inherently dangerous. Manual database manipulation risks breaking the control loop, corrupting the synchronized state, or leaving the hypervisor with invalid local configurations.

The Logical Migration

The logical migration phase focuses entirely on the cloud databases. Because an SDN encompasses much more than just VM ports-managing routers, firewall rules, and metadata-the backend script (sdn_cp_migrator) must perfectly clone these relationships. The process consists of three main tasks:

1. Analysis of Resources: The utility scans the source SDN to determine if migration is possible. Because Server-level migration is an irreversible process, we cannot perform partial migrations (e.g., migrating only half of a subnet). Once a tenant is migrated, legacy Neutron support for that tenant is permanently blocked. Due to this irreversibility, L2 support engineers often create a mock copy of the client’s topology to perform a “dry run” rehearsal before touching production.

2. Preparation and Transfer: The script injects the cloned state into the Sprut database. A critical challenge during this phase was the migration of Floating IPs (Public IPs). You cannot simply move a public IP from a Neutron edge router to a Sprut edge router without causing severe BGP routing downtime. Previously, we relied on DNS round-robin hacks, but for this migration, we developed a specialized internal REST API handle that allowed administrators to seamlessly transfer the Floating IP bindings between the two SDNs at the backend level.

3. Deletion in the Source SDN: Once the transfer is verified, the logical entities in the Neutron database are purged to prevent the control plane from managing duplicate states.

The Physical Migration: The QEMU/KVM Dataplane

With the databases cloned, we must perform the physical migration. This involves dropping down to the Southbound interface (the Linux shell of the hypervisor) to manually hot-swap the network cables. To understand how we do this without shutting down the VM, we must look at how a virtual machine connects to the network in a QEMU/KVM environment.

When an application inside a VM sends data, the guest OS encapsulates it into an Ethernet frame and pushes it to its virtual driver. QEMU intercepts this write operation in the guest’s memory space and injects it into a tap interface on the host machine.

From the tap interface, the paths diverge depending on the SDN (as seen in Figure 6):

• Neutron’s Path: The frame enters a small Linux bridge (qbr). Here, the host kernel applies iptables rules to enforce OpenStack Security Groups. It then traverses a virtual patch cable (veth pair) into the Open vSwitch integration bridge (br-int), where it is tagged and sent to the tunnel bridge.
• Sprut’s Path: The architecture is significantly simpler. Sprut drops the intermediate Linux bridge entirely. The tap interface plugs directly into the Sprut OVS bridge (br-sprut).Neutron required the intermediate Linux bridge because older versions of OVS did not natively support stateful firewalls. Sprut utilizes modern OVS connection tracking (conntrack), allowing security groups to be enforced directly within the OpenFlow tables.

Step-by-Step: Hot-Swapping the Port

The physical migration (sdn_dp_migrator) executes the swap in three stages.

Stage 1: Preparation

Thanks to the logical migration phase, the target Sprut OVS bridge and network topology already exist on the hypervisor. The VM is currently operating on the original Neutron path.

Stage 2: The Switch

The script executes a sequence of strict Southbound commands:

• 2.1: It forcefully unplugs the VM’s tap interface from the legacy Linux bridge. (Network connectivity drops here).
• 2.2: It instantly plugs the tap interface into br-sprut.
• 2.3: It calls the Sprut API to officially bind the port to the VM in the control plane.
• 2.4: It calls the Neutron API to unbind the old port. (Network connectivity is restored).

Stage 3: The Cleanup

With the VM successfully passing traffic over Sprut, the script cleans up the host hypervisor. It detaches the veth pairs (3.1, 3.4), disables and deletes the Linux bridge (3.2, 3.3), and completely removes the virtual cables (3.5).

The Inconsistent State (The Libvirt Paradox)

If we stop after Stage 3, the packets flow perfectly, but we have created a ticking time bomb.

While we successfully hot-swapped the network on the hypervisor, the VM’s persistent libvirt XML configuration file (which defines the VM’s hardware for QEMU) still believes the VM is connected to the old Neutron Linux bridge. The server is now in an inconsistent state: the logical database says Sprut, the physical dataplane says Sprut, but the hypervisor’s local definition says Neutron. If the VM were to be hard-rebooted, it would attempt to connect to a Linux bridge that no longer exists, resulting in a fatal crash.

We cannot simply edit the libvirt XML file manually without risking severe hypervisor corruption. The only safe way to resolve this inconsistency is to perform a Live Migration. By instructing Nova to live-migrate the VM to a different bare-metal host, Nova is forced to read the new database state and generate a fresh, accurate libvirt XML file on the destination host, fully finalizing the transition.

Downtime Expectations

Despite operating on the backend, Server-level migration still incurs downtime, split into two distinct windows:

1. Downtime 1 (The Dataplane Swap): The time between unplugging the tap interface from Neutron and the Sprut OpenFlow rules fully propagating. This typically lasts 10 to 30 seconds.
2. Downtime 2 (The Live Migration): The standard memory cutover time required by QEMU when moving the VM between physical hosts to fix the libvirt XML.

The Server-Level Migration Algorithm

To successfully safely manage this across thousands of tenants, the complete Server-level migration utility follows this strict algorithmic checklist:

1. State Retrieval: Queries the source API to download the complete state of the client’s project.
2. Health Check: Validates the state for known issues (e.g., VMs in transitioning states or missing gateway IPs).
3. Soft Limitation Warning: Flags non-critical limitations and prompts the operator to accept the risks.
4. State Compilation: Compiles a clean Target State file suitable for import into the Sprut database.
5. Sprut Discovery: Analyzes the current state of the project in the Sprut SDN (if any prior resources exist) and backs it up.
6. External Network Mapping: Identifies external subnets to correctly remap Floating IPs and external gateways.
7. Execution Plan Generation: Compiles and outputs the strict sequence of transfer operations.
8. Anomaly Detection: Halts and alerts the operator if unknown entities exist in the Sprut target destination.
9. Conflict Resolution: Alerts the operator if UUID conflicts are detected between Neutron and Sprut.
10. Logical Execution: Commits the migration to the databases. If the default security group ID differs, it forcefully recreates it to match the Neutron UUID.
11. Retry Mechanism: If the logical migration drops a connection, the script is fully idempotent and can be safely re-run with the same arguments.
12. Physical Execution: Triggers the sdn_dp_migrator to perform the Southbound tap interface hot-swap.
13. Consistency Enforcement: Triggers the Nova Live Migration to rewrite the libvirt XML and finalize the transition.

IV. The Client-Level Migration: Safe Tooling for VIPs

While the internal backend tools were efficient for smaller workloads, the sheer complexity of our VIP clients’ environments demanded a different approach. We needed surgical tools that respected the OpenStack state machine, utilized the public APIs, and could be audited and executed directly by the client’s engineering teams.

All the tools, scripts, and Terraform test stands developed for this project are available in the public VK Cloud neutron-2-sprut repository.

The Tooling Choice: Why Bash and CLI?

The first architectural decision was choosing the right language for the migration tools. OpenStack is written in Python, and its Python SDKs (openstacksdk) are powerful and expressive. A standard engineering instinct would be to write the migration utility as a comprehensive Python application. We explicitly chose not to do this.

Instead, I developed the entire toolset using Bash, standard OpenStack CLI commands, curl, and jq (for JSON processing).

This decision was driven by client friction. This tool was designed to be run by the client’s DevOps engineers under the scrutiny of their InfoSec teams. Almost all engineers operating within an OpenStack cloud are already intimately familiar with standard OpenStack CLI commands. Providing a Python application meant forcing the client to review thousands of lines of unfamiliar code, manage virtual environments, resolve Python version discrepancies, and handle dependency conflicts on their jump hosts.In enterprise environments, introducing new Python dependencies to a hardened jump host often requires InfoSec approval, which can delay a critical maintenance window by weeks. Bash and Curl are universally available.

Bash scripts utilizing standard CLI commands are transparent. A client engineer can read the script, instantly understand the API calls being made, and trust the execution. There was one caveat: because the OpenStack CLI was not originally designed to manage overlapping SDNs natively, some resources (like creating a base Network) defaulted to Neutron. To bypass this, we used curl to make direct REST API calls for specific Sprut entity creation, while relying on the CLI for everything else.

The Core Mechanics: Migrating the VM Port

The foundation of the migration is the virtual machine port swap. One of the greatest advantages of OpenStack is the ability to create a networking port with a specifically requested MAC and IP address. This capability is the linchpin of our strategy, as it allows us to swap the underlying network without forcing the guest OS to reconfigure its internal network interfaces (e.g., netplan or systemd-networkd). The only action required by the guest OS post-migration is a simple DHCP renewal (dhclient).

We iterated through several versions of this core script. The initial proof-of-concept, migrator.sh, was interactive, prompting the user for the VM name and destination subnets. While it proved the methodology was safe, interactivity is an anti-pattern for mass automation.

The production version, migrator-multiple.sh, accepts a CSV configuration file and processes VMs sequentially. Crucially, it includes idempotency checks. If a migration fails mid-execution, the pre-created ports are not deleted; the script can simply be re-run, detect the existing resources, and pick up where it left off.

Here is the core logic sequence executed by the script:

# 1. Capture original MAC and IP
capture_info_full 

# 2. Create the new port on Sprut with the exact same MAC/IP
create_port_with_mac_ip 

# 3. Disconnect the old port from the Nova instance
detach_source_port 

# 4. Attach the newly created Sprut port
attach_new_port 

# 5. Apply original Security Groups and Floating IPs to the new port
set_security_groups
attach_floating_ip

While most IaaS VMs only have a single port, specialized instances (like Next-Generation Firewalls or virtual routers) often possess multiple interfaces. For these complex, multi-port edge cases, we recommended that clients execute the migration manually via the CLI to ensure proper interface ordering and routing preservation.

Dependency Graphing: Migrating Complex Services

Migrating flat networks and basic VMs is trivial. The true complexity of a client-level migration emerges when dealing with specialized services like VPNaaS (IPsec) and LBaaS (Load Balancers). You cannot simply “copy” an IPsec tunnel.

To automate the migration of a complex service, you must execute strict Dependency Graphing.

As illustrated in the Entity Relation Diagram (Figure 13), when you query the API for an IPSEC SITE CONNECTION, the response doesn’t just contain string fields (like the Pre-Shared Key). It also contains IDs referencing other discrete OpenStack entities (e.g., ikepolicy_id, ipsecpolicy_id, local_ep_group_id, vpnservice_id). Those child entities must be queried, and they often contain their own dependencies.

If you attempt to create the top-level IPsec tunnel on the destination SDN before creating its child policies, the API will reject the request. Therefore, the migration algorithm must collect data top-down, but execute creation bottom-up.

This dependency logic forms a universal 4-stage architecture used in almost all of our service migration scripts, as demonstrated in copy-ipsec-v2.sh and Figure 14:

1. Stage 1: Collection. Query the source Neutron entities and all hierarchical children.
2. Stage 2: Target Discovery. Query the Sprut environment to see if any of these entities have already been created (preventing duplicates and conflicts).
3. Stage 3: Delta Creation. Create the missing child objects (IKE policies, Endpoint groups) in Sprut.
4. Stage 4: Final Assembly. Create the top-level IPsec connection binding the new child objects together.

# Example from copy-ipsec-v2.sh: Stage 4 Assembly via REST
curl_response=$(curl -s -X POST "${sprut_api_base}/vpn/ipsec-site-connections" \
    -H "Content-Type: application/json" \
    -H "X-SDN:SPRUT" \
    -d '{
          "ipsec_site_connection": {
              "psk": "'$psk'",
              "ikepolicy_id": "'$ikepolicy_id'",
              "ipsecpolicy_id": "'$ipsecpolicy_id'",
              "peer_address": "'$peer_address'",
              "name": "'$name'"
          }
        }')

Arguably the most complex entity to map was the Load Balancer (LBaaS). OpenStack Octavia load balancers are provisioned as active/standby “Amphora” VMs. Because spinning up these VMs takes time, we split the logic into two scripts. First, copy-loadbalancer.sh pre-provisions the heavy Amphora instances on Sprut days before the maintenance window. Later, copy-loadbalancer-rules.sh is executed during the downtime to instantly attach the migrated backend VMs to the new Sprut load balancer pools.

Platform Services (PaaS)

Migrating PaaS instances (like Managed Kubernetes or DBaaS) presents a unique challenge. Because PaaS orchestration engines (like Magnum or Trove) maintain their own state databases that synchronize with Nova and Neutron, manually swapping the underlying network ports via scripts immediately corrupts the PaaS controller’s state.

To migrate PaaS safely, the client must use the native PaaS backup/restore tools (e.g., Velero for Kubernetes) to spin up a completely new instance directly on the Sprut network.

This introduces a routing dilemma, highlighted in Figure 15. If a group of VMs and a PaaS instance share the same legacy subnet, moving them asynchronously breaks connectivity. If half of a client’s project (the VMs) is migrated to Sprut, and the other half (the PaaS cluster) remains on Neutron, how do they communicate?

VK Cloud’s “Advanced Router” utilizes BGP peering to seamlessly bridge Sprut and Neutron subnets, allowing partial project migrations.A router cannot bridge two networks possessing the exact same IP address space. To utilize cross-SDN routing during a partial migration, the target Sprut network must be provisioned with a different CIDR block than the original Neutron network. By establishing static routing on the bridge, we allowed clients to decouple their VM migrations from their heavy PaaS rebuilds, mitigating the risk of a massive “all-at-once” cutover.

The IaC Trap: Fixing Terraform State

The final hurdle of the Client-level migration was Infrastructure-as-Code (IaC).

For massive clients like Uchi.ru, deleting and recreating thousands of resources via Terraform was operationally impossible. While our scripts successfully migrated the physical network, Terraform was completely unaware of the changes. If a client ran terraform apply after our scripts finished, Terraform would detect that the VM was no longer attached to the Neutron port defined in its code, and it would violently attempt to recreate the VM to “fix” the discrepancy.

Terraform, like the cloud itself, is a state machine. To prevent recreation of the whole infrastructure, we had to manually manipulate the .tfstate file to force it to recognize the new Sprut UUIDs.

To automate this, I wrote modify_terraform_state.sh. The client updates their .tf files to declare sdn = "sprut" on their network resources. The script then executes a sequence of terraform state rm (to untrack the old Neutron ports) and terraform import commands to pull the newly generated Sprut UUIDs directly into the local state file.

# Removing the legacy tracking
terraform state rm vkcs_networking_port.port_1

# Importing the new Sprut UUID into the state file
terraform import vkcs_networking_port.port_1 <NEW_SPRUT_PORT_UUID>

By manually updating the state file, the client regains declarative control over their newly migrated Sprut infrastructure, completing the migration cycle with zero loss of automation.

The Comprehensive Migration Runbook

While building the underlying automation scripts is a massive engineering hurdle, safely executing them across 200,000 ports requires strict operational discipline. You cannot rely on ad-hoc decision-making during a live infrastructure swap. Just as technical support uses strict runbooks to resolve incidents, we needed to design a definitive “Workbook” for the client-level migration. Each step needed to be clear, sequential, and entirely unambiguous so that any vendor engineer or client DevOps team could follow it flawlessly.

To achieve this, I developed the comprehensive flowchart shown in Figure 16, categorizing the entire migration lifecycle into four distinct, color-coded stages:

• White (Planning & Inventory): This is the foundational prep work. No scripts are executed here. Teams generate configurations, assess the tenant’s inventory, verify quotas, and plan the migration strategy.
• Blue (Zero-Downtime Preparation): This stage can be executed days or weeks in advance. Using the dependency-graphing scripts discussed earlier, engineers copy networks, subnets, routers, security groups, empty LBaaS instances, and IPsec tunnels to the Sprut SDN. Because these operations only create parallel infrastructure, they cause absolutely zero downtime to the active Neutron environment.
• Green (Specialized PaaS/Custom Prep): If the tenant utilizes DBaaS, KaaS, or highly customized deployments, these clusters are rebuilt and synchronized on the newly created Sprut networks during this phase, utilizing the BGP bridging strategies mentioned above.
• Red (The Maintenance Window): This is the critical downtime event. The operations team executes migrator-multiple.sh to forcefully hot-swap the VM ports, runs copy-loadbalancer-rules.sh to attach the newly migrated VMs to the Sprut load balancers, and finally runs modify_terraform_state.sh to update the client’s IaC backend.

By strictly adhering to this color-coded runbook, we decoupled the complex, time-consuming API creation tasks (Blue) from the high-stress port swapping tasks (Red). This ensured that when the actual maintenance window opened, the only thing left to do was flip the switch.

V. The Human Element: Selling Downtime to an Angry CTO

Engineering an elegant migration tool is only half the battle. The other half is business and politics. You can develop the most sophisticated Bash utilities and Server-level migration tools in the world, but due to the technical limitations of distributed systems, it is practically impossible to create a universal, zero-downtime “magic button” that covers every edge case.

Ultimately, migration requires maintenance windows. It requires convincing clients to briefly turn off their infrastructure. For a Solution Architect, this transforms a deeply technical networking problem into an exercise in stakeholder management, negotiation, and de-escalation.

The Myth of the Universal Tool

The first hurdle is internal pragmatism. When a massive architectural flaw is discovered, the initial instinct of management and engineering is to request a “perfect” universal migration tool-a backend system that moves everyone seamlessly without the client ever noticing.

However, a cloud provider is still a business with fixed budgets, roadmaps, and limited engineering hours. An SDN migration is fundamentally a Non-Functional Requirement (NFR).A Non-Functional Requirement (NFR) defines system attributes such as performance, reliability, and maintainability. Because NFRs do not introduce new, marketable features to the end-user, it is historically difficult to convince executives to allocate massive development budgets toward them. We were not adding new, marketable products; we were fixing technical debt. Furthermore, our SREs, developers, and product managers already had their standard operational workloads to manage.

Dedicating the massive budget and time required to develop a flawless, universal tool using a Waterfall development methodology was a dangerous trap. It seemed like a waste of the company’s most valuable resource: time.

The Catalyst: Agility Over Waterfall

The amplifying factor that broke this internal hesitation was the Full Sync disaster. A power loss that lasted only a few seconds resulted in an outage that lasted nearly a day, followed by a long, painful aftermath of manual fixes.

That disaster changed the math. Waiting months to develop a perfect, universal migration tool meant risking another catastrophic outage in the meantime. We needed an Agile approach. The strategy was to build the tools iteratively-starting with Public API scripts-and begin migrating the most critical ports immediately.

By migrating the largest VIP projects first, we aggressively reduced the potential blast radius of a future outage. Every tenant we successfully moved to Sprut reduced the port count in the legacy Neutron database. Furthermore, this iterative process created a continuous feedback loop. Every edge case we discovered while working directly with the clients fed valuable data back to the internal SDN developers, allowing them to refine the Server-level mass-migration tools for the rest of the cloud.

The “Shoot the Messenger” Dynamic

While the iterative approach was the correct engineering decision, it forced us into difficult conversations. After the outage, VIP clients were furiously demanding answers-and rightfully so.

As a Lead Solution Architect, my role is to advise VIP clients on how to build highly available, scalable architectures within our cloud. I am not a member of the datacenter Operations team, and I cannot physically protect their applications from a facility power loss. However, after a major outage, the client associates you with the problem. You become the face of the failure.

When we approached these executives to propose the Sprut migration as the permanent fix, we were met with intense resistance. From the client’s perspective, this was our problem and our fault. They expected a fully automated backend fix that required zero involvement from their engineers.

We had to explain the reality of the cloud “black box.” As an IaaS provider, we supply the virtual hardware and the network roads, but we do not look inside the guest virtual machines.This is a core tenet of the Cloud Shared Responsibility Model. The provider secures the infrastructure of the cloud, but the customer secures and configures the workloads in the cloud. Bypassing the client to manually alter networking paths violates this boundary. We cannot foresee every custom routing table, legacy OS requirement, or specialized application configuration. Attempting to force a universal backend migration without their collaborative involvement risked silently breaking their applications and causing another outage.

Even the largest global hyperscalers do not take on this risk; as discussed in Chapter I, they simply enforce a deadline and force the customer to figure it out. But a “cold, do-it-yourself” approach is unacceptable when dealing with an angry VIP client who just suffered an outage.

Translating Technical Debt into Business Value

To successfully sell the downtime to a skeptical CTO, you cannot simply say, “Please allocate your engineers so we can fix our backend technical debt.” You must translate the migration into tangible business value for them.

My sole goal during these de-escalations was to help the customer’s business. We shifted the narrative away from the past outage and focused entirely on the future capabilities they were unlocking:

• Speed: We demonstrated that Sprut would make their Terraform CI/CD pipelines execute 80% faster, saving developer time.
• New Features: We showed them how the new “Advanced Router” with BGP capabilities would unlock hybrid-cloud architectures they previously couldn’t achieve.
• Ultimate Reliability: We provided technical audits proving that moving to the new declarative SDN made them mathematically immune to the message-queue failures that caused the previous outage.

We backed up this pitch with extreme “white-glove” support. We didn’t just hand them the scripts. We conducted dedicated webinars, provided comprehensive guides, hosted personal meetings, and built customized test stands so their engineers could practice the migration safely.

This collaborative approach completely transformed the dynamic. As we iteratively refined the tools and the documentation grew, a remarkable shift occurred. The friction disappeared. Many of our largest VIP clients became so comfortable with the scripts and the newfound stability of the Sprut SDN that they began independently migrating their remaining projects without our supervision. We turned a crisis of confidence into a successful, collaborative infrastructure overhaul.

VI. Conclusion & Retrospective

Migrating 200,000 network ports across 160,000 virtual machines was a massive test of distributed systems architecture, risk management, and client communication.

As of writing this article, approximately three years have passed since the original “Full Sync” disaster that catalyzed this project. Since completing the migration and fully depreciating legacy Neutron, VK Cloud has not experienced a single major networking outage of that scale. The declarative REST architecture of Sprut proved its worth, fundamentally stabilizing the hyperscale environment.

Looking back at the project, there are several critical architectural and operational takeaways that apply to any massive infrastructure overhaul, whether you are managing 10,000 nodes or 1,000,000:

Key Architectural Takeaways

1. The “Blast Radius” Strategy is Faster Than “Complete Migration”: In hyperscale, you do not need to migrate 100% of your users to stop a cascading failure. By targeting our top-tier VIP clients first, we migrated the workloads that consumed the vast majority of the network ports. This instantly relieved the pressure on the legacy Neutron database, meaning our VIP migrations effectively saved the legacy users who were still waiting in the queue.
2. The Cloud is a State Machine; Respect the API: Backend database hacks (Server-level migrations) are tempting because they are fast and preserve UUIDs. However, at scale, edge cases can potentially cause another outage. Utilizing the Public API (Client-level migrations) is the only way to mathematically guarantee that the physical infrastructure and the logical database remain synchronized through the cloud’s native validation checks.
3. Migrate the State File: You can physically rewire a datacenter, but if you do not provide a mechanism to patch the client’s Infrastructure-as-Code state files (like our .tfstate modifier script), you have essentially destroyed their automation. A modern cloud migration is only successful if the client’s IaC survives it.
4. Enterprise Autonomy is the Ultimate Scalability: You cannot scale a migration if a Solution Architect has to manually hold the hand of every client. By building transparent tools and thorough runbooks, you empower massive organizations to self-serve. Watching enterprise giants like Philip Morris independently migrate their own tenants using our scripts was the ultimate validation of our tooling.

Retrospective: What I Would Do Differently Today

While the migration was a success, hindsight offers a clearer view of the operational friction we faced. If I were to architect this migration program again today, there are two distinct strategic changes I would implement.

1. A “Dual-Track” Tooling Architecture (Go + Bash) Because this tool was designed to be executed by the client’s DevOps engineers on their own jump hosts, we strictly used Bash and curl to avoid dependency management issues (like Python virtual environments). More importantly, Bash is transparent; an enterprise InfoSec team can read and approve a Bash script in minutes.

However, as a former developer, I know we left a massive amount of performance on the table. Later in my tenure, I wrote our internal infrastructure auditing tools in Go (Golang), which allowed for vastly superior error handling, concurrent API execution, and native JSON parsing without external dependencies.

If I did this again, I would officially maintain a dual-track repository. I would offer the transparent Bash scripts for strict-compliance enterprise clients, and a high-performance, pre-compiled Go binary for startups and smaller clients who didn’t require InfoSec audits and simply wanted the migration executed as concurrently and rapidly as possible.

2. Utilizing Migration as an “Infrastructure-as-Code” Upsell While our VIPs heavily utilized Terraform, many legacy clients still relied entirely on “ClickOps”-manually building infrastructure via the web UI. This creates immense technical debt; if the engineer who built the network leaves the company, the infrastructure becomes a fragile black box that no one knows how to manage or restore if it breaks.

Instead of simply executing a 1-to-1 migration of this undocumented infrastructure, we should have utilized our Professional Services team to turn this mandate into an IaC modernization campaign. There are tools capable of reverse-engineering existing OpenStack infrastructure via the API and generating raw Terraform code (I actually authored an internal guide for our VK Cloud team on how to do this).

We could have approached these clients with a combined offer: “Instead of just migrating your ports, our Professional Services team will migrate your entire infrastructure into a fully codified Terraform repository on the new SDN.” It would have been a significantly easier sell to CTOs. Not only would they get the performance benefits of Sprut, but they would immediately eliminate their “ClickOps” technical debt, making their cloud operations exponentially safer and easier to manage going forward.

Replacing the foundational component like SDN of a live public cloud is a difficult task, and standard industry practice usually dictates externalizing work directly onto the customer. By treating the cloud as a strict state machine, graphing our dependencies, and focusing on collaborative tooling rather than forced deprecation, we proved that hyperscale infrastructure upgrades do not have to come at the cost of your clients’ trust.

VII. References & Further Reading

The operational strategies, tooling methodologies, and historical context discussed in this article are synthesized from hands-on hyperscale engineering experience and the following resources.

VK Cloud SDN Migration Tools & Architecture

• Surviving 200,000 Ports: Why OpenStack Neutron Breaks at Hyperscale (Ilya Nyrkov) – The precursor to this post, detailing the mathematical bottlenecks of legacy message queues, the “Full Sync” disaster, and the architectural shift to the declarative Sprut SDN.
• VK Cloud: neutron-2-sprut Migration Repository (GitHub) – The complete open-source suite of Bash-based API migration tools, dependency graphing scripts (IPsec, LBaaS), and Terraform test stands developed for the VIP client migrations.

Hyperscaler Network & Control Plane Migrations

• EC2-Classic is Retiring – Here’s How to Prepare (AWS News Blog) – The timeline and standard industry playbook for Amazon’s deprecation of their original flat network architecture.
• Using Legacy Networks (Google Cloud Documentation) – Official documentation highlighting the manual “recreate your resources” approach required for enterprise multi-region legacy network deprecation on GCP.
• Automation: AWSSupport-MigrateXenToNitro (AWS Systems Manager Docs) – The technical runbook detailing the strict OS-level prerequisites and drivers required for swapping a live hypervisor backend.

Infrastructure-as-Code & State Management

• Terraform CLI: Manipulating State (HashiCorp Developer) – The official documentation for the terraform state rm and terraform import operations utilized to preserve client automation frameworks during the backend SDN swap.
• OpenStack Command-Line Interface Reference (OpenStack Docs) – The primary Northbound API interactions used to safely orchestrate the VM port swaps without corrupting the cloud state machine.