The Cloud Should Adapt to Its Applications, Not the Other Way Around!

Sachin Ashok†*, Karuna Grewal§*, Talha Waheed†, Gangmuk Lim†, Gerard Matthew†¶, Aditya S Prerepa†,
P. Brighten Godfrey†, Justin Hsu§, Radhika Mittal†
†University of Illinois Urbana-Champaign   §Cornell University   Amazon   *Equal contribution    HotNets ’26
Paper 2-minute video Website by Sachin Ashok
Status quo
Production controllers, such as autoscalers and load balancers, treat each application as an opaque box. They see only basic signals: request rate, latency and CPU. Section 1
Opportunity
Controllers could manage applications much better if they knew application insights, such as call graphs and concurrency models, and tuned their approach to each application.
Problem
Research prototypes use such insights, but each builds its own extractor. These extractors are tied to one system and break as the application changes, so few of these ideas reach production.
Why now
LLM agents can read code and telemetry, work across frameworks, and run existing tools.
Solution
An architectural change: an Application Insight Plane (AIP) between applications and controllers, where LLM agents
  1. extract each insight once,
  2. keep it valid as the application changes, and
  3. share it with every controller.
Sections 2 to 4
Early results
With AIP’s concurrency insight, Envoy cuts p95 latency by up to 30%. Across 116 services in 8 apps, AIP's concurrency model provider extracted a model for each service with source code (100 of them) and returned no model for ones without (16 of them).Section 5
Open questions
Which insights to standardize, how to keep them valid, and how to share them safely. Section 6

This page is interactive. Follow the orange instructions in each section.

1. Production controllers typically operate assuming an opaque-box view

However, a much richer, fine-grained application-level view is possible.

Switch to the application-level view, then pick an insight.

Insight:

Application

an opaque box to its controllers

Basic signals

    2. An Application Insight Plane between applications and controllers

    Click through the four steps to follow one insight.

      3. A few kinds of insights serve many kinds of controllers

      Click a “Use case” to see how a controller uses that insight. Dashed boxes mark pairs that do not apply.

      4. Extracting an insight, and keeping it valid

      The provider extracts again only when a change touches what the insight was built from.

      • Click “Fetch insight”.
      • Make a change, then click “Fetch insight” again.
      • Click a change again to undo it.

      Make a change

      
    
      
    
      
      
      Concurrency model provider
      no insight yet
      
      
        
      
      
      
    
        
  
        
  
        Please note: the provider’s output on this page is precomputed.
        

        


        
  
        5. Preliminary evaluation
        
  
        
    
        Utility
        
      
        • Up to 30% lower p95 latency under overload.
        • Goodput at most 0.4% lower.
        
      
        Switch the setup to see which pool sets the limit.
        
      
        
      
      
        
    
        
    
        Generality
        
      
        • 8 microservice applications from 7 benchmark suites, 116 services in total.
          • 
        
        • 100 services had source code. The provider extracted a concurrency model for each one.
          • 
        
        • 16 services had only deployment files. The provider abstained instead of guessing.
        
      
        LLM:
        GPT-5.5 via OpenAI Codex, the same setup for every application
        Schema:
        one for every service, ConcurrencyModel.v1
        Audit:
        a second agent checks each result against the code and configuration lines it cites
        
      
        Concurrency mechanisms found: 11 kinds across 105 components
        
      
        Servlet thread pool (Tomcat)
        53
        Goroutine per RPC (Go gRPC)
        12
        Thread per connection (C++ Thrift)
        11
        Event loop (Node.js, Netty)
        11
        Thread pool (Python gRPC, Java executors)
        5
        Queue worker (RabbitMQ consumers)
        4
        Goroutine per request (Go HTTP)
        3
        Runtime-managed pool (Java gRPC, ASP.NET Core)
        2
        Process pool (Gunicorn, uWSGI)
        2
        Async runtime (Go)
        1
        PHP worker pool (Apache)
        1
        
    
        
  
        

        



        
  
        6. Open questions
        
  
        
    
        Which insights to standardize?
        The few kinds many controllers need, each with one schema.
        
    
        How to keep them valid?
        Record what each insight was derived from, and check it again when that changes.
        
    
        How to share them safely?
        Give each controller only what it needs, such as a thread limit rather than the code.
        
  
        

        


        
  
        7. The paper in two minutes