Argo Rollouts Canary Deployments and Istio xDS

Argo Rollouts does not process application traffic itself.

• Argo Rollouts
  • Scales stable/canary ReplicaSets
  • Changes VirtualService weights
  • Changes Service selectors or DestinationRule subset labels
• Istio
  • Watches those resource changes
  • Recalculates proxy configuration
  • Sends the effective changes to Envoy through xDS

Argo Rollouts
├─ Pod scaling ───────────────> EndpointSlice ─> EDS
├─ VirtualService weight ─────> RDS
├─ Service selector ──────────> Full push + EDS
└─ DestinationRule labels ────> Full push + EDS subset filtering

Assumptions:

• Canary steps: 0% -> 50% -> 100%
• HTTP traffic splitting
• Stable/canary identification: rollouts-pod-template-hash
• Controller event order omitted
  • Kubernetes watches are asynchronous.
  • Istio may merge nearby events through debounce.
• Istio source: commit ab413ac

Common Rollout Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
        - name: rollouts-demo
          image: example.com/rollouts-demo:v2
          ports:
            - name: http
              containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 0
        - pause: { duration: 1m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

• setWeight- Controls the Istio route weight. - Does not directly mean the same percentage of Pods.
• Replica count
  • Controls the capacity of each revision.
  • Can be managed separately with setCanaryScale.

Host-level Traffic Splitting

• Resources
  • Rollout
  • Stable Service
  • Canary Service
  • VirtualService
• Revision selection
  • Stable Service selector → stable ReplicaSet hash
  • Canary Service selector → canary ReplicaSet hash
• Traffic splitting
  • VirtualService routes to two Service hostnames

                       ┌─> stable Service ─> stable Pods
client ─> Istio route ─┤
                       └─> canary Service ─> canary Pods

Configuration

Add the following fields to the common Rollout:

spec:
  strategy:
    canary:
      stableService: rollouts-demo-stable
      canaryService: rollouts-demo-canary
      trafficRouting:
        istio:
          virtualService:
            name: rollouts-demo
            routes:
              - primary

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-stable
spec:
  selector:
    app: rollouts-demo
    # Injected by Rollouts:
    # rollouts-pod-template-hash: <stable-hash>
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-canary
spec:
  selector:
    app: rollouts-demo
    # Injected by Rollouts:
    # rollouts-pod-template-hash: <canary-hash>
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: rollouts-demo
spec:
  hosts:
    - rollouts-demo.example.com
  gateways:
    - rollouts-demo-gateway
  http:
    - name: primary
      route:
        - destination:
            host: rollouts-demo-stable
          weight: 100
        - destination:
            host: rollouts-demo-canary
          weight: 0

Official host-level configuration

Resource Changes by Deployment Phase

Canary starts

• Rollout
  • Creates/scales the canary ReplicaSet.
• Canary Service
  • Selector changes to the canary hash.
• EndpointSlice
  • Canary endpoints change.

Canary 0 -> 50

• Rollout
  • Scales canary Pods.
• EndpointSlice
  • Endpoints change as Pods scale.
• VirtualService
  • Stable: 100 -> 50
  • Canary: 0 -> 50

Canary 50 -> 100

• Rollout
  • Scales canary Pods.
• EndpointSlice
  • Endpoints change as Pods scale.
• VirtualService
  • Stable: 50 -> 0
  • Canary: 50 -> 100

Canary is promoted to stable

• Stable Service
  • Selector changes to the promoted hash.
• VirtualService
  • Stable: 0 -> 100
  • Canary: 100 -> 0
• Old stable ReplicaSet
  • Scaled down after the configured delay.

The final stable 100/canary 0 state is not a rollback:

• The stable Service now selects the new revision.
• The route is reset for the next rollout.

Subset-level Traffic Splitting

• Resources
  • Rollout
  • One Service
  • VirtualService
  • DestinationRule with stable/canary subsets
• Revision selection
  • Stable subset labels → stable ReplicaSet hash
  • Canary subset labels → canary ReplicaSet hash
• Traffic splitting
  • VirtualService routes to two subsets of the same host

                       ┌─> stable subset ─> stable Pods
client ─> Istio route ─┤
                       └─> canary subset ─> canary Pods

Configuration

Add the following fields to the common Rollout:

spec:
  strategy:
    canary:
      trafficRouting:
        istio:
          virtualService:
            name: rollouts-demo
            routes:
              - primary
          destinationRule:
            name: rollouts-demo
            stableSubsetName: stable
            canarySubsetName: canary

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo
spec:
  selector:
    app: rollouts-demo
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: rollouts-demo
spec:
  hosts:
    - rollouts-demo.example.com
  gateways:
    - rollouts-demo-gateway
  http:
    - name: primary
      route:
        - destination:
            host: rollouts-demo
            subset: stable
          weight: 100
        - destination:
            host: rollouts-demo
            subset: canary
          weight: 0
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: rollouts-demo
spec:
  host: rollouts-demo
  subsets:
    - name: stable
      labels:
        app: rollouts-demo
        # Injected by Rollouts:
        # rollouts-pod-template-hash: <stable-hash>
    - name: canary
      labels:
        app: rollouts-demo
        # Injected by Rollouts:
        # rollouts-pod-template-hash: <canary-hash>

Official subset-level configuration

Resource Changes by Deployment Phase

Canary starts

• Rollout
  • Creates/scales the canary ReplicaSet.
• DestinationRule
  • Canary subset label changes to the canary hash.
• EndpointSlice
  • The shared Service gains canary endpoints.

Canary 0 -> 50

• Rollout
  • Scales canary Pods.
• EndpointSlice
  • Shared Service endpoints change.
• VirtualService
  • Stable: 100 -> 50
  • Canary: 0 -> 50

Canary 50 -> 100

• Rollout
  • Scales canary Pods.
• EndpointSlice
  • Shared Service endpoints change.
• VirtualService
  • Stable: 50 -> 0
  • Canary: 50 -> 100

Canary is promoted to stable

• DestinationRule
  • Stable subset label changes to the promoted hash.
• VirtualService
  • Stable: 0 -> 100
  • Canary: 100 -> 0
• Old stable ReplicaSet
  • Scaled down after the configured delay.

Additional Subsets

• Examples
  • legacy
  • experiment
• Behavior
  • The additional subset keeps its configured weight.
  • Its weight is subtracted from the stable share.
• Example
  • Additional: 20
  • setWeight: 50
  • Stable: 30
  • Canary: 50
• Constraint
  • Avoid canary + additional > 100.

xDS Impact by Resource Change

1. EndpointSlice Changes from Pod Scaling

• Trigger
  • ReplicaSet scaling
  • Pod readiness/removal
• Istio flow

EndpointSlice watch
-> convert endpoints to IstioEndpoint
-> update EndpointIndex
-> EDS push debounce
-> build the affected ClusterLoadAssignment
-> send EDS

• Push type
  • Normally IncrementalPush
  • Does not rebuild the global PushContext
• Main xDS change
  • EDS
• Full-push exceptions
  • New service
  • Service account change
  • Headless-service listener change
  • Network topology change
• Propagation scope
  • Proxies subscribing to the affected service/subset cluster
  • Restricted by the effective Sidecar scope

EndpointIndex.UpdateServiceEndpoints returns NoPush, IncrementalPush, or FullPush.

2. VirtualService Weight Changes

• Trigger
  • Argo Rollouts setWeight
• Istio flow

VirtualService watch
-> Full PushRequest
-> debounce
-> rebuild VirtualService index
-> select affected proxies
-> regenerate proxy configuration
-> send effective changes

• Main xDS change
  • RDS weighted_clusters.weight

weighted_clusters:
  clusters:
    - name: outbound|80||rollouts-demo-stable.default.svc.cluster.local
      weight: 50
    - name: outbound|80||rollouts-demo-canary.default.svc.cluster.local
      weight: 50
  total_weight: 100

• Other xDS resources
  • LDS
    • May be regenerated.
    • Usually unchanged.
  • CDS
    • May be regenerated.
    • Usually unchanged.
  • EDS
    • Not affected by a weight-only change.
    • VirtualService is in skippedEdsConfigs .
• Propagation scope
  • Gateway route
    • Selected gateway proxies
  • mesh route - VirtualService exportTo scope - Sidecars importing the configuration and services - Proxies for which the route is relevant

3. Host-level Service Selector Changes

• Trigger
  • Stable/canary hash change
• Important detail
  • The selector itself is not sent to Envoy.
  • Istio stores it in ServiceAttributes.LabelSelectors.

Attributes: model.ServiceAttributes{
    ServiceRegistry: provider.Kubernetes,
    Name:            svc.Name,
    Namespace:       svc.Namespace,
    Labels:          svc.Labels,
    ExportTo:        exportTo,
    LabelSelectors:  svc.Spec.Selector,
}

• Service-model flow

Kubernetes Service watch
-> ConvertService
-> detect LabelSelectors change
-> Full PushRequest for the service hostname
-> rebuild PushContext and proxy scopes
-> regenerate relevant xDS resources

• Endpoint flow

Service selector change
-> Kubernetes EndpointSlice reconciliation
-> EndpointIndex update
-> EDS

• Push behavior
  • Service and EndpointSlice events are independent.
  • They may be merged in one debounce window.
  • A Service update uses a kind.ServiceEntry config key in this xDS path.
  • EDS can therefore be restricted by canSendPartialFullPushes.
• Effective xDS changes
  • EDS
    • Endpoints change after EndpointSlice reconciliation.
  • RDS/LDS/CDS
    • Recalculated during the full push.
    • Usually unchanged when hostname, VIP, and ports stay the same.

ServiceAttributes.Equals compares LabelSelectors.

4. Subset-level DestinationRule Label Changes

• Trigger
  • Stable/canary subset hash change
• Istio flow

DestinationRule watch
-> Full PushRequest
-> rebuild DestinationRule index
-> select affected proxies using current/previous Sidecar scopes
-> regenerate subset clusters
-> compare subset labels with IstioEndpoint.Labels
-> build subset ClusterLoadAssignments
-> send effective changes

• Effective xDS changes
  • EDS
    • Endpoints move between stable/canary subset clusters.
  • CDS
    • Recalculated.
    • Usually unchanged for a labels-only edit.
  • RDS
    • Recalculated.
    • Usually unchanged when host, subset names, and weights stay the same.
• Push behavior
  • DestinationRule is not in skippedEdsConfigs.
  • canSendPartialFullPushes returns false.
  • State-of-the-world EDS may rebuild subscribed resources for each affected proxy.
  • Cache hits and resource comparison can reduce actual work or transmission.
• Propagation scope
  • Proxies whose current or previous Sidecar scope depends on the DestinationRule

Full, Incremental, and Partial Full Pushes

// Can EDS inside a full push be restricted to changed services?
func canSendPartialFullPushes(req *model.PushRequest) bool {
    if req.Forced {
        return false
    }
    for cfg := range req.ConfigsUpdated {
        if skippedEdsConfigs.Contains(cfg.Kind) {
            continue
        }
        if cfg.Kind != kind.ServiceEntry {
            return false
        }
    }
    return true
}

Source: pilot/pkg/xds/eds.go

Full Push

• Rebuilds the PushContext.
• Recalculates proxy-specific configuration.
• Does not mean every xDS resource is transmitted.
  • Unaffected proxies can be filtered.
  • Cached resources can be reused.
  • Unchanged output can be skipped.

Incremental Push

• Normally used for endpoint-only changes.
• Does not rebuild the PushContext.
• Generates EDS for the changed service.

Partial Full EDS

• Not a separate top-level push type.
• EDS optimization inside a full push.
• Rebuilds the PushContext.
• Restricts CLA generation to changed services when possible.

Delta xDS

• Full/incremental
  • Decides what Istio recomputes.
• DELTA_GRPC- Decides how resources are delivered.
• Behavior
  • Sends changed resources.
  • Sends deletions through removed_resources.
  • A full push can produce a small wire diff.
  • Full-push recomputation still occurs.

Comparison

Common behavior during 0 -> 50 -> 100:

• Pod scaling
  • EndpointSlice change
  • EDS change
• VirtualService weight change
  • Full push
  • Effective RDS change

Main difference at canary start and promotion:

• Host-level
  • Changes a Service selector.
• Subset-level
  • Changes DestinationRule subset labels.

Inspection Commands

kubectl argo rollouts get rollout rollouts-demo --watch

kubectl get service,endpointslice,virtualservice,destinationrule \
  -n default --watch

istioctl proxy-config routes <pod> -n <namespace>
istioctl proxy-config clusters <pod> -n <namespace>
istioctl proxy-config endpoints <pod> -n <namespace>
istioctl proxy-status

• Routes
  • Check stable/canary weights.
• Clusters
  • Check stable/canary hosts or subsets.
• Endpoints
  • Check the Pod IPs selected by each cluster.

Operational Notes

• Traffic weights apply to matching requests, not Pod counts.
• A 50/50 weight is statistical, not exact for every small request sample.
• Long-lived HTTP/2, gRPC, or TCP connections may not rebalance immediately.
• Only traffic passing through the Istio data plane follows the VirtualService.
• Rollouts mutates live VirtualService weights and DestinationRule labels.
  • GitOps tools should ignore or coordinate those controller-owned fields.
• Do not rely on the event order of Service, EndpointSlice, VirtualService, and DestinationRule.

For a quick reference:

• Route selection → VirtualService → RDS
• Endpoint membership → EndpointSlice or DestinationRule subset labels → EDS