Tracking Issue: QUIC testing

[jasnell]

QUIC Testing Check-list.. set up as a tracking issue so that folks can follow along / help if they want. This should also give folks a good idea of the size of the work required still to finish.

When discussing any particular item, please refer to the number for easier reference.

• 1.0.0 Core Protocol Implementation
  • 1.1.0 Endpoint Testing
    • 1.1.1 Endpoint creation with various configurations
    • 1.1.2 IPv4 binding
    • 1.1.3 IPv6 binding
    • 1.1.4 IPv6-only mode
      • TODO: The ergonomics of configuring for ipv6 are... not great. Need to revisit this part of the JS API.
    • 1.1.5 Dual-stack mode
    • 1.1.6 Random port allocation
    • 1.1.7 Specific port binding
    • 1.1.8 Port reuse scenarios
    • 1.1.9 Endpoint can be use for both client and server
  • 1.2.0 Endpoint lifecycle
    • 1.2.1 Graceful shutdown with active sessions
    • 1.2.2 Forced shutdown/destroy
      • 1.2.2.1 With no sessions
      • 1.2.2.2 With sessions, no streams
      • 1.2.2.3 With sessions and streams
    • 1.2.3 Cleanup of sessions on endpoint graceful close
    • 1.2.4 Async.dispose implementation
  • 1.3.0 Endpoint options validation
    • 1.3.1 UDP receive/send buffer sizes
    • 1.3.2 UDP TTL configuration
    • 1.3.3 Token expiration settings
    • 1.3.4 Retry token expiration
    • 1.3.5 Connection limits (per-host, total)
    • 1.3.6 Stateless reset limits
    • 1.3.7 Address LRU cache sizing
    • 1.3.8 Reset token secret handling
    • 1.3.9 Token secret handling
  • 1.4.0 Server mode
    • 1.4.1 Listen configuration
    • 1.4.2 Busy state handling
    • 1.4.3 Rejection of new connections when busy
    • 1.4.4 Multiple session handling
    • 1.4.5 Session acceptance callback
  • 1.5.0 Client mode
    • 1.5.1 Connect to remote endpoint
    • 1.5.2 Multiple concurrent connections
    • 1.5.3 Connection to same host/port
    • 1.5.4 Endpoint reuse across connections
  • 1.6.0 Packet handling
    • 1.6.1 Send packet queuing
    • 1.6.2 Receive packet processing
    • 1.6.3 Packet validation
    • 1.6.4 Malformed packet handling
    • 1.6.5 Version negotiation packets
    • 1.6.6 Retry packets
    • 1.6.7 Stateless reset packets
    • 1.6.8 Immediate connection close
  • 1.7.0 Statistics collection
    • 1.7.1 Bytes sent/received
    • 1.7.2 Packets sent/received
    • 1.7.3 Session counts (client/server)
    • 1.7.4 Server busy count
    • 1.7.5 Timestamp tracking
• 2.0.0 Session Testing
  • 2.1.0 Session establishment
    • 2.1.1 Client-initiated session
    • 2.1.2 Server-side session acceptance
    • 2.1.3 TLS handshake completion
    • 2.1.4 Handshake timeout enforcement
    • 2.1.5 Certificate validation
    • 2.1.6 ALPN negotiation
    • 2.1.7 SNI handling
    • 2.1.8 Early data (0-RTT) support
  • 2.2.0 Session configuration
    • 2.2.1 QUIC version selection
    • 2.2.2 Minimum version enforcement
    • 2.2.3 Transport parameters negotiation
    • 2.2.4 Congestion control algorithm
      • 2.2.4.1 Reno
      • 2.2.4.2 Cubic
      • 2.2.4.3 BBR
    • 2.2.5 Flow control windows
    • 2.2.6 Max payload size
    • 2.2.7 Unacknowledged packet threshold
    • 2.2.8 QLog enablement
  • 2.3.0 Session lifecycle
    • 2.3.1 Normal closure
    • 2.3.2 Graceful shutdown
    • 2.3.3 Immediate destruction
    • 2.3.4 Idle timeout
    • 2.3.5 Closing period behavior
    • 2.3.6 Draining period behavior
    • 2.3.7 Async.dispose implementation
  • 2.4.0 Connection migration
    • 2.4.1 Path validation
    • 2.4.2 Preferred address handling
      • 2.4.2.1 ignore
      • 2.4.2.2 use
      • 2.4.2.3 default
      • 2.4.2.4 advertising preferred address
    • 2.4.3 Active connection ID management
    • 2.4.4 NAT rebinding detection
  • 2.5.0 Session tickets
    • 2.5.1 Ticket generation (server)
    • 2.5.2 Ticket storage (client)
    • 2.5.3 Session resumption with ticket
    • 2.5.4 Ticket expiration
    • 2.5.5 Application data in tickets
  • 2.6.0 Key updates
    • 2.6.1 Initiating key update
    • 2.6.2 Responding to key update
    • 2.6.3 Key logging (for debugging)
    • 2.6.4 TLS tracing
  • 2.7.0 Error handling
    • 2.7.1 Transport errors
    • 2.7.2 Application errors
    • 2.7.3 Version negotiation errors
    • 2.7.4 Connection failures
    • 2.7.5 TLS errors
    • 2.7.6 Certificate verification errors
  • 2.8.0 Statistics collection
    • 2.8.1 Bytes sent/received
    • 2.8.2 Packets sent/received/lost
    • 2.8.3 Stream counts
    • 2.8.4 RTT measurements
    • 2.8.5 Handshake timing
    • 2.8.6 Congestion window tracking
  • 2.9.0 Stateless reset
• 3.0.0 Stream Testing
  • 3.1.0 Stream creation
    • 3.1.1 Client-initiated streams
      • 3.1.1.1 Bidirectional
      • 3.1.1.2 Unidirectional
    • 3.1.2 Server-initiated streams
      • 3.1.2.1 Bidirectional
      • 3.1.2.2 Unidirectional
    • 3.1.3 Pending stream handling
    • 3.1.4 Stream concurrency limits
    • 3.1.5 Opening streams before handshake completion
    • 3.1.6 Sending early data
    • 3.1.7 Receiving early data
  • 3.2.0 Stream directionality
    • 3.2.1 Bidirectional: both sides can send
    • 3.2.2 Unidirectional: single sender
    • 3.2.3 Read/write state combinations
    • 3.2.4 Local vs remote initiated streams
  • 3.3.0 Data transfer
    • 3.3.1 Sending data via outbound queue
    • 3.3.2 Receiving data via inbound queue
    • 3.3.3 In-memory data sources
      • 3.3.3.1 ArrayBuffer/SharedArrayBuffer data sources
      • 3.3.3.2 ArrayBufferView data sources
      • 3.3.3.3 Blob data sources
      • 3.3.6.4 String data sources
    • 3.3.4 Stream data sources
      • 3.3.7.1 Node.js stream data sources
      • 3.3.7.2 WHATWG stream data sources
      • 3.3.7.3 Sync/Async iterator data sources
      • 3.3.7.4 File System / FD data sources
    • 3.3.5 Zero-length writes
    • 3.3.6 Large data transfers
      • 3.3.6.1 Individual
      • 3.3.6.2 Concurrent
    • 3.3.7 Backpressure handling
  • 3.4.0 Flow control
    • 3.4.1 Stream-level flow control
    • 3.4.2 Session-level flow control
    • 3.4.3 Window updates
    • 3.4.4 Blocked stream handling
    • 3.4.5 High watermark behavior
  • 3.5.0 Stream closure
    • 3.5.1 Normal stream end (FIN)
    • 3.5.2 Stream reset by peer
    • 3.5.3 Stop sending requests
    • 3.5.4 Graceful vs forceful close
    • 3.5.5 Bidirectional shutdown
    • 3.5.6 Half-closed states
  • 3.6.0 Stream priority (HTTP/3)
    • 3.6.1 Priority levels (high/default/low)
    • 3.6.2 Incremental vs non-incremental
    • 3.6.3 Priority updates
    • 3.6.4 Priority enforcement
  • 3.7.0 Stream states
    • 3.7.1 Pending state
    • 3.7.2 Open state
    • 3.7.3 Readable/writable states
    • 3.7.4 End-of-stream detection
    • 3.7.5 Final size tracking
  • 3.8.0 Error scenarios
    • 3.8.1 Stream errors
    • 3.8.2 Reset with error codes
    • 3.8.3 Data after FIN
    • 3.8.4 Protocol violations
    • 3.8.5 Timeouts
    • 3.8.6 Loss recovery
  • 3.9.0 Statistics collection
    • 3.9.1 Bytes sent/received
    • 3.9.2 Acknowledgments
    • 3.9.3 Created/destroyed timestamps
• 4.0.0 Datagram Support
  • 4.1.0 Datagram transmission
    • 4.1.1 Send unreliable datagrams
    • 4.1.2 String data encoding
    • 4.1.3 ArrayBufferView data
    • 4.1.4 Size limits
    • 4.1.5 Datagram ID generation
    • 4.1.6 Datagram not supported
  • 4.2.0 Datagram reception
    • 4.2.1 Receive callback
    • 4.2.2 Early datagram handling
    • 4.2.3 Out-of-order delivery
  • 4.3.0 Datagram status
    • 4.3.1 Acknowledgment notification
    • 4.3.2 Loss notification
    • 4.3.3 Status callback
• 5.0.0 Transport Parameters Testing
  • 5.1.0 Local transport params
    • 5.1.1 Initial max stream data (bidi local/remote, uni)
    • 5.1.2 Initial max data
    • 5.1.3 Initial max streams (bidi/uni)
    • 5.1.4 Max idle timeout
    • 5.1.5 Active connection ID limit
    • 5.1.6 ACK delay exponent
    • 5.1.7 Max ACK delay
    • 5.1.8 Max datagram frame size
    • 5.1.9 Preferred address (IPv4/IPv6)
    • 5.1.10 Disable active migration
    • 5.1.11 Max UDP payload size
  • 5.2.0 Remote transport params
    • 5.2.1 Parsing received params
    • 5.2.2 Validation of received params
    • 5.2.3 Incompatible parameters
    • 5.2.4 Parameter negotiation
    • 5.2.5 Client-server agreement
    • 5.2.6 Minimum value selection
    • 5.2.7 Default value fallbacks
• 6.0.0 TLS Integration Testing
  • 6.1.0 Certificate handling
    • 6.1.1 Single certificate
    • 6.1.2 Multiple certificates
    • 6.1.3 Certificate chains
    • 6.1.4 Certificate validation
    • 6.1.5 Self-signed certificates
    • 6.1.6 Expired certificates
    • 6.1.7 Revoked certificates (CRL)
  • 6.2.0 Key management
    • 6.2.1 KeyObject support
    • 6.2.2 CryptoKey support
    • 6.2.3 Multiple keys
    • 6.2.4 Private key verification
    • 6.2.5 Key type validation
  • 6.3.0 CA and trust
    • 6.3.1 Custom CA certificates
    • 6.3.2 CA chain validation
    • 6.3.3 Client certificate verification
  • 6.4.0 Cipher configuration
    • 6.4.1 Default ciphers
    • 6.4.2 Custom cipher strings
    • 6.4.3 Cipher suite selection
    • 6.4.4 Groups/curves configuration
  • 6.5.0 Session resumption
    • 6.5.1 Session ticket generation
    • 6.5.2 Session ticket reuse
    • 6.5.3 0-RTT data with resumption
    • 6.5.4 Ticket rotation
  • 6.6.0 ALPN
    • 6.6.1 Protocol negotiation
    • 6.6.2 Multiple protocols
    • 6.6.3 No common protocol handling
  • 6.7.0 SNI
    • 6.7.1 Server name indication
    • 6.7.2 SNI-based routing
    • 6.7.3 Virtual hosting
  • 6.8.0 Tokens
    • 6.8.1 Retry token validity / timeouts
      • TODO: Works for ipv4, appears to have issues with some ipv6 paths
    • 6.8.2 Regular token validity /timeouts
    • 6.8.3 Stateless reset token validity / timeouts
• 7.0.0 HTTP/3 Application Layer Testing
  • 7.1.0 HTTP/3 basics
    • 7.1.1 Request/response flow
    • 7.1.2 Stream creation for requests
    • 7.1.3 Multiple concurrent requests
  • 7.2.0 Headers
    • 7.2.1 Header encoding/decoding
    • 7.2.2 QPACK compression
    • 7.2.3 Header limits (pairs, length, field section size)
    • 7.2.4 Pseudo-headers (:method, :path, :authority, :scheme)
    • 7.2.5 Early headers / early-hints / 1xx
    • 7.2.6 Regular headers
    • 7.2.7 Trailing headers
    • 7.2.8 Header ordering
    • 7.2.9 Duplicate headers
  • 7.3.0 QPACK
    • 7.3.1 Dynamic table management
    • 7.3.2 Encoder/decoder streams
    • 7.3.3 Table capacity limits
    • 7.3.4 Blocked streams handling
  • 7.4.0 HTTP/3 settings
    • 7.4.1 SETTINGS frame exchange
    • 7.4.2 Max header list size
    • 7.4.3 QPACK settings
    • 7.4.4 Extended CONNECT protocol
  • 7.5.0 Control streams
    • 7.5.1 Bidirectional control stream
    • 7.5.2 Unidirectional control streams
    • 7.5.3 Stream type identification
  • 7.6.0 Priority (extensible priorities)
    • 7.6.1 PRIORITY_UPDATE frames
    • 7.6.2 Urgency parameter
    • 7.6.3 Incremental parameter
• 8.0.0 Error Handling and Edge Cases
  • 8.1.0 Protocol errors
    • 8.1.1 Invalid frames
    • 8.1.2 Stream ID violations
    • 8.1.3 Flow control violations
    • 8.1.4 Connection ID errors
    • 8.1.5 Token validation failures
  • 8.2.0 Resource exhaustion
    • 8.2.1 Maximum streams exceeded
    • 8.2.2 Memory pressure
    • 8.2.3 Connection limits
    • 8.2.4 Packet queue overflow
  • 8.3.0 Network conditions
    • 8.3.1 Packet loss simulation
    • 8.3.2 Packet reordering
    • 8.3.3 Delayed ACKs
    • 8.3.4 Path MTU discovery
    • 8.3.5 ECN marking
  • 8.4.0 Timeout scenarios
    • 8.4.1 Handshake timeout
    • 8.4.2 Idle timeout
    • 8.4.3 Drain timeout
    • 8.4.4 Keep-alive
  • 8.5.0 Malicious input
    • 8.5.1 Oversized packets
    • 8.5.2 Invalid QUIC versions
    • 8.5.3 Replay attacks
    • 8.5.4 Connection ID spoofing
    • 8.5.5 Amplification attack prevention
• 9.0.0 Interoperability Testing
  • 9.1.0 QUIC version compatibility
    • 9.1.1 v1 (RFC 9000)
    • 9.1.2 Version negotiation
    • 9.1.3 Draft version support (if any)
  • 9.2.0 Cross-implementation testing
    • 9.2.1 Against other QUIC implementations
      • 9.2.1.1 quiche
      • 9.2.1.2 msquic
      • 9.2.1.3 picoquic
    • 9.2.2 Public QUIC endpoints
    • 9.2.3 QUIC interop runner participation
  • 9.3.0 HTTP/3 compatibility
    • 9.3.1 Against other HTTP/3 servers/clients
    • 9.3.2 Public HTTP/3 endpoints
    • 9.3.3 Browser compatibility
• 10.0.0 Performance Testing
  • 10.1.0 Throughput
    • 10.1.1 Single stream throughput
    • 10.1.2 Multiple stream throughput
    • 10.1.3 Bidirectional throughput
    • 10.1.4 Large file transfers
    • 10.1.5 Small message latency
• 10.2.0 Scalability
  • 10.2.1 Many concurrent sessions
  • 10.2.2 Many concurrent streams per session
  • 10.2.3 Connection establishment rate
  • 10.2.4 Memory usage under load
• 10.3.0 Congestion control
  • 10.3.1 Algorithm comparison (Reno vs Cubic vs BBR)
  • 10.3.2 Recovery from loss
  • 10.3.3 Bandwidth utilization
  • 10.3.4 Fairness
• 10.4.0 0-RTT performance
  • 10.4.1 Resumption latency reduction
  • 10.4.2 Early data volume
• 10.5.0 [ ] Benchmarks
• 11.0.0 API Testing
  • 11.1.0 JavaScript API surface
    • 11.1.1 quic.connect() function
    • 11.1.2 quic.listen() function
    • 11.1.3 QuicEndpoint class
    • 11.1.4 QuicSession class
    • 11.1.5 QuicStream class
    • 11.1.6 Constants export
  • 11.2.0 Options validation
    • 11.2.1 Type checking
    • 11.2.2 Range validation
    • 11.2.3 Required vs optional
    • 11.2.4 Default values
    • 11.2.5 Invalid combinations
  • 11.3.0 Event handling
    • 11.3.1 Diagnostics channels
    • 11.3.2 Callback invocations
    • 11.3.3 Event ordering
    • 11.3.4 Error propagation
• 12.0.0 Integration Testing
  • 12.1.0 Simple echo server
  • 12.2.0 File transfer
  • 12.3.0 Streaming media
  • 12.4.0 WebTransport-like usage
• 13.0.0 Debugging and Observability
  • 13.1.0 QLog output
    • 13.1.1 Event logging
    • 13.1.2 Format compliance
    • 13.1.3 Visualization compatibility
  • 13.2.0 Key logging
    • 13.2.1 SSLKEYLOGFILE format
    • 13.2.2 Wireshark integration
  • 13.3.0 Statistics
    • 13.3.1 Endpoint stats accuracy
    • 13.3.2 Session stats accuracy
    • 13.3.3 Stream stats accuracy
    • 13.3.4 Timestamp consistency
  • 13.4.0 Diagnostics channels
    • 13.4.1 All events emitted correctly
    • 13.4.2 Event payload validation
    • 13.4.3 Performance overhead
• 14.0.0 Network environments
  • 14.1.0 IPv4 only
  • 14.2.0 IPv6 only
  • 14.3.0 Dual stack
  • 14.4.0 NAT traversal
  • 14.5.0 Firewalls
• 15.0.0 Compliance Testing
  • 15.1.0 RFC 9000 (QUIC v1)
    • 15.1.1 All MUST requirements
    • 15.1.2 All SHOULD requirements
    • 15.1.3 Recommended behaviors
  • 15.2.0 RFC 9001 (QUIC TLS)
    • 15.2.1 TLS 1.3 integration
    • 15.2.2 Key derivation
    • 15.2.3 Handshake messages
  • 15.3.0 RFC 9002 (Loss Detection and Congestion Control)
    • 15.3.1 Loss detection algorithms
    • 15.3.2 Congestion control requirements
    • 15.3.3 Pacing
  • 15.4.0 RFC 9114 (HTTP/3)
    • 15.4.1 Frame types
    • 15.4.2 Stream mapping
    • 15.4.3 Error codes
  • 15.5.0 RFC 9204 (QPACK)
    • 15.5.1 Encoding/decoding
    • 15.5.2 Dynamic table
    • 15.5.3 Stream blocking
  • 15.6.0 RFC 9221 (Unreliable Datagrams)
    • 15.6.1 Datagram frames
    • 15.6.2 Flow control exemption
• 16.0.0 Fuzzing and Security
  • 16.1.0 Packet fuzzing
  • 16.2.0 Frame fuzzing
  • 16.3.0 Input validation fuzzing
  • 16.4.0 Security scenarios
    • 16.4.1 Amplification attack resistance
    • 16.4.2 DoS resistance / Packet flooding
    • 16.4.3 Resource limits enforcement
    • 16.4.4 Timing attack resistance
    • 16.4.5 Slow-loris-type attacks (slow transmission, slow acks)
• 17.0.0 Documentation Testing
  • 17.1.0 Example code validity
    • 17.1.1 All examples run successfully
    • 17.1.2 Examples follow best practices
  • 17.2.0 API documentation accuracy
  • 17.3.0 All public APIs documented
  • 17.4.0 Parameter types correct
  • 17.5.0 Return types correct
  • 17.6.0 Error conditions documented

[martenrichter]

Here is some work (actually not on the list):
https://github.com/nodejs/node/pull/60237/files#diff-ea3f0a30d20c7e97567c28c8ac8ab672fb82138c02b5fa38eb4146649cfd7aa8
includes a test (as a side effect) for 3.1.2.2 .
Other combinations are possible. I am currently debugging it and finding and filing issues in quic, that are uncovered. (I have filed them separately, but I can also try to patch them within the PR.):
#60309

[martenrichter]

Also added 3.1.1.2 to the PR, which is outstanding for review. However, the test does not work currently as it uncovered a race condition (which is the purpose of a test), which is now on the TODO list.

[Ethan-Arrowood]

This is awesome. I’d love to try and help here. Any easier places for me to start and get familiar with first? Thank you for organizing this list!