Parallelize the SSH integration tests across OS targets and reduce
per-container overhead:
- CI: use GitHub Actions matrix strategy to run all 4 OS containers
(ubuntu:focal, ubuntu:jammy, ubuntu:noble, alpine:latest) in parallel
instead of sequentially (~4x wall-clock improvement)
- Makefile: run docker builds in parallel for local dev too
- Dockerfile: consolidate ~20 separate RUN commands into 5 (one per
test phase), eliminating Docker layer overhead. Combine test binary
invocations where no state mutation is needed between them. Fix a bug
where TestDoDropPrivileges was silently not being run (was passed as a
second positional arg to -test.run instead of using regex alternation).
- TestMain: replace tail -F + 2s sleep with synchronous log read,
eliminating 2s overhead per test binary invocation. Set debugTest once
in TestMain instead of redundantly in each test function.
- session.read(): close channel on EOF so non-shell tests return
immediately instead of waiting for the 1s silence timeout.
Updates #19244
Change-Id: I2cc8588964fbce0dd7b654fb94e7ff33440b8584
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Start using a common helper for tests to declare that they require root.
This is step 1. A later step will then make this helper track which tests were
skipped so a subsequent pass will run these test as root.
Updates tailscale/corp#40007
Change-Id: I4979e1def0fa3691d38c83f48c89aaa443e7f62e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This reverts commit b25920dfc07452833895ad00b42db7e581b3cec8.
The `log.Printf` messages are causing panics in corp, in particular:
> panic: please use tailscale.com/logger.Logf instead of the log package
Fixing the TKA code to plumb through a logger properly is going to be
a hassle, so for now remove these logs to unblock merges to corp.
Updates tailscale/corp#39455
Signed-off-by: Alex Chan <alexc@tailscale.com>
ipn/local: add netmap mutations to the ipn bus
updates tailscale/tailscale#1909
This adds a new new NotifyWatchOpt that allows watchers to
receive PeerChange events (derived from node mutations)
on the IPN bus in lieu of a complete netmap. We'll continue
to send the full netmap for any map response that includes it,
but for mutations, sending PeerChange events gives the client
the option to manage it's own models more selectively and cuts
way down on json serialization overhead.
On chatty tailnets, this will vastly reduce the amount of
chatter on the bus.
This change should be backwards compatible, it is
purely additive. Clients that subscribe to NotifyNetmap will
get the full netmap for every delta. New clients can
omit that and instead opt into NotifyPeerChanges.
Signed-off-by: Jonathan Nobels <jonathan@tailscale.com>
On dual-stack clusters defaulting to IPv6, the ProxyGroup egress
service only got an IPv6 address, which causes request failures.
Individual egress proxies already set PreferDualStack correctly.
Fixes: #18768
Signed-off-by: Fernando Serboncini <fserb@tailscale.com>
Validated against a modern Debian install, fixes a typo.
Updates #cleanup
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I7b26012f54dbd2f0f9fea98722e8edc2fe97645a
As a warm-up to making natlab support multiple operating systems,
start with an easy one (in that it's also Unixy and open source like
Linux) and add FreeBSD 15.0 as a VM OS option for the vmtest
integration test framework, and add TestSubnetRouterFreeBSD which
tests subnet routing through a FreeBSD VM (Gokrazy → FreeBSD →
Gokrazy).
Key changes:
- Add FreeBSD150 OSImage using the official FreeBSD 15.0
BASIC-CLOUDINIT cloud image (xz-compressed qcow2)
- Add GOOS()/IsFreeBSD() methods to OSImage for cross-compilation
and OS-specific behavior
- Handle xz-compressed image downloads in ensureImage
- Refactor compileBinaries into compileBinariesForOS to support
multiple GOOS targets (linux, freebsd), with binaries registered
at <goos>/<name> paths on the file server VIP
- Add FreeBSD-specific cloud-init (nuageinit) user-data generation:
string-form runcmd (nuageinit doesn't support YAML arrays),
fetch(1) instead of curl, FreeBSD sysctl names for IP forwarding,
mkdir /usr/local/bin, PATH setup for tta
- Skip network-config in cidata ISO for FreeBSD (DHCP via rc.conf)
Updates tailscale/tailscale#13038
Change-Id: Ibeb4f7d02659d5cd8e3a7c3a66ee7b1a92a0110d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This commit modifies the kubernetes operator to use the `tailscale-client-go-v2`
package instead of the internal tailscale client it was previously using. This
now gives us the ability to expand out custom resources and features as they
become available via the API module.
The tailnet reconciler has also been modified to manage clients as tailnets
are created and removed, providing each subsequent reconciler with a single
`ClientProvider` that obtains a tailscale client for the respective tailnet
by name, or the operator's default when presented with a blank string.
Fixes: https://github.com/tailscale/corp/issues/38418
Signed-off-by: David Bond <davidsbond93@gmail.com>
Log whenever we:
* Commit an AUM which was previously soft-deleted (which we don't expect
to happen in practice, and may indicate an issue with our sync code)
* Purge AUMs during a Compact operation.
* Successfully commit AUMs as part of a bootstrap or sync operation.
All three logs mention `tka` for easy of discoverability.
Updates tailscale/corp#39455
Change-Id: I2b07bb0ef075877f40ec34b80bb668be59e1cdc3
Signed-off-by: Alex Chan <alexc@tailscale.com>
Add tstest/natlab/vmtest, a high-level framework for running multi-VM
integration tests with mixed OS types (gokrazy + Ubuntu/Debian cloud
images) connected via natlab's vnet virtual network.
The vmtest package provides:
- Env type that orchestrates vnet, QEMU processes, and agent connections
- OS image support (Gokrazy, Ubuntu2404, Debian12) with download/cache
- QEMU launch per OS type (microvm for gokrazy, q35+KVM for cloud)
- Cloud-init seed ISO generation with network-config for multi-NIC
- Cross-compilation of test binaries for cloud VMs
- Debug SSH NIC on cloud VMs for interactive debugging
- Test helpers: ApproveRoutes, HTTPGet, TailscalePing, DumpStatus,
WaitForPeerRoute, SSHExec
TTA enhancements (cmd/tta):
- Parameterize /up (accept-routes, advertise-routes, snat-subnet-routes)
- Add /set, /start-webserver, /http-get endpoints
- /http-get uses local.Client.UserDial for Tailscale-routed requests
- Fix /ping for non-gokrazy systems
TestSubnetRouter exercises a 3-VM subnet router scenario:
client (gokrazy) → subnet-router (Ubuntu, dual-NIC) → backend (gokrazy)
Verifies HTTP access to the backend webserver through the Tailscale
subnet route. Passes in ~30 seconds.
Updates tailscale/tailscale#13038
Change-Id: I165b64af241d37f5f5870e796a52502fc56146fa
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new envknob that allows connections from trusted CIDR ranges
to access debug endpoints without Tailscale authentication. This is
useful for in-cluster scrapers like Prometheus that are not on a
tailnet, do not have static IP addresses and cannot use debug keys.
Fixes#19282
Signed-off-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Add misc/install-git-hooks.go and misc/git_hook/ to the OSS repo,
adapted from the corp repo. The primary motivation is Change-Id
generation in commit messages, which provides a persistent identifier
for a change across cherry-picks between branches.
The installer uses "git rev-parse --git-common-dir" instead of go-git
to find the hooks directory, avoiding a new direct dependency while
still supporting worktrees.
Hooks included:
- commit-msg: adds Change-Id trailer
- pre-commit: blocks NOCOMMIT / DO NOT SUBMIT markers
- pre-push: blocks local-directory replace directives in go.mod
- post-checkout: warns when the hook binary is outdated
Also update docs/commit-messages.md to reflect that Change-Id is no
longer optional in the OSS repo.
Updates tailscale/corp#39860
Change-Id: I09066b889118840c0ec6995cc03a9cf464740ffa
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
go.cmd used cmd.exe to invoke PowerShell, which mangled arguments:
cmd.exe treats ^ as an escape character (so -run "^$" became -run "$",
running all tests instead of none) and = signs also caused issues in
the PowerShell→cmd.exe argument passing layer.
Replace it with a tiny no_std Rust binary (19KB, 32-bit x86 for
universal Windows compat: x86/x64/ARM64) that directly invokes the
Tailscale Go toolchain via CreateProcessW. The raw command line from
GetCommandLineW is passed through to CreateProcessW with only argv[0]
replaced, so arguments are never parsed or re-escaped.
The binary also handles first-run toolchain download natively using
curl.exe and tar.exe (both ship with Windows 10+), so PowerShell is
no longer required for normal operation. The PowerShell fallback is
only used for the rare TS_USE_GOCROSS=1 path.
PowerShell prefers go.exe over go.cmd when resolving ./tool/go, so
this is a drop-in replacement.
With go.exe in place, the CI can use the natural -bench=. -benchtime=1x
-run="^$" flags directly.
Also removes tool/go-win.ps1 which is now unused.
Updates #19255
Change-Id: I80da23285b74796e7694b89cff29a9fa0eaa6281
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Multi-NIC support:
- Add nodeNIC type and node.extraNICs for secondary network interfaces
- Add netForMAC/macForNet to route packets to the correct network by MAC
- Update initFromConfig to allocate a MAC + LAN IP per network
- Fix handleEthernetFrameFromVM, ServeUnixConn to use netForMAC
- Fix MACOfIP, writeEth, WriteUDPPacketNoNAT, gVisor write path, and
createARPResponse to use macForNet (return the MAC actually on that
network, not the node's primary MAC)
- Fix createDHCPResponse for multi-NIC (correct client IP and subnet)
- Add nodeNICMac for secondary NIC MAC generation
- Add Node accessors: NumNICs, NICMac, Networks, LanIP
DHCP fixes:
- Include LeaseTime, SubnetMask, Router, DNS in DHCP Offer (not just
Ack). systemd-networkd requires these to accept an Offer.
- Fix DHCP response source IP: use gateway IP instead of echoing
the request's destination (which was 255.255.255.255 for discovers)
New VIPs:
- cloud-init.tailscale: serves per-node cloud-init meta-data, user-data,
and network-config for VMs booting with nocloud datasource
- files.tailscale: serves binary files (tta, tailscale, tailscaled)
registered via RegisterFile for cloud VM provisioning
- Add ControlServer() accessor for test control server
This is necessary for a three-VM natlab subnet router
integration test, coming later.
Updates #13038
Change-Id: I59f9f356bae9b5509c117265237983972dfdd5af
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When SetSubnetRoutes is called, also send updatePeerChanged to all
other connected nodes so they re-fetch their MapResponse and learn
about the updated AllowedIPs. Without this, peers never see new
subnet routes until they happen to reconnect to the control server.
Discovered while working on a three-VM natlab subnet router
integration test, coming later.
Updates #13038
Change-Id: I20e7a2fda994a8ab0e7a24240e6eae536f4f5f15
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Instead of generating the full netmap, just fetch the peers out the the
existing peers map.
The extra usage was introduced with netmap caching, but there is no need
to call the netmap to get this information, rather the existing peermap
can be used.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Fixes UDP listeners on VIP Service addresses not receiving inbound traffic.
- Modified shouldProcessInbound to check for registered UDP transport endpoints when processing packets to service VIPs
- Uses FindTransportEndpoint to determine if a UDP listener exists for the destination VIP/port
- Supports both IPv4 and IPv6
The aim was to mirror the existing TCP logic, providing feature parity for UDP-based services on VIP Services.
Fixes#18971
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Add ExtraRootCAs *x509.CertPool to tsd.System and plumb it through
the control client, noise transport, DERP, and wgengine layers so
that platforms like Android can inject user-installed CA certificates
into Go's TLS verification.
tlsdial.Config now honors base.RootCAs as additional trusted roots,
tried after system roots and before the baked-in LetsEncrypt fallback.
SetConfigExpectedCert gets the same treatment for domain-fronted DERP.
The Android client will set sys.ExtraRootCAs with a pool built from
x509.SystemCertPool + user-installed certs obtained via the Android
KeyStore API, replacing the current SSL_CERT_DIR environment variable
approach.
Updates #8085
Change-Id: Iecce0fd140cd5aa0331b124e55a7045e24d8e0c2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
CPU profiling a containerboot subnet router on a large tailnet showed
roughly 40% of CPU spent in serveWatchIPNBus JSON-encoding the full
netmap on every update. containerboot only reads SelfNode fields from
those notifications (and does a peer lookup when TailnetTargetFQDN is
set), so it does not need every intermediate netmap delta.
Set ipn.NotifyRateLimit on all three WatchIPNBus calls so netmap
notifications are coalesced to one per 3s. Initial-state delivery is
unaffected since the rateLimitingBusSender flushes the first send
immediately.
Updates #cleanup
Signed-off-by: Doug Bryant <dougbryant@anthropic.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Consolidate the duplicated WebSocket frame-parsing logic from Read
and Write into a shared processFrames loop, fixing several bugs in
the process:
- Mixed control and data frames in a single Read/Write call buffer
were not handled: a control frame would cause merged data frames
to be skipped.
- Multiple data frames into one Write call weren't being correctly
parsed: only the first frame was processed, ignoring the rest in
the buffer.
- msg.isFinalized was being set before confirming the fragment was
complete, so an incomplete msg fragment, could've been sometimes
marked as finalized.
- Continuation frames without any payload were being treated as if
they didn't have stream ID, even thought the id is already known
from the initial fragment.
Fixestailscale/corp#39583
Signed-off-by: Fernando Serboncini <fserb@tailscale.com>
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Co-authored-by: chaosinthecrd <tom@tmlabs.co.uk>
Move the ipn/desktop blank import from cmd/tailscaled/tailscaled_windows.go
into feature/condregister/maybe_desktop_sessions.go, consistent with how
all other modular features are registered. tailscaled already imports
feature/condregister, so it still gets ipn/desktop on Windows.
Updates #12614
Change-Id: I92418c4bf0e67f0ab40542e47584762ac0ffa2b2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
GetMessage can call back into Go, triggering stack growth and causing the stack
to be copied to a new memory region, which invalidates the original stack pointer
passed to the syscall. Since GetMessage uses that pointer to write the message
before returning, this leads to memory corruption.
In this PR, we fix this by using runtime.Pinner, which requires the pointer to refer
to heap-allocated memory.
Fixes#19263Fixes#17832
Signed-off-by: Nick Khyl <nickk@tailscale.com>
Add a new "ipnbus" build feature tag so the watch-ipn-bus LocalAPI
endpoint can be independently controlled, rather than being gated
behind HasDebug || HasServe. Minimal/embedded builds that omit both
debug and serve were getting 404s on watch-ipn-bus, breaking
"tailscale up --authkey=..." and other CLI flows that depend on
WatchIPNBus.
In the CLI, check buildfeatures.HasIPNBus before attempting to watch
the IPN bus in "tailscale up"/"tailscale login", and exit early with
an informational message when the feature is omitted.
Also add the missing NewCounterFunc stub to clientmetric/omit.go,
which caused compilation errors when building with
ts_omit_clientmetrics and netstack enabled.
Fixes#19240
Change-Id: I2e3c69a72fc50fa02542b91b8a54859618a463d1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
If an entry in the tsmpLearnedDisco does not match the disco key of the
key currently being processed, overwrite the key, and leave the entry in
the map for later processing.
In reality, this should not happen, but is put in as a safety measure
with logging of the situation so we can replicate the behaviour and
correct it should it happen.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
After moving around locks in 4334dfa7d5ccbee1daf5acf30b33557bbca66525,
a data race were made possible.
Introduce an RWlock to the mapSession itself for fetching peers.
Fixes#19260
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
When a recording upload fails mid-session, the recording goroutine
cancels the session context. This triggers two concurrent paths:
exec.CommandContext kills the process (causing cmd.Wait to return),
and killProcessOnContextDone tries to write the termination message
via exitOnce.Do. If cmd.Wait returns first, the main goroutine's
exitOnce.Do(func(){}) steals the once, and the termination message
is never written to the client.
Fix by waiting for killProcessOnContextDone to finish writing the
termination message (via <-ss.exitHandled) before claiming exitOnce,
when the context is already done.
Also fix the fallback path when launchProcess itself fails due to
context cancellation: use SSHTerminationMessage() with the correct
"\r\n\r\n" framing instead of fmt.Fprintf with the internal error
string.
Deflakes TestSSHRecordingCancelsSessionsOnUploadFailure, which was
failing consistently at a low rate due to the exitOnce race. After
this fix, flakestress passes with 8,668 runs, 0 failures.
Fixes#7707 (again. hopefully for good.)
Change-Id: I5ab911c71574db8d3f9d979fb839f273be51ecf9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Brings in a newer version of Gliderlabs SSH with added socket forwarding support.
Fixes#12409Fixes#5295
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Investigating battery costs on a busy tailnet I noticed a large number
of nodes regularly reconnecting to control and DERP. In one case I was
able to analyze closely `pmset` reported the every-minute wake-ups being
triggered by bluetooth. The node was by side effect reconnecting to
control constantly, and this was at times visible to peers as well.
Three changes here improve the situation:
- Short time jumps (less than 10 minutes) no longer produce "major
network change" events, and so do not trigger full rebind/reconnect.
- Many "incidental" fields on interfaces are ignored, like MTU, flags
and so on - if the route is still good, the rest should be manageable.
- Additional log output will provide more detail about the cause of
major network change events.
Updates #3363
Signed-off-by: James Tucker <james@tailscale.com>
Set csrf.Path("/") so the CSRF cookie is available across all routes,
not just the path where it was set.
Add helpers to expose the gorilla/csrf token for use.
Updates #19264
Signed-off-by: Fernando Serboncini <fserb@tailscale.com>
Commit f905871fb moved host key generation from the ipnLocalBackend
interface (GetSSH_HostKeys) to the standalone getHostKeys function,
which requires either system host keys in /etc/ssh/ or a valid
TailscaleVarRoot to generate keys into. The testBackend returned ""
for TailscaleVarRoot, and the Docker test containers only install
openssh-client (no server host keys), so getHostKeys always failed.
When getHostKeys fails, HandleSSHConn returns the error but never
closes the TCP connection, so SSH clients hang forever waiting for
the server hello.
Fix by creating a temp directory in TestMain and returning it from
testBackend.TailscaleVarRoot().
Regression from f905871fb #18949 ("ipn/ipnlocal, feature/ssh: move SSH code
out of LocalBackend to feature").
I was apparently too impatient to wait for the test to complete
and didn't connect the dots: https://github.com/tailscale/tailscale/actions/runs/22930275950
We should make that test faster (#19244) for the patience issue, but
also fail more nicely if this happens in the future.
Updates #19244
Change-Id: If82393b8f35413b04174e6f7d09a1ee3a2125a6b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The cloner and viewer code generators didn't handle named types
with basic underlying types (map/slice) that have their own Clone
or View methods. For example, a type like:
type Map map[string]any
func (m Map) Clone() Map { ... }
func (m Map) View() MapView { ... }
When used as a struct field, the cloner would descend into the
underlying map[string]any and fail because it can't clone the any
(interface{}) value type. Similarly, the viewer would try to create
a MapFnOf view and fail.
Fix the cloner to check for a Clone method on the named type
before falling through to the underlying type handling.
Fix the viewer to check for a View method on named map/slice types,
so the type author can provide a purpose-built safe view that
doesn't leak raw any values. Named map/slice types without a View
method fall through to normal handling, which correctly rejects
types like map[string]any as unsupported.
Updates tailscale/corp#39502 (needed by tailscale/corp#39594)
Change-Id: Iaef0192a221e02b4b8e409c99ef8398090327744
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
To denoise log output, to make it easier to find real failures.
Updates #19252
Change-Id: Iae64a9278c70de24a236c39e3d181a509a512a0b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The -run "^$" flag was being mangled by cmd.exe's argument processing.
The ^ character is cmd.exe's escape character, so go.cmd's cmd.exe layer
eats it, turning -run "^$" into -run "$" which matches all test names.
This caused the benchmark job to run every test, leading to timeouts
and Go runtime crashes.
Use -run XXXXNothingXXXX instead, which avoids special characters
entirely.
Updates #19252
Change-Id: I888c124254dd2767a40b61bcd68dbc9b22ad35a1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The upload-client-metrics handler called metricCapture without
checking if it was nil or if the metrics slice was empty. Most
tests pass nil for metricCapture, so if a metrics upload races
in during the test, it panics.
Fixes#19252
Change-Id: Ib904d1fe6779067dc2a153d1680b8f50cba9c773
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new vet analyzer that checks t.Run subtest names don't contain
characters requiring quoting when re-running via "go test -run". This
enforces the style guide rule: don't use spaces or punctuation in
subtest names.
The analyzer flags:
- Direct t.Run calls with string literal names containing spaces,
regex metacharacters, quotes, or other problematic characters
- Table-driven t.Run(tt.name, ...) calls where tt ranges over a
slice/map literal with bad name field values
Also fix all 978 existing violations across 81 test files, replacing
spaces with hyphens and shortening long sentence-like names to concise
hyphenated forms.
Updates #19242
Change-Id: Ib0ad96a111bd8e764582d1d4902fe2599454ab65
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
TestGocrossWrapper will fail when run inside a git linked worktree
because Go 1.26 and earlier cannot get the current revision hash.
Since this will be fixed in Go 1.27, see golang/go#58218, this patch
skips this test until that release.
Fixes#19217
Signed-off-by: Simon Law <sfllaw@tailscale.com>
The test sets up an HTTP-over-Unix server and a reverse proxy pointed at
this server, but prior to this change did not round-trip anything to the
backing server. This change ensures that we test code paths which proxy
Unix sockets for serve.
Fixes#19232
Signed-off-by: Harry Harpham <harry@tailscale.com>
This is a follow-up to #19117, adding a debug CLI command allowing the operator
to explicitly discard cached netmap data, as a safety and recovery measure.
Updates #12639
Change-Id: I5c3c47c0204754b9c8e526a4ff8f69d6974db6d0
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
