The WebSocket tests have been very flaky because we weren't able to tell when a WebSocket was fully connected and subscribed to events.
We reworked the websocket subscription code to accept the websocket only after subscribing.
This should eliminate all flakiness in these tests. 🤞 (We can follow-up in an enterprise PR to simplify some of the tests after this fix is merged.)
I ran this locally a bunch of times and with data race detection enabled, and did not see any failures.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
We grab the state lock and check that the core is not shutting down.
This panic mostly seems to happen if Vault is shutting down, usually
in a test.
Also, we try clean up the go-bexpr test by sending duplicates, and
deduplicating in the receive loop.
When #22835 was merged, it was auto-squashed, so the `experiments`
import was removed, but the test still referenced it.
This removes the (now unnecessary) experiment from the test.
Subscribing to events through a WebSocket now support boolean
expressions to filter only the events wanted based on the fields
* `event_type`
* `operation`
* `source_plugin_mount`
* `data_path`
* `namespace`
Example expressions:
These can be passed to `vault events subscribe`, e.g.,:
* `event_type == abc`
* `source_plugin_mount == secret/`
* `event_type != def and operation != write`
```sh
vault events subscribe -filter='source_plugin_mount == secret/' 'kv*'
```
The docs for the `vault events subscribe` command and API endpoint
will be coming shortly in a different PR, and will include a better
specification for these expressions, similar to (or linking to)
https://developer.hashicorp.com/boundary/docs/concepts/filtering
The flag `events.alpha1` will no longer do anything, but we keep it
to prevent breaking users who have it in their configurations or
startup flags, or if it is referenced in other code.
The more I looked at this test, the more I realized it wasn't testing
anything except that the namespace parameter was being parsed by the
websocket.
So, I moved that parameter parsing check to `TestEventsSubscribe()`,
which is not flaky, and removed the flaky test altogether.
There is a similar set of tests in the enterprise repo that I will
try to simplify and make less flaky, though.
Previously, when a user initiated a websocket subscription,
the access to the `sys/events/subscribe` endpoint was checked then,
and only once.
Now, perform continuous policy checks:
* We check access to the `sys/events/subscribe` endpoint every five
minutes. If this check fails, then the websocket is terminated.
* Upon receiving any message, we verify that the `subscribe`
capability is present for that namespace, data path, and event type.
If it is not, then the message is not delivered. If the message is
allowed, we cache that result for five minutes.
Tests for this are in a separate enterprise PR.
Documentation will be updated in another PR.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
CI is sometimes slow, so 100ms was not enough time for all events
to be sent and processed in `http/events_test.go`.
We bumped that timeout to a full 1 second, but also added a trick at
the end to shorten the timeout once the expected number of events
have been receieved. This way, once the test has passed, we only
wait 100ms for any "extra" events to make the test fail, instead
of waiting for the full 1 second before we let the test pass. This
should keep the test relatively fast, while still allowing for it to
be slow sometimes.
Events from multiple namespaces can be subscribed to via
glob patterns passed to the subscription.
This does not do policy enforcement yet -- that will come in PR soon.
I tested this manually as well by pulling it into Vault Enterprise
so I could create namespaces and check that subscriptions work as
expected.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Biggest change: we rename `Send` to `SendEvent` in `logical.EventSender`..
Initially we picked `Send` to match the underlying go-eventlogger
broker's `Send` method, and to avoid the stuttering of `events.SendEvent`.
However, I think it is more useful for the `logical.EventSender`
interface to use the method `SendEvent` so that, for example,
`framework.Backend` can implement it.
This is a relatively change now that should not affect anything
except the KV plugin, which is being fixed in another PR.
Another change: if the `secret_path` metadata is present, then
the plugin-aware `EventBus` will prepend it with the plugin mount.
This allows the `secret_path` to be the full path to any referenced
secret.
This change is also backwards compatible, since this field was not
present in the KV plugin. (It did use the slightly different `path`
field, which we can keep for now.)
For now, only the leader of a cluster can handle subscription requests,
so we forward the connection request otherwise.
We forward using a 307 temporary redirect (the fallback way).
Forwarding a request over gRPC currently only supports a single request
and response, but a websocket connection is long-lived with potentially
many messages back and forth.
We modified the `vault events subscribe` command to honor those
redirects. `wscat` supports them with the `-L` flag.
In the future, we may add a gRPC method to handle forwarding WebSocket
requests, but doing so adds quite a bit of complexity (even over
normal request forwarding) due to the intricate nature of the `http` /
`vault.Core` interactions required. (I initially went down this path.)
I added tests for the forwarding header, and also tested manually.
(Testing with `-dev-three-node` is a little clumsy since it does not
properly support experiments, for some reason.)
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
The [WebSockets spec](https://www.rfc-editor.org/rfc/rfc6455) states
that text messages must be valid UTF-8 encoded strings, which protobuf
messages virtually never are. This now correctly sends the protobuf events
as binary messages.
We change the format to correspond to CloudEvents, as originally intended,
and remove a redundant timestamp and newline.
We also bump the eventlogger to fix a race condition that this code triggers.
This checks the request against the `read` permission for
`sys/events/subscribe/{eventType}` on the initial subscribe.
Future work includes moving this to its own verb (`subscribe`)
and periodically rechecking the request.
Tested locally by minting a token with the wrong permissions
and verifying that they are rejected as expected, and that
they work if the policy is adjusted to `sys/event/subscribe/*`
(or the specific topic name) with `read` permissions.
I had to change the `core.checkToken()` to be publicly accessible,
as it seems like the easiest way to check the token on the
`logical.Request` against all relevant policies, but without
going into all of the complex logic further in `handleLogical()`.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Also updates the event receieved to include a timestamp.
Websockets support both JSON and protobuf binary formats.
This can be used by either `wscat` or the new
`vault events subscribe`:
e.g.,
```sh
$ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
and
```sh
$ vault events subscribe abc
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
