43 Commits

Author SHA1 Message Date
Vault Automation
0c6c13dd38
license: update headers to IBM Corp. (#10229) (#10233)
* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed

Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
2025-10-21 15:20:20 -06:00
Vault Automation
8d2cb89704
VAULT-38463: Addressing ldap pipeline failure (#8817) (#8911)
* VAULT-38463: Addressing ldap pipeline failure

* testing ldap tests

* testing ldap tests

* debugging ldap issue

* testing pipeline

* testing pipeline

* testing pipeline

* testing pipeline

* testing pipeline

* testing pipeline

* testing pipeline

* debugging ldap failure

* debugging ldap failure

* debugging pipeline

* adding dependency for verify secrets

* removing extra code

* undo changes

* undo changes

Co-authored-by: Tin Vo <tintvo08@gmail.com>
2025-08-26 12:46:36 -07:00
Charles Nwokotubo
0187338dd8
[Enos] VAULT-30196: SSH Secrets Engine (#29534) 2025-08-06 19:22:06 -04:00
Luis (LT) Carbonell
4036485739
(enos) Add KMIP Enos Test Suite (#31378)
* (enos) Add KMIP Enos Test Suite

* skip KMIP for CE runs

* reads...

* cleanup variables

* fix
2025-07-29 14:13:28 -04:00
kelly
f0201408b4
VAULT-31185 & 31186/use identity token auth for Artifactory in Vault CE & Ent (#31255)
* removed artifactory_username

* updated artifactory token

* ran enos fmt

* ran terraform fmt

* debugging/ testing - pinned enos version, added null username

* byyyyy
2025-07-28 12:16:25 -04:00
Tin Vo
857e66b3e2
VAULT-35602: Adding Enos OpenLDAP test (#30801)
* VAULT-35602: adding Enos LDAP Tests

* adding godaddy tests

* updating external integration target module name
2025-07-23 13:11:12 -07:00
Luis (LT) Carbonell
403720c1fd
Add non-leader test for enos (#30657)
* Add non-leader test for enos

* Make clearer comments
2025-05-22 11:25:19 -04:00
Luis (LT) Carbonell
ed52371b10
Upgrade FIPS 1402 -> 1403 (#30576)
* Upgrade FIPS 1402 -> 1403

* Clean up

* changelog
2025-05-12 15:01:30 -05:00
Tin Vo
4c36d90281
VAULT-30187: Create Enos AWS Engine tests (#29566)
* Testing Enos AWS Engine tests

* Testing Enos AWS Engine tests

* Testing Enos AWS Engine tests

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine test

* testing enos aws engine

* testing enos aws engine

* updating test for enterprise

* updating test for enterprise

* updating test for enterprise

* removing testing output

* removing testing output

* removing testing github action

* fixing lint

* removing sensitive flag

* including sensitive flag due to terraform errors

* removing testing action workflow
2025-04-21 10:30:43 -07:00
Tin Vo
ac3bb7b2d4
VAULT-32188: Enos test for PKI certificates (#29007)
* updating pki test

* updating pki test

* updating pki test

* updating pki script

* resolving conflicts

* adding pki cert verifications

* resolving conflicts

* updating test

* removing comments

* addressing bash formatting

* updating test

* adding description

* fixing lint error

* fixing lint error

* fixing lint issue

* removing unneeded scenario

* resolving conflicts

* debugging pipeline error

* fixing pipeline tests'

* fixing pipeline tests'

* testing smoke test

* fixing pipeline error

* debugging pipeline error

* debugging pipeline error

* debugging pipeline error

* debugging agent test ci failure

* fixing ci errors

* uncomment token

* updating script

* updating hosts

* fixing lint

* fixing lint

* fixing lint

* adding revoked certificate

* undo kv.tf change

* updating cert issuing

* updating issuing certs to include issuer

* updating pki cert verification

* addressing comments

* fixing lint

* fixing lint

* fixing lint

* fixing lint

* updating verify_secrets_engine_read module

* fixing lint

* fixing lint

* fixing lint

* debugging lint

* testing pipeline

* adding verify variables for autopilot

* adding pki read variable for autopilot

* updating vault engine read variables

* addressing comments

* fixing lint

* update test for enterprise

* update pki tests to adapt to enterprise
2025-01-23 11:30:20 -08:00
Rebecca Willett
8cee664204
Add 'how to run' instructions to each Enos scenario (#29299)
* Add 'how to run' instructions for each scenario
2025-01-10 21:17:09 +00:00
Ryan Cragun
3b31b3e939
VAULT-32206: verify audit log and systemd journal secret integrity (#28932)
Verify vault secret integrity in unauthenticated I/O streams (audit log, STDOUT/STDERR via the systemd journal) by scanning the text with Vault Radar. We search for both known and unknown secrets by using an index of KVV2 values and also by radar's built-in heuristics for credentials, secrets, and keys.

The verification has been added to many scenarios where a slight time increase is allowed, as we now have to install Vault Radar and scan the text. In practice this adds less than 10 seconds to the overall duration of a scenario.

In the in-place upgrade scenario we explicitly exclude this verification when upgrading from a version that we know will fail the check. We also make the verification opt-in so as to not require a Vault Radar license to run Enos scenarios, though it will always be enabled in CI.

As part of this we also update our enos workflow to utilize secret values from our self-hosted Vault when executing in the vault-enterprise repo context.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-11-22 11:14:01 -07:00
Ryan Cragun
1082629d1f
VAULT-30819: Fix two potential flakes in DR replication (#28409)
Fix two occasional flakes in the DR replication scenario:
* Always verify that all nodes in the cluster are unsealed before
  verifying test data. Previously we only verified seal status on
  followers.
* Fix an occasional timeout when waiting for the cluster to unseal by
  rewriting the module to retry for a set duration instead of
  exponential backoff.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-09-17 12:32:15 -06:00
Ryan Cragun
392412829b
[VAULT-30189] enos: verify identity and OIDC tokens (#28274)
* [VAULT-30189] enos: verify identity and OIDC tokens

Expand our baseline API and data verification by including the identity
and identity OIDC tokens secrets engines. We now create a test entity,
entity-alias, identity group, various policies, and associate them with
the entity. For the OIDC side, we now configure the OIDC issuer, create
and rotate named keys, create and associate roles with the named key,
and issue and introspect tokens.

During a second phase we also verify that the those some entities,
groups, keys, roles, config, etc all exist with the expected values.
This is useful to test durability after upgrades, migrations, etc.

This change also includes new updates our prior `auth/userpass` and `kv`
verification. We had two modules that were loosely coupled and
interdependent. This restructures those both into a singular module with
child modules and fixes the assumed values by requiring the read module
to verify against the created state.

Going forward we can continue to extend this secrets engine verification
module with additional create and read checks for new secrets engines.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-09-09 14:29:11 -06:00
Ryan Cragun
74b6cc799a
VAULT-29583: Modernize default distributions in enos scenarios (#28012)
* VAULT-29583: Modernize default distributions in enos scenarios

Our scenarios have been running the last gen of distributions in CI.
This updates our default distributions as follows:
  - Amazon: 2023
  - Leap:   15.6
  - RHEL:   8.10, 9.4
  - SLES:   15.6
  - Ubuntu: 20.04, 24.04

With these changes we also unlock a few new variants combinations:
  - `distro:amzn seal:pkcs11`
  - `arch:arm64 distro:leap`

We also normalize our distro key for Amazon Linux to `amzn`, which
matches the uname output on both versions that we've supported.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-08-09 13:43:28 -06:00
Ryan Cragun
720e942662
[VAULT-2937] Verify the /sys/version-history in enos scenarios (#27947)
When verifying the Vault version, in addition to verifying the CLI
version we also check that the `/sys/version-history` contains the
expected version.

As part of this we also fix a bug where when doing an in-place upgrade
with a Debian or Redhat package we also remove the self-managed
`vault.service` systemd unit to ensure that correctly start up using the
new version of Vault.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-08-02 13:26:39 -06:00
Ryan Cragun
174da88b9d
VAULT-28146: Add IPV6 support to enos scenarios (#27884)
* VAULT-28146: Add IPV6 support to enos scenarios

Add support for testing all raft storage scenarios and variants when
running Vault with IPV6 networking. We retain our previous support for
IPV4 and create a new variant `ip_version` which can be used to
configure the IP version that we wish to test with.

It's important to note that the VPC in IPV6 mode is technically mixed
and that target machines still associate public IPV6 addresses. That
allows us to execute our resources against them from IPV4 networks like
developer machines and CI runners. Despite that, we've taken care to
ensure that only IPV6 addresses are used in IPV6 mode.

Because we previously had assumed the IP Version, Vault address, and
listener ports in so many places, this PR is essentially a rewrite and
removal of those assumptions. There are also a few places where
improvements to scenarios have been included as I encountered them while
working on the IPV6 changes.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-07-30 11:00:27 -06:00
Ryan Cragun
89e9e0f2cd
VAULT-28307 enos: allow arm64 fips1402 and hsm editions (#27571)
In preperation for arm64 builds of hsm, fips1402, and hsm.fips1402
editions of Vault Enterprise we'll allow them in our test scenarios.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-06-21 15:24:46 -06:00
Ryan Cragun
84935e4416
[QT-697] enos: add descriptions and quality verification (#27311)
In order to take advantage of enos' ability to outline scenarios and to
inventory what verification they perform we needed to retrofit all of
that information to our existing scenarios and steps.

This change introduces an initial set of descriptions and verification
declarations that we can continue to refine over time.

As doing this required that I re-read every scenanario in its entirety I
also updated and fixed a few things along the way that I noticed,
including adding a few small features to enos that we utilize to make
handling initial versions programtic between versions instead of having a
delta between our globals in each branch.

* Update autopilot and in-place upgrade initial versions
* Programatically determine which initial versions to use based on Vault
  version
* Partially normalize steps between scenarios to make comparisons easier
* Update the MOTD to explain that VAULT_ADDR and VAULT_TOKEN have been
  set
* Add scenario and step descriptions to scenarios
* Add initial scenario quality verification declarations to scenarios
* Unpin Terraform in scenarios as >= 1.8.4 should work fine
2024-06-13 11:16:33 -06:00
Rebecca Willett
1f0639a79c
Remove Leap 15.4 from testing matrices and AMI data sources; remove vestiges of Ubuntu 18.04 testing (#27416) 2024-06-10 11:44:32 -04:00
Rebecca Willett
c28739512a
Add Amazon Linux, openSUSE Leap, and SUSE SLES support to Enos scenarios and modules (#25983)
Add Consul edition support to Enos scenarios and modules
Add Linux distros and Consul edition to Enos samples
Bump RHEL versions to 9.3 and 8.9
2024-06-05 12:58:35 -04:00
Ryan Cragun
27ab988205
[QT-695] Add config_mode variant to some scenarios (#26380)
Add `config_mode` variant to some scenarios so we can dynamically change
how we primarily configure the Vault cluster, either by a configuration
file or with environment variables.

As part of this change we also:
* Start consuming the Enos terraform provider from public Terraform
  registry.
* Remove the old `seal_ha_beta` variant as it is no longer required.
* Add a module that performs a `vault operator step-down` so that we can
  force leader elections in scenarios.
* Wire up an operator step-down into some scenarios to test both the old
  and new multiseal code paths during leader elections.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-04-22 12:34:47 -06:00
Ryan Cragun
a087f7b267
[QT-627] enos: add pkcs11 seal testing with softhsm (#24349)
Add support for testing `+ent.hsm` and `+ent.hsm.fips1402` Vault editions
with `pkcs11` seal types utilizing a shared `softhsm` token. Softhsm2 is
a software HSM that will load seal keys from a local disk via pkcs11.
The pkcs11 seal implementation is fairly complex as we have to create a
one or more shared tokens with various keys and distribute them to all
nodes in the cluster before starting Vault. We also have to ensure that
each sets labels are unique.

We also make a few quality of life updates by utilizing globals for
variants that don't often change and update base versions for various
scenarios.

* Add `seal_pkcs11` module for creating a `pkcs11` seal key using
  `softhsm2` as our backing implementation.
* Require the latest enos provider to gain access to the `enos_user`
  resource to ensure correct ownership and permissions of the
  `softhsm2` data directory and files.
* Add `pkcs11` seal to all scenarios that support configuring a seal
  type.
* Extract system package installation out of the `vault_cluster` module
  and into its own `install_package` module that we can reuse.
* Fix a bug when using the local builder variant that mangled the path.
  This likely slipped in during the migration to auto-version bumping.
* Fix an issue where restarting Vault nodes with a socket seal would
  fail because a seal socket sync wasn't available on all nodes. Now we
  start the socket listener on all nodes to ensure any node can become
  primary and "audit" to the socket listner.
* Remove unused attributes from some verify modules.
* Go back to using cheaper AWS regions.
* Use globals for variants.
* Update initial vault version for `upgrade` and `autopilot` scenarios.
* Update the consul versions for all scenarios that support a consul
  storage backend.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-12-08 14:00:45 -07:00
Ryan Cragun
a46def288f
[QT-616] Add seal_ha enos scenario (#23812)
Add support for testing Vault Enterprise with HA seal support by adding
a new `seal_ha` scenario that configures more than one seal type for a
Vault cluster. We also extend existing scenarios to support testing
with or without the Seal HA code path enabled.

* Extract starting vault into a separate enos module to allow for better
  handling of complex clusters that need to be started more than once.
* Extract seal key creation into a separate module and provide it to
  target modules. This allows us to create more than one seal key and
  associate it with instances. This also allows us to forego creating
  keys when using shamir seals.
* [QT-615] Add support for configuring more that one seal type to
  `vault_cluster` module.
* [QT-616] Add `seal_ha` scenario
* [QT-625] Add `seal_ha_beta` variant to existing scenarios to test with
  both code paths.
* Unpin action-setup-terraform
* Add `kms:TagResource` to service user IAM profile

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-10-26 15:13:30 -06:00
Ryan Cragun
391cc1157a
[QT-602] Run proxy and agent test scenarios (#23176)
Update our `proxy` and `agent` scenarios to support new variants and
perform baseline verification and their scenario specific verification.
We integrate these updated scenarios into the pipeline by adding them
to artifact samples.

We've also improved the reliability of the `autopilot` and `replication`
scenarios by refactoring our IP address gathering. Previously, we'd ask
vault for the primary IP address and use some Terraform logic to determine
followers. The leader IP address gathering script was also implicitly
responsible for ensuring that a found leader was within a given group of
hosts, and thus waiting for a given cluster to have a leader, and also for
doing some arithmetic and outputting `replication` specific output data.
We've broken these responsibilities into individual modules, improved their
error messages, and fixed various races and bugs, including:
* Fix a race between creating the file audit device and installing and starting
  vault in the `replication` scenario.
* Fix how we determine our leader and follower IP addresses. We now query
  vault instead of a prior implementation that inferred the followers and sometimes
  did not allow all nodes to be an expected leader.
* Fix a bug where we'd always always fail on the first wrong condition
  in the `vault_verify_performance_replication` module.

We also performed some maintenance tasks on Enos scenarios  byupdating our
references from `oss` to `ce` to handle the naming and license changes. We
also enabled `shellcheck` linting for enos module scripts.

* Rename `oss` to `ce` for license and naming changes.
* Convert template enos scripts to scripts that take environment
  variables.
* Add `shellcheck` linting for enos module scripts.
* Add additional `backend` and `seal` support to `proxy` and `agent`
  scenarios.
* Update scenarios to include all baseline verification.
* Add `proxy` and `agent` scenarios to artifact samples.
* Remove IP address verification from the `vault_get_cluster_ips`
  modules and implement a new `vault_wait_for_leader` module.
* Determine follower IP addresses by querying vault in the
  `vault_get_cluster_ips` module.
* Move replication specific behavior out of the `vault_get_cluster_ips`
  module and into it's own `replication_data` module.
* Extend initial version support for the `upgrade` and `autopilot`
  scenarios.

We also discovered an issue with undo_logs that has been described in
the VAULT-20259. As such, we've disabled the undo_logs check until
it has been fixed.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-09-26 15:37:28 -06:00
Marc Boudreau
e30c50321c
enable all audit devices in Enos's vault_cluster module (#22408) 2023-09-15 10:44:23 -04:00
Ryan Cragun
5f1d2c56a2
[QT-506] Use enos scenario samples for testing (#22641)
Replace our prior implementation of Enos test groups with the new Enos
sampling feature. With this feature we're able to describe which
scenarios and variant combinations are valid for a given artifact and
allow enos to create a valid sample field (a matrix of all compatible
scenarios) and take an observation (select some to run) for us. This
ensures that every valid scenario and variant combination will
now be a candidate for testing in the pipeline. See QT-504[0] for further
details on the Enos sampling capabilities.

Our prior implementation only tested the amd64 and arm64 zip artifacts,
as well as the Docker container. We now include the following new artifacts
in the test matrix:
* CE Amd64 Debian package
* CE Amd64 RPM package
* CE Arm64 Debian package
* CE Arm64 RPM package

Each artifact includes a sample definition for both pre-merge/post-merge
(build) and release testing.

Changes:
* Remove the hand crafted `enos-run-matrices` ci matrix targets and replace
  them with per-artifact samples.
* Use enos sampling to generate different sample groups on all pull
  requests.
* Update the enos scenario matrices to handle HSM and FIPS packages.
* Simplify enos scenarios by using shared globals instead of
  cargo-culted locals.

Note: This will require coordination with vault-enterprise to ensure a
smooth migration to the new system. Integrating new scenarios or
modifying existing scenarios/variants should be much smoother after this
initial migration.

[0] https://github.com/hashicorp/enos/pull/102

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-09-08 12:46:32 -06:00
Sarah Thompson
a9a4b0b9ff
Onboard Vault to CRT version bump automation (#18311)
* adding new version bump refactoring

* address comments

* remove changes used for testing

* add the version bump event!

* fix local enos scenarios

* remove unnecessary local get_local_metadata steps from scenarios
* add version base, pre, and meta to the get_local_metadata module
* use the get_local_metadata module in the local builder for version
  metadata
* update the version verifier to always require a build date

Signed-off-by: Ryan Cragun <me@ryan.ec>

* Update to embed the base version from the VERSION file directly into version.go.
This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs.
We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos).

* update comment

* fixing bad merge

* removing actions-go-build as it won't work with the latest go caching changes

* fix logic for getting version in enos-lint.yml

* fix version number

* removing unneeded module

---------

Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Claire <claire@hashicorp.com>
Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 17:08:48 +01:00
hashicorp-copywrite[bot]
0b12cdcfd1
[COMPLIANCE] License changes (#22290)
* Adding explicit MPL license for sub-package.

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Adding explicit MPL license for sub-package.

This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.

* Updating the license from MPL to Business Source License.

Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.

* add missing license headers

* Update copyright file headers to BUS-1.1

* Fix test that expected exact offset on hcl file

---------

Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2023-08-10 18:14:03 -07:00
Ryan Cragun
6b21994d76
[QT-588] test: fix drift between enos directories (#21695)
* Sync missing scenarios and modules
* Clean up variables and examples vars
* Add a `lint` make target for enos
* Update enos `fmt` workflow to run the `lint` target.
* Always use ipv4 addresses in target security groups.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-20 14:09:44 -06:00
Ryan Cragun
aed2783658
enos: use on-demand targets (#21459)
Add an updated `target_ec2_instances` module that is capable of
dynamically splitting target instances over subnet/az's that are
compatible with the AMI architecture and the associated instance type
for the architecture. Use the `target_ec2_instances` module where
necessary. Ensure that `raft` storage scenarios don't provision
unnecessary infrastructure with a new `target_ec2_shim` module.

After a lot of trial, the state of Ec2 spot instance capacity, their
associated APIs, and current support for different fleet types in AWS
Terraform provider, have proven to make using spot instances for
scenario targets too unreliable.

The current state of each method:
* `target_ec2_fleet`: unusable due to the fact that the `instant` type
  does not guarantee fulfillment of either `spot` or `on-demand`
  instance request types. The module does support both `on-demand` and
  `spot` request types and is capable of bidding across a maximum of
  four availability zones, which makes it an attractive choice if the
  `instant` type would always fulfill requests. Perhaps a `request` type
  with `wait_for_fulfillment` option like `aws_spot_fleet_request` would
  make it more viable for future consideration.
* `target_ec2_spot_fleet`: more reliable if bidding for target instances
  that have capacity in the chosen zone. Issues in the AWS provider
  prevent us from bidding across multiple zones succesfully. Over the
  last 2-3 months target capacity for the instance types we'd prefer to
  use has dropped dramatically and the price is near-or-at on-demand.
  The volatility for nearly no cost savings means we should put this
  option on the shelf for now.
* `target_ec2_instances`: the most reliable method we've got. It is now
  capable of automatically determing which subnets and availability
  zones to provision targets in and has been updated to be usable for
  both Vault and Consul targets. By default we use the cheapest medium
  instance types that we've found are reliable to test vault.

* Update .gitignore
* enos/modules/create_vpc: create a subnet for every availability zone
* enos/modules/target_ec2_fleet: bid across the maximum of four
  availability zones for targets
* enos/modules/target_ec2_spot_fleet: attempt to make the spot fleet bid
  across more availability zones for targets
* enos/modules/target_ec2_instances: create module to use
  ec2:RunInstances for scenario targets
* enos/modules/target_ec2_shim: create shim module to satisfy the
  target module interface
* enos/scenarios: use target_ec2_shim for backend targets on raft
  storage scenarios
* enos/modules/az_finder: remove unsed module

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-06-26 16:06:03 -06:00
Ryan Cragun
5de6af6076
enos: use linux/amd64 for consul storage backend (#21436)
We seem to hit occasional capacity issues when attempting to launch spot
fleets with arm64 instance types. After checking pricing in the regions
that we use, it appears that current and older generation amd64 t2 and
t3 instance types are running at quite a discount whereas t4 arm64
instances are barely under on-demand price, suggesting limited capacity
for arm64 spot instances at this time. We'll change our default backend
instance architecture to amd64 to bid for the cheaper t2 and t3
instances and increase our `max_price` globally to that of a RHEL
machine running on-demand with a t3.medium.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-06-22 22:28:52 +00:00
Ryan Cragun
8d22142a3e
[QT-572][VAULT-17391] enos: use ec2 fleets for consul storage scenarios (#21400)
Begin the process of migrating away from the "strongly encouraged not to
use"[0] Ec2 spot fleet API to the more modern `ec2:CreateFleet`.
Unfortuantely the `instant` type fleet does not guarantee fulfillment
with either on-demand or spot types. We'll need to add a feature similar
to `wait_for_fulfillment` on the `spot_fleet_request` resource[1] to
`ec2_fleet` before we can rely on it.

We also update the existing target fleets to support provisioning generic
targets. This has allowed us to remove our usage of `terraform-enos-aws-consul`
and replace it with a smaller `backend_consul` module in-repo.

We also remove `terraform-enos-aws-infra` and replace it with two smaller
in-repo modules `ec2_info` and `create_vpc`. This has allowed us to simplify
the vpc resources we use for each scneario, which in turn allows us to
not rely on flaky resources.

As part of this refactor we've also made it possible to provision
targets using different distro versions.

[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html#which-spot-request-method-to-use
[1] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/spot_fleet_request#wait_for_fulfillment

* enos/consul: add `backend_consul` module that accepts target hosts.
* enos/target_ec2_spot_fleet: add support for consul networking.
* enos/target_ec2_spot_fleet: add support for customizing cluster tag
  key.
* enos/scenarios: create `target_ec2_fleet` which uses a more modern
  `ec2_fleet` API.
* enos/create_vpc: replace `terraform-enos-aws-infra` with smaller and
  simplified version. Flatten the networking to a single route on the
  default route table and a single subnet.
* enos/ec2_info: add a new module to give us useful ec2 information
  including AMI id's for various arch/distro/version combinations.
* enos/ci: update service user role to allow for managing ec2 fleets.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-06-22 12:42:21 -06:00
Ryan Cragun
2ec5a28f51
test: handle occasional lower capacity in zone d (#21143)
We seen instances where we try to schedule a spot fleet in the
us-east-1d of the vault CI AWS account and cannot get capacity for our
instance type. That zone currently supports far fewer instance types so
we'll bump our max bid to handle cases where slightly more expensive
instances are available. Most of the time we'll be using much cheaper
instances but it's better to pay a fraction of a cent more than have to
retry the pipeline. As such, we increase our max bid price to something
that will almost certainly be fullfilled.

We also allow our package installer to go ahead when cloud init does not
update sources like we expect. This should handle occasional failures
where cloud-init doesn't update the sources within a reasonable amount
of time.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-06-12 10:49:58 -06:00
Ryan Cragun
27621e05d6
[QT-527][QT-509] enos: use latest version of enos-provider (#21129)
Use the latest version of enos-provider and upstream consul module.
These changes allow us to configure the vault log level in configuration
and also support configuring consul with an enterprise license.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-06-12 10:00:16 -04:00
Mike Baum
0115b5e43a
[QT-426] Add support for enabling the file audit device for enos scenarios (#20552) 2023-06-02 13:07:33 -04:00
Ryan Cragun
1329a6b506
[QT-525] enos: use spot instances for Vault targets (#20037)
The previous strategy for provisioning infrastructure targets was to use
the cheapest instances that could reliably perform as Vault cluster
nodes. With this change we introduce a new model for target node
infrastructure. We've replaced on-demand instances for a spot
fleet. While the spot price fluctuates based on dynamic pricing, 
capacity, region, instance type, and platform, cost savings for our
most common combinations range between 20-70%.

This change only includes spot fleet targets for Vault clusters.
We'll be updating our Consul backend bidding in another PR.

* Create a new `vault_cluster` module that handles installation,
  configuration, initializing, and unsealing Vault clusters.
* Create a `target_ec2_instances` module that can provision a group of
  instances on-demand.
* Create a `target_ec2_spot_fleet` module that can bid on a fleet of
  spot instances.
* Extend every Enos scenario to utilize the spot fleet target acquisition
  strategy and the `vault_cluster` module.
* Update our Enos CI modules to handle both the `aws-nuke` permissions
  and also the privileges to provision spot fleets.
* Only use us-east-1 and us-west-2 in our scenario matrices as costs are
  lower than us-west-1.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 15:44:43 -04:00
Hamid Ghaf
e55c18ed12
adding copyright header (#19555)
* adding copyright header

* fix fmt and a test
2023-03-15 09:00:52 -07:00
Jaymala
65cb09c75f
Add Vault log level support (#19083)
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
2023-02-08 17:41:16 -05:00
Ryan Cragun
206db2f529
[QT-436] Pseudo random artifact test scenarios (#18056)
Introducing a new approach to testing Vault artifacts before merge
and after merge/notorization/signing. Rather than run a few static
scenarios across the artifacts, we now have the ability to run a
pseudo random sample of scenarios across many different build artifacts.

We've added 20 possible scenarios for the AMD64 and ARM64 binary
bundles, which we've broken into five test groups. On any given push to
a pull request branch, we will now choose a random test group and
execute its corresponding scenarios against the resulting build
artifacts. This gives us greater test coverage but lets us split the
verification across many different pull requests.

The post-merge release testing pipeline behaves in a similar fashion,
however, the artifacts that we use for testing have been notarized and
signed prior to testing. We've also reduce the number of groups so that
we run more scenarios after merge to a release branch.

We intend to take what we've learned building this in Github Actions and
roll it into an easier to use feature that is native to Enos. Until then,
we'll have to manually add scenarios to each matrix file and manually
number the test group. It's important to note that Github requires every
matrix to include at least one vector, so every artifact that is being
tested must include a single scenario in order for all workflows to pass
and thus satisfy branch merge requirements.

* Add support for different artifact types to enos-run
* Add support for different runner type to enos-run
* Add arm64 scenarios to build matrix
* Expand build matrices to include different variants
* Update Consul versions in Enos scenarios and matrices
* Refactor enos-run environment
* Add minimum version filtering support to enos-run. This allows us to
  automatically exclude scenarios that require a more recent version of
  Vault
* Add maximum version filtering support to enos-run. This allows us to
  automatically exclude scenarios that require an older version of
  Vault
* Fix Node 12 deprecation warnings
* Rename enos-verify-stable to enos-release-testing-oss
* Convert artifactory matrix into enos-release-testing-oss matrices
* Add all Vault editions to Enos scenario matrices
* Fix verify version with complex Vault edition metadata
* Rename the crt-builder to ci-helper
* Add more version helpers to ci-helper and Makefile
* Update CODEOWNERS for quality team
* Add support for filtering matrices by group and version constraints
* Add support for pseudo random test scenario execution

Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 13:46:04 -07:00
Ryan Cragun
e445c8b4f5
enos: use amd64 for consul backend (#17917)
Previously we'd pass the matrix variant to the backend cluster module which is currently unsupported by the consul module. Instead we'll always pass the ubuntu/amd64 AMI ID to the consul backend module.

This should resolve the enos-verify-stable matrix failures here https://github.com/hashicorp/vault/actions/runs/3448137968/jobs/5754873688#step:11:191

Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-11-15 11:44:05 -07:00
Ryan Cragun
fc9dfa2a7d
[QT-358] Unify CRT and local builder workflows (#17766)
Here we make the following major changes:

* Centralize CRT builder logic into a script utility so that we can share the
  logic for building artifacts in CI or locally.
* Simplify the build workflow by calling a reusable workflow many times
  instead of repeating the contents.
* Create a workflow that validates whether or not the build workflow and all
  child workflows have succeeded to allow for merge protection.

Motivation

* We need branch requirements for the build workflow and all subsequent
  integration tests (QT-353)
* We need to ensure that the Enos local builder works (QT-558)
* Debugging build failures can be difficult because one has to hand craft the
  steps to recreate the build
* Merge conflicts between Vault OSS and Vault ENT build workflows are quite
  painful. As the build workflow must be the same file and name we'll reduce
  what is contained in each that is unique. Implementations of building
  will be unique per edition so we don't have to worry about conflict
  resolution.
* Since we're going to be touching the build workflow to do the first two
  items we might as well try and improve those other issues at the same time
  to reduce the overhead of backports and conflicts.

Considerations

* Build logic for Vault OSS and Vault ENT differs
* The Enos local builder was duplicating a lot of what we did in the CRT build
  workflow
* Version and other artifact metadata has been an issue before. Debugging it
  has been tedious and error prone.
* The build workflow is full of brittle copy and paste that is hard to
  understand, especially for all of the release editions in Vault Enterprise
* Branch check requirements for workflows are incredibly painful to use for
  workflows that are dynamic or change often. The required workflows have to be
  configured in Github settings by administrators. They would also prevent us
  from having simple docs PRs since required integration workflows always have
  to run to satisfy branch requirements.
* Doormat credentials requirements that are coming will require us to modify
  which event types trigger workflows. This changes those ahead of time since
  we're doing so much to build workflow. The only noticeable impact will be
  that the build workflow no longer runs on pushes to non-main or release
  branches. In order to test other branches it requires a workflow_dispatch
  from the Actions tab or a pull request.

Solutions

* Centralize the logic that determines build metadata and creates releasable
  Vault artifacts. Instead of cargo-culting logic multiple times in the build
  workflow and the Enos local modules, we now have a crt-builder script which
  determines build metadata and also handles building the UI, Vault, and the
  package bundle. There are make targets for all of the available sub-commands.
  Now what we use in the pipeline is the same thing as the local builder, and
  it can be executed locally by developers. The crt-builder script works in OSS
  and Enterprise so we will never have to deal with them being divergent or with
  special casing things in the build workflow.
* Refactor the bulk of the Vault building into a reusable workflow that we can
  call multiple times. This allows us to define Vault builds in a much simpler
  manner and makes resolving merge conflicts much easier.
* Rather than trying to maintain a list and manually configure the branch check
  requirements for build, we'll trigger a single workflow that uses the github
  event system to determine if the build workflow (all of the sub-workflows
  included) have passed. We'll then create branch restrictions on that single
  workflow down the line.

Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 13:14:43 -07:00
Violet Hynes
ab26576873
VAULT-9688 Vault Agent Enos test (#17837)
* VAULT-9688 First attempt at Vault Agent Enos test

* VAULT-9688 remove TODO, correct indentation

* VAULT-9688 enos fmt

* VAULT-9688 terraform fmt

* VAULT-9688 small updates

* VAULT-9688 add extra comment
2022-11-08 10:08:43 -05:00