dependabot[bot]
525dba3a97
chore(deps): bump cryptography from 46.0.6 to 46.0.7 in /tools/spanner ( #2223 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (mysql) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (postgres) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (spanner) (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-postgres (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-postgres-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / postgres-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-mysql (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-mysql-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / mysql-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-spanner (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-spanner-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / spanner-e2e-tests (push) Has been cancelled
Build, Tag and Push Container Images to GAR / check (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-mysql (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Bumps [cryptography](https://github.com/pyca/cryptography ) from 46.0.6 to 46.0.7.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/46.0.6...46.0.7 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-version: 46.0.7
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-22 12:24:21 +03:00
dependabot[bot]
e32579f251
chore(deps-dev): bump pytest from 9.0.2 to 9.0.3 in /tools/spanner ( #2221 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (mysql) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (postgres) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (spanner) (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-postgres (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-postgres-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / postgres-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-mysql (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-mysql-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / mysql-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-spanner (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-spanner-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / spanner-e2e-tests (push) Has been cancelled
Build, Tag and Push Container Images to GAR / check (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-mysql (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3 )
---
updated-dependencies:
- dependency-name: pytest
dependency-version: 9.0.3
dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 16:15:45 +03:00
Taddes
af1c5fb68a
feat: add python utils and integrate into workflow ( #2176 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (mysql) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (postgres) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (spanner) (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-postgres (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-postgres-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / postgres-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-mysql (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-mysql-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / mysql-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-spanner (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-spanner-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / spanner-e2e-tests (push) Has been cancelled
Build, Tag and Push Container Images to GAR / check (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-mysql (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
feat: add python utils and integrate into workflow
2026-04-01 16:02:34 -04:00
Taddes
6b09e9947a
chore: upgrade Python for all utils and refactor ( #2127 )
...
Checks / python-checks (push) Has been cancelled
Checks / rust-checks (push) Has been cancelled
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / check (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-enterprise-gar (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-python-utils-enterprise-gar (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-mysql (push) Has been cancelled
MySQL Build and Test / build-and-test-mysql (push) Has been cancelled
MySQL Build and Test / build-mysql-image (push) Has been cancelled
MySQL Build and Test / mysql-e2e-tests (push) Has been cancelled
Postgres Build and Test / build-and-test-postgres (push) Has been cancelled
Postgres Build and Test / build-postgres-image (push) Has been cancelled
Postgres Build and Test / postgres-e2e-tests (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Spanner Build, Test, and Push / build-and-test-spanner (push) Has been cancelled
Spanner Build, Test, and Push / build-spanner-image (push) Has been cancelled
Spanner Build, Test, and Push / spanner-e2e-tests (push) Has been cancelled
chore: upgrade Python for all utils and refactor
2026-03-24 16:23:04 -04:00
Barry Chen
33d41fbf3b
chore: poetry update python deps ( #2063 )
2026-02-13 11:53:36 -08:00
dependabot[bot]
cccdac909d
chore(deps): bump pyasn1 from 0.6.1 to 0.6.2 in /tools/spanner ( #2008 )
...
Bumps [pyasn1](https://github.com/pyasn1/pyasn1 ) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/pyasn1/pyasn1/releases )
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst )
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.1...v0.6.2 )
---
updated-dependencies:
- dependency-name: pyasn1
dependency-version: 0.6.2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-16 14:07:34 -08:00
Philip Jenvey
5f8ab5483f
chore: kill remaining requirements.txt
...
they're only used in the docker build and generated on the fly by
poetry there
2026-01-15 16:32:58 -08:00
dependabot[bot]
ea0055df75
chore(deps): bump urllib3 from 2.6.0 to 2.6.3 in /tools/spanner ( #1994 )
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 2.6.0 to 2.6.3.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.3 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-version: 2.6.3
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-15 08:24:23 -08:00
dependabot[bot]
78ee28e70a
chore(deps): bump urllib3 from 2.5.0 to 2.6.0 in /tools/spanner
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-version: 2.6.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-12-06 05:26:51 +00:00
Taddes
19b6176d5b
chore: update sync python version ( #1774 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
chore: update sync python version
2025-08-28 20:27:58 -04:00
Taddes
8995db6268
feat: ruff for python lint and format ( #1742 )
...
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
feat: ruff for python lint and format
2025-08-12 20:26:55 -04:00
Taddes
77254b4a6e
refactor: python imports ( #1730 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
refactor: unified python imports for consistency in local, production, and test environments
2025-08-04 15:37:46 -04:00
Taddes
f8715d4e91
feat: use poetry for dependency management ( #1706 )
...
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
feat: use poetry for dependency management
2025-07-28 21:48:46 -04:00
Taddes
d716ac5d10
feat: spanner scripts parse gcp project ( #1714 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: spanner scripts parse gcp project
2025-07-22 20:18:30 -04:00
JR Conlin
8f9e1c27cf
chore: Update to debian bookworm / Python 3.12 ( #1567 )
...
* chore: Update to debian bookworm / Python 3.12
2024-06-10 07:22:57 -07:00
JR Conlin
1b9ebbf40f
Chore/contributor rollup ( #1479 )
...
* These needed to be rolled in due to various dependency changes.
* Fix the URL people should set to use their Custom Server (https://github.com/mozilla-services/syncstorage-rs/pull/1453 ) -
[Ekleog](https://github.com/Ekleog )
* Make docker files more podman friendly (https://github.com/mozilla-services/syncstorage-rs/pull/1431 ) -
[mb](https://github.com/mb )
* This PR also updates a number of dependencies
2023-10-18 14:24:41 -07:00
Ethan Donowitz
0ae5fd2059
refactor: add settings crates ( #1306 )
...
This is a breaking change. This commit separates syncstorage and tokenserver settings into separate structs that are contained by a parent `Settings` struct. This means that any env vars that hold settings specific to syncstorage (e.g. `SYNC_DATABASE_URL`) have been renamed to `SYNC_SYNCSTORAGE__DATABASE_URL`. Any settings that were moved from the top-level `Settings` struct to the lower level, syncstorage-specific struct will now have a `SYNC_SYNCSTORAGE__` prefix instead of a `SYNC_` prefix.
Closes #1276
2022-10-19 17:15:56 -04:00
JR Conlin
487ac11ed0
feat: Add "auto-split" arg to auto-gen UID prefixes ( #1035 )
...
Closes : #1034
2021-04-02 15:34:20 -07:00
JR Conlin
a79f8407de
feat: switch from regex_contains to starts_with ( #805 )
...
* added add'l command aliases
* `--ids` == `--collection_ids`
* `--prefix` == `--uid_prefixes`
* switched to parameterized DML
Issue #799
Co-authored-by: Donovan Preston <donovanpreston@gmail.com>
2020-09-01 11:11:38 -04:00
jrconlin
7263202c59
f r's
2020-08-25 19:44:52 +00:00
jrconlin
8cfbb8510c
f add conditions to batches
2020-08-24 16:00:47 +00:00
jrconlin
714168d107
bug: fix purge_ttl advanced features
...
* fix `fxa_uaid` typo
* fix `collection_ids` handler to deal with default empty ids.
Closes #799
2020-08-24 15:13:26 +00:00
Mark Drobnak
59aa28a4e5
feat: More purge_ttl features ( #776 )
...
* Support a mode option in purge_ttl
* Support an expiry mode option in purge_ttl
* Support serially deleting prefixes by regex
Also changes the collection IDs option from a JSON list to an args list
(i.e. [item1,item2,item3]).
Closes #735
Closes #743
2020-08-11 15:05:11 -04:00
jrconlin
695722a9b5
feat: option to limit purgettl to range of fxa_uids
...
Added `--uid_starts` option which will limit purge_ttl bso deletes to
fxa_uids that begin with specified characters.
(e.g.
`purge_ttl.py --uid_starts="AA" --collection_ids=[7,8]`
will limit to scanning bso tables to fxa_uid values that begin with "AA"
and only the 7 & 8 collections
Closes #713
2020-07-07 14:23:58 +00:00
JR Conlin
198eb816bc
feat: limit purge ttl to prior midnight ( #708 )
...
Closes #707
Co-authored-by: Donovan Preston <donovanpreston@gmail.com>
2020-07-02 07:54:59 -04:00
JR Conlin
2a14eb2973
feat: add conditions, args to purge_ttl script ( #668 )
...
* feat: add conditions, args to purge_ttl script
attempt to try and provide a way to allow the purge_ttl script to
complete.
* Adds arguments (ENV VARS):
--instance_id (INSTANCE_ID) Spanner instance id
--database_id (DATABASE_ID) Spanner database id
--sync_database_url (SYNC_DATABASE_URL) Spanner DSN
`spanner://instance/database`
--collection_ids (COLLECTION_IDS)
JSON formatted list of collections to limit deletions
e.g. `--collection_ids=123` limits to just collection 123
`--collection_ids=[123,456]` limits to both 123 & 456
default is all collections
Issue #631
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2020-06-12 15:35:39 -07:00
Erik
2060601c48
feat: build spanner python utils image ( #661 )
...
feat: build spanner python utils image
2020-06-10 22:53:33 -07:00
Donovan Preston
dd768048d1
Move purge_ttl to src/bin/purge_ttl.rs to save 10 minutes on ci
2020-04-09 16:22:54 -04:00
Donovan Preston
da5741b2fe
Merge branch 'master' into release/0.2b-merge
2020-04-02 14:50:39 -04:00
Rachel Tublitz
530b61c5de
Merge branch 'master' into fix-README-typos
2020-04-02 09:56:33 -04:00
Eman
7da2154bcc
docs: fix typos in README.md files
...
Fixed typos in README.md files to improve readiblity.
Closes #529
2020-04-02 13:24:21 +03:00
Philip Jenvey
f58b3bc9b7
feat: add an extra sanity check of the db url
...
via Url::parse
2020-03-31 15:26:44 -07:00
Philip Jenvey
ec1b73a630
wip
2020-03-27 13:00:40 -07:00
Philip Jenvey
0930a84f6f
Merge branch 'master' into release/0.2b-merge
2020-03-26 16:46:38 -07:00
Philip Jenvey
adf039e651
merge master
2020-03-26 12:40:33 -07:00
jrconlin
7825ead153
chore: Update dependencies 2020-03
...
* Update vendor libs to use protobuf 2_11_0
* removed unused vendored subcomponents (bigtable, pubsub)
* updated README.md to include better instructions on how to do this
again in the future
* Temporarily disable cargo audit due to sudden spike in package
maintenance triggering deployment warnings
* updated generate.sh script to only generated needed components, and
remind about mod.rs
Closes #537
2020-03-26 09:03:53 -07:00
Philip Jenvey
9c784055e1
fix: allow hostnames for STATSD_HOST
...
and improve the missing db env error message
also don't package purge_ttl's target/ into the docker
Closes #548
2020-03-25 15:47:00 -07:00
Philip Jenvey
58f2051f42
chore: adapt googleapis-raw dep to 0.2 branch
...
(cherry picked from commit e6b795442bd9d7a31268a5f2c9cae8d92f614cc9)
2020-03-24 15:09:03 -07:00
mirefly
2d351956c2
refactor: rewrite purge_ttl in Rust
...
Closes : #385
(cherry picked from commit 5d6d7c1a8aef941134aae2ea24a8d3ed0c4a0c15)
2020-03-24 15:08:51 -07:00
mirefly
5d6d7c1a8a
refactor: rewrite purge_ttl in Rust
...
Closes : #385
2020-03-20 18:58:43 -06:00
Erik Olson
67fd4dd777
Remove old comments
2020-01-13 16:43:54 -05:00
Erik Olson
13d2490df4
feat: script to count total users in spanner
2020-01-13 15:05:50 -05:00
Erik Olson
c8134fda4a
purg_ttl.py pep8
2020-01-07 21:52:51 -05:00
Erik Olson
74b95544b0
purge_ttl.py updates for PR comments
2020-01-07 15:57:46 -05:00
Erik Olson
92a57e653d
feat: add basic logging to stdout and statsd metrics for purge_ttl.py
2020-01-07 15:51:45 -05:00
Rachel Tublitz
e1f48b48c8
chore: add python to docker image
...
also upgrading to rust 1.40.0 and adding a few details to docs for running via docker
closes : #384
2020-01-02 12:28:30 -05:00
jrconlin
7d07a8948f
feat: Add debugging tools
...
This adds a few useful inline testing tools for syncstorage-rs
2019-11-08 15:00:07 -08:00
Philip Jenvey
beddaf600f
refactor: schema renames (again)
...
- always prefix primary key names with their table name (id -> bso_id,
collection_id). a best practice in spanner, as when tables
interleave, their shared id names must always match
- bso -> bsos (every other table name's plural)
- give batches an expiry index
Closes #313
2019-11-04 11:26:19 -08:00
jrconlin
c085ff8b2c
f read the spanner data from the spanner DSN
2019-10-18 14:39:30 -07:00
jrconlin
30a77a9d1b
f r's
...
* make purge_ttl a real script
* pull data from env
* add `batches` purge
* delete to expiry
2019-10-18 13:58:45 -07:00
