Barry Chen
857739dd4c
refactor: stop passing collection id to get_quota_usage ( #2170 )
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-env (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-env (push) Has been cancelled
Build, Tag and Push Container Images to GAR / check (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Main Workflow - Lint, Build, Test / python-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / rust-checks (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (mysql) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (postgres) (push) Has been cancelled
Main Workflow - Lint, Build, Test / clippy (spanner) (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-postgres (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-postgres-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / postgres-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-mysql (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-mysql-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / mysql-e2e-tests (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-and-unit-test-spanner (push) Has been cancelled
Main Workflow - Lint, Build, Test / build-spanner-image (push) Has been cancelled
Main Workflow - Lint, Build, Test / spanner-e2e-tests (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-mysql (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
2026-03-28 21:19:42 -05:00
Barry Chen
c1d53b604f
feat: (re-)enable timestamp+offset based pagination optimization ( #2145 )
...
Checks / python-checks (push) Has been cancelled
Checks / rust-checks (push) Has been cancelled
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR / check (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR / build-and-push-syncserver-mysql (push) Has been cancelled
MySQL Build and Test / build-and-test-mysql (push) Has been cancelled
MySQL Build and Test / build-mysql-image (push) Has been cancelled
MySQL Build and Test / mysql-e2e-tests (push) Has been cancelled
Postgres Build and Test / build-and-test-postgres (push) Has been cancelled
Postgres Build and Test / build-postgres-image (push) Has been cancelled
Postgres Build and Test / postgres-e2e-tests (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Spanner Build, Test, and Push / build-and-test-spanner (push) Has been cancelled
Spanner Build, Test, and Push / build-spanner-image (push) Has been cancelled
Spanner Build, Test, and Push / spanner-e2e-tests (push) Has been cancelled
* feat: (re-)enable timestamp+offset based pagination optimization
https://github.com/mozilla-services/syncstorage-rs/issues/53 enabled
https://github.com/mozilla-services/syncstorage-rs/issues/559 disabled
And this patch enables it for MySQL and Postgres but not Spanner.
* Apply suggestion from @pjenvey
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
---------
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2026-03-26 14:28:03 -05:00
Philip Jenvey
019bf46adf
feat: adapt PoolState to usage of deadpool everywhere ( #2128 )
...
Checks / python-checks (push) Has been cancelled
Checks / rust-checks (push) Has been cancelled
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-enterprise-gar (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-python-utils-enterprise-gar (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-mysql (push) Has been cancelled
MySQL Build and Test / build-and-test-mysql (push) Has been cancelled
MySQL Build and Test / build-mysql-image (push) Has been cancelled
MySQL Build and Test / mysql-e2e-tests (push) Has been cancelled
Postgres Build and Test / build-and-test-postgres (push) Has been cancelled
Postgres Build and Test / build-postgres-image (push) Has been cancelled
Postgres Build and Test / postgres-e2e-tests (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Spanner Build, Test, and Push / build-and-test-spanner (push) Has been cancelled
Spanner Build, Test, and Push / build-spanner-image (push) Has been cancelled
Spanner Build, Test, and Push / spanner-e2e-tests (push) Has been cancelled
just killing it for deadpool::Status since it provides a public constructor
and kill the long unused syncstorage-spanner bb8 based manager
Closes STOR-368
2026-03-17 15:15:31 -07:00
Philip Jenvey
0d780c74cf
fix: get_storage_timestamp/lock_for_read should ignore PRETOUCH_DT ( #2067 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncserver-mysql (push) Has been cancelled
MySQL Build and Test / build-and-test-mysql (push) Has been cancelled
Postgres Build and Test / build-and-test-postgres (push) Has been cancelled
Publish Sync docs to pages / build-mdbook (push) Has been cancelled
Publish Sync docs to pages / build-openapi (push) Has been cancelled
Spanner Build, Test, and Push / build-and-test-spanner (push) Has been cancelled
MySQL Build and Test / build-mysql-image (push) Has been cancelled
MySQL Build and Test / mysql-e2e-tests (push) Has been cancelled
Postgres Build and Test / build-postgres-image (push) Has been cancelled
Postgres Build and Test / postgres-e2e-tests (push) Has been cancelled
Publish Sync docs to pages / combine-and-prepare (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Spanner Build, Test, and Push / build-spanner-image (push) Has been cancelled
Spanner Build, Test, and Push / spanner-e2e-tests (push) Has been cancelled
Closes STOR-484
2026-02-13 14:21:36 -08:00
Barry Chen
0ddc3b0af1
chore: upgrade to Rust 2024 edition ( #2048 )
2026-02-11 11:00:29 -08:00
Philip Jenvey
935b468cb0
feat: optimize postgres get_or_create_collection_id ( #2028 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-mysql (push) Has been cancelled
MySQL Build and Test / build-and-test-mysql (push) Has been cancelled
MySQL Build and Test / build-mysql-image (push) Has been cancelled
MySQL Build and Test / mysql-e2e-tests (push) Has been cancelled
MySQL Build and Test / deploy-mysql-dockerhub (push) Has been cancelled
Postgres Build and Test / build-and-test-postgres (push) Has been cancelled
Postgres Build and Test / build-postgres-image (push) Has been cancelled
Postgres Build and Test / postgres-e2e-tests (push) Has been cancelled
Postgres Build and Test / deploy-postgres-dockerhub (push) Has been cancelled
Publish Sync docs to pages / build (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Spanner Build, Test, and Push / build-and-test-spanner (push) Has been cancelled
Spanner Build, Test, and Push / build-spanner-image (push) Has been cancelled
Spanner Build, Test, and Push / spanner-e2e-tests (push) Has been cancelled
Spanner Build, Test, and Push / deploy-spanner-dockerhub (push) Has been cancelled
making it (and mysql's) match spanner's, also fixing/simplifying their
create_collection test method
and fix mysql/postgres transactions to begin first and foremost
Closes STOR-465
Closes STOR-371
2026-01-29 10:46:26 -08:00
Philip Jenvey
1999620cdd
fix per e2e test_bulk_update_of_ttls_without_sending_data
2026-01-20 12:45:40 -08:00
Philip Jenvey
eff8f805d4
feat: optimize post_bsos w/ MERGE INTO
...
ideally we'd pass data to MERGE INTO's USING w/ a VALUES statement,
but rendering VALUES here w/ diesel's QueryFragment trait is painful
instead it's UNNEST'ed from a single bind parameter of post_bso[] (an
Array of a new, named composite type). this composite type must be
explicitly named/declared as postgres disallows unnamed/anonymous
composite types to be used as bind parameters
Closes STOR-423
2026-01-20 12:45:36 -08:00
Philip Jenvey
4b3992327e
refactor: extract a PgDb::check_quota method
...
and misc small cleanup
2026-01-20 12:21:49 -08:00
Philip Jenvey
9ed0d0c2cd
feat: optimize batch_commit w/ MERGE INTO ( #2003 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres-python-utils (push) Has been cancelled
Publish Sync docs to pages / build (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
Closes STOR-457
2026-01-15 09:33:23 -08:00
Taddes
77868a3890
docs: sync api docs github ( #1986 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-spanner-python-utils (push) Has been cancelled
Build, Tag and Push Container Images to GAR Repository / build-and-push-syncstorage-rs-postgres-python-utils (push) Has been cancelled
Publish Sync docs to pages / build (push) Has been cancelled
Publish Sync docs to pages / deploy (push) Has been cancelled
docs: sync api docs github
2026-01-09 10:38:46 -05:00
Philip Jenvey
0cca7c7338
feat: get the timestamp from postgres ( #1962 )
...
also enable cfg(debug_assertions) for Db test methods (where we easily
can)
Closes STOR-441
2026-01-06 13:15:25 -08:00
Taddes
8b15ac53de
bug: address reserve and custom collections ( #1950 )
...
address reserve and custom collections
2025-12-19 16:22:25 -05:00
Philip Jenvey
daec191739
feat: re-enable warnings ( #1947 )
...
and update some comments
Closes STOR-430
2025-12-19 11:30:25 -08:00
Barry Chen
424ab1e7c9
fix: preserve existing value on batch append if new value is null/empty ( #1943 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2025-12-19 06:43:12 -08:00
Taddes
ad12d1873e
update collections for standard collections ( #1935 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
update collections for standard collections
2025-12-17 20:04:58 -05:00
Barry Chen
acc060dd45
Fix: use PRETOUCH_DT when inserting new user collection in Postgres ( #1938 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2025-12-17 11:58:40 -08:00
Barry Chen
a94d702ce7
fix: ensure Postgres return types and Rust value types match ( #1940 )
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
2025-12-16 18:44:37 -08:00
Philip Jenvey
5c022c04df
feat: switch to TIMESTAMPTZ ( #1932 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
- make SyncTimestamp::from_datetime generic over timezones
- compare expiry to CURRENT_TIMESTAMP now that they're TIMESTAMPTZ
- fix get_collection_timestamps not ignoring tombstones
- fix the gcs upload breaking due to failing tests
Closes STOR-420
2025-12-16 13:50:21 -08:00
Barry Chen
dc228c6a9e
Fix: ensure user collection exists before adding bsos or batches ( #1934 )
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
2025-12-15 11:32:28 -08:00
Barry Chen
9bc614d7b9
fix: set collection id col to auto-incr and fix get_collection_id ( #1929 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Inserting into collections in `get_or_create_collection_id` was failing.
The wrong column name was used in `get_collection_id`. It's also been
converted to Diesel ORM calls.
Closes STOR-422
2025-12-13 11:58:29 -08:00
Taddes
56c21e0fef
feat: UserIdentifier modification in lock and timestamp methods ( #1927 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
UserIdentifier modification in lock and timestamp methods
2025-12-12 21:02:49 -05:00
Philip Jenvey
cb8e620a39
feat: support query logging via diesel's instrumentation ( #1925 )
...
* feat: support query logging via diesel's instrumentation
- upgrades to latest diesel-async/diesel for diesel-async's new
migration harness
- removes diesel_logger
Closes STOR-417
2025-12-12 15:43:37 -08:00
Taddes
855066ced4
feat: create post/put bsos postgres ( #1911 )
...
feat: create post/put bsos postgres
2025-12-12 16:04:01 -05:00
Barry Chen
45b45a7d56
Update doc and types to match latest Postgres schema
2025-12-08 16:20:44 -06:00
Barry Chen
4bc0f3a47b
Review feedback changes
...
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2025-12-08 15:56:45 -06:00
Barry Chen
4208f037f1
feat: impl {create,commit,append_to,get}_batch for Postgres
2025-12-08 15:56:45 -06:00
Taddes
278ffcc74a
feat: quota methods ( #1920 )
...
feat: quota methods
2025-12-08 15:41:18 -05:00
Philip Jenvey
1edfd870b3
feat: add the remaining postgres get methods ( #1912 )
...
* feat: add the remaining postgres get methods
- add a FromSql<Timestamp, _> for SyncTimestamp
- utilize Selectable for GetBso
- simplify bsos_query to rely on type inference of the result
Closes STOR-337
2025-12-08 11:53:41 -08:00
Barry Chen
8790e5a207
Use SyncTimestamp::as_naive_datetime
...
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2025-12-04 16:43:02 -06:00
Barry Chen
f548ea1fc3
feat: add delete fns for Postgres db impls
2025-12-04 15:39:00 -06:00
Taddes
6b0b41c6d5
feat: impl for update collection ( #1900 )
...
feat: impl for update collection
2025-12-04 15:41:04 -05:00
Taddes
4d6fe26a35
feat: postgres read and write locks ( #1891 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: postgres read and write locks
2025-12-02 20:38:45 -05:00
Philip Jenvey
f3a33250b6
feat: add get_bsos/bso_ids ( #1899 )
...
- switch sortindex to 32 bit: the API defines it as 9 digits max
- switch payload to TEXT
Closes STOR-336
Closes STOR-402
2025-12-02 15:32:54 -08:00
Taddes
455ad7aed2
feat: load and map collections impl ( #1898 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: load and map collections impl
2025-11-20 18:17:19 -05:00
Philip Jenvey
70a7854b8f
Merge pull request #1896 from mozilla-services/feat/syncstorage-uid-pk-STOR-413
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: prefer tokenserver's assigned uid as the PK
2025-11-18 14:25:36 -08:00
Taddes
ac10f5a7c2
feat: impl get collection and collection id methods ( #1887 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: impl get collection and collection id methods
2025-11-13 14:08:13 -05:00
Philip Jenvey
30bb0556f6
refactor: divvy up the db module
...
Closes STOR-304
2025-11-10 13:45:13 -08:00
Taddes
a65d461094
feat: postgres db pool and session implementation ( #1875 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
We need to create the Postgres specific Db struct and associated methods. The majority of API methods will be implemented upon this struct, so this is to initialize the main logic so that the traits can be implemented.
This also encompasses additions to the Postgres-specific database pool implementation.
Several of the trait bounds (methods) have to be satisfied, but this stubs them out so that the PgDb has a common interface that is the same as those for MySQL and Spanner.
Doc comments added in shared error.rs crate for clarity, since it is a shared interface for all databases.
Removed dead code annotations for clippy that are no longer needed in Tokenserver.
Closes STOR-399
2025-11-10 13:14:22 -08:00
Taddes
7fa191e699
feat: create schema and orm models for sync postgres ( #1873 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: create schema and orm models for sync postgres
2025-10-31 18:17:34 -04:00
Taddes
0c1198496d
feat: sync postgres schema ( #1853 )
...
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
feat: sync postgres schema
2025-10-28 17:53:33 -04:00
Philip Jenvey
bcc5990eea
feat: add syncstorage-postgres
...
Closes STOR-330
2025-10-17 16:13:05 -07:00
