* bug: Fix typos in tick, string replacements
* f multi-tread gen_bso_users
* added `--start_bso`, `--end_bso` to `gen_bso_users.py`
* added `bso_num` arg (same as `--start_bso=# --end_bso=#`) to `migrate_node.py`
* `gen_bso_users.py` takes same `bso_users_file` template as `migrate_node.py`
* f remove default value for BSO_Users.run bso_num
* f fix lock issue in gen_bso_users, trap for `` states in gen_fxa_users
* f make threading optional.
There's a locking issue that appears to be inside of the mysql.
Turning threading off for now (can be run in parallel)
* f fix tick, threading flag
* f rename confusing args in gen_bso and gen_fxa
gen_bso_users:
`--bso_users_file` => `--output_file`
gen_fxa_users:
`--fxa_file` => `--users_file`
`--fxa_users_file` => `--output_file`
* f more tick fixes
* f don't use threading on Report if threading isn't available.
* f make `--bso_users_file` / `--fxa_users_file` consistent
* `--bso_user_file` is now `--bso_users_file`
Issue #407
Yeah, this one's full of stuff.
* `gen_fxa_users.py` takes the tokendata file, and generates a file
containing a the converted uid => fxa_uid/fxa_kid values. See
`gen_fxa_users.py --help` for arguments.
* `gen_bso_users.py` takes the generated `fxa_users_{date}.lst` file
from `gen_fxa_users.py` pulls the users from the `--bso_num` and dumps
them to `bso_users_{bso_num}_{date}.lst`
* `{success,failure}_*` files are now only generated when needed. In addition, they are now suffixed with `.log`. Hopefully a bit easier to find and clean up.
* `migrate_node.py` now takes `--bso_users_file` which is either the
name of a file that will be used for all BSOs, or a template that will
be used to find the bso_users_file (e.g. if you specify
`--bso_users_file=users/bso_users_#_2020_04_14.lst` and
`--bso_start=1 --bso_end=3`, migrate user will pull from
`users/bso_users_1_2020_04_14.lst` for users in BSO#1,
`users/bso_users_2_2020_04_14.lst` for users in BSO#2, etc.
*NOTE*: by default scripts will date stamp various cached files, ideally, we should take reasonably "fresh" ones to avoid potentially missing users that are suddenly added to nodes. This is not a requirement, and all scripts allow for a custom file name.
since bsos INTERLEAVE's w/ DELETE CASCADE
- persist unique_key_filter across writes
- fix new bundling of of bso_values for inserting bsos
- add TODO for fixing user_collections' modified time
use `ms_delay` to pause between spanner transaction `--readchunk`s. This allows
some primative throttling for feeding spanner data.
Reminder: `readchunk` sets the max number of items to try to write per chunk
to spanner in any given transaction, default value 1000.
The `--user_percent` option will divvy up the users into blocks and
move the specified block. It takes an option formatted as
"block#:percentage". Block numbers are 1 based. For example,
--user_percent=2:33 will divide the total distinct users into
non-overlapping blocks of approximately 33%, and then move the second
block (e.g. the 33-65th users in the list). Extra users that may not be
evenly divided into percentage blocks will be appended to the last
block. (e.g. for `--user_percent=3:33`, users 66-99 would be copied
over, a total of 34 users)
Issue #407
* feat: more user_migration stuff
* create script to move users by node directly
* moved old scripts to `old` directory (for historic reasons, as well as
possible future use)
* cleaned up README
* try to solve the `parent row` error
an intermittent error may be responsible from one of two things:
1) a transaction failure resulted in a premature add of the unique key
to the UC filter.
2) an internal spanner update error resulting from trying to write the
bso before the user_collection row was written.
* Added "fix_collections.sql" script to update collections table to add
well known collections for future rectification.
* returned collection name lookup
* add "--user" arg to set bso and user id
* add `--dryrun` mode
