The new Update() performs the same tasks as the old Resize()
in addition to formatting previously-unformatted partitions. This
allows children disk-layouts to repartition the base layout in
addition to resizing.
I started to move board files under a boards/ directory similar to how
the SDK is under sdk/ but didn't do so everywhere. This should finish
the job so everything is consistent now.
Note: This prefix is only used in developer and buildbot uploads. When
final releases are copied to $channel.release.core-os.net it doesn't use
the prefix since a) I already published urls without the prefix and b)
no sdk files are ever posted to the public release locations.
The new --developer_data option can be used to specify a path to a cloud
config to bundle into the image. If none is provided but a shared user
password (for core) is set then generate a config to set that password.
This lets us use the same mechanism for setting the default password for
both disk and PXE images.
Only the key is needed, and currently the vagrant OEM is completely
broken outside of vagrant. This gets vmware_insecure images back into
the state that they were before cloud config came along. :)
This adds two new optional build steps. The first user of these is the
vagrant images but many of the targets can be simplified now.
- fs_hook: Anything that needs to happen before unmounting the image.
This happens after the OEM is installed but before disk images are
made. It can be used to copy any data out of the image.
- bundle_format: Many VM types ship as some sort of archive format
rather than plain disk images as this script originally assumed.
Adding this final step lets us stop using the conf step awkwardly.
Vagrant now ships with a Vagrantfile and related code included in the
OEM package. This lets us version our vagrant-side code along with the
images themselves as well making the coreos-vagrant repo optional again.
The coreos-vagrant code will still be useful for handling the fancier
cluster configuration stuff but no longer has to carry the plugin code.
Installing to a temporary directory and then copying over the final
contents of /usr/share/oem allows more complicated OEM packages such as
python to be configured with --prefix=/usr/share/oem while previously
the atypical use of ROOT=/usr/share/oem would have complicated things.
This can be used by update_engine as a quick test to determine if it is
running on a system that it can handle. This avoids needing something
like the 'coreos.diskless' kernel command line flag.
If QEMU is given a uuid systemd will detect that and in turn use it for
the machine-id. This made the bug causing the machine-id to be always
re-generated on boot harder to notice since it didn't happen on QEMU.
Taking a bit of a new approach to booting PXE images here for both
amd64-generic and amd64-usr. Instead of requiring the user to specify
squashfs and tmpfs on the kernel command line we can simply provide
defaults in the initrd's fstab.
cgpt now supports generating hybrid MBRs and the classic style mbr.bin
from any version of SYSLINUX should work the same with the hybrid MBR.
The other code, gptmbr.bin, changes after SYSLINUX 3. Switching lets me
play with different versions of SYSLINUX without breaking everything.
With this change all images feature a hybrid MBR so the special case for
some VM platforms has been removed.
While attempting to fix the easy to mix up DIGESTS names in feb59db9f I
stumbled across yet another way that the DIGESTS names were a bit
unpredictable: previously a .bz2 got stuck into the file name when
upload_images automatically compressed some file types. The new code
missed this and never added the .bz2. Correct this now, both for
image_to_vm which has a pile of glue to keep the legacy names in place
for now and build_image which I never intended to change.
Switch from naming DIGESTS based in disk image name to a common prefix.
old: coreos_production_qemu_image.img.DIGESTS ->
new: coreos_production_qemu.DIGESTS
The old behavior wasn't very consistent since plain disk images aren't
used by all types and the code implementing that was easy to brake,
namely by mistake coreos_production_pxe_image.cpio.gz.DIGESTS became
coreos_production_pxe.vmlinuz.DIGESTS a couple releases ago.
The old names will continue to exist as well for the time being to avoid
breaking existing install/download scripts and the original pxe DIGESTS
name is back.
For multi-file uploads we should explicitly declare what the name of the
.DIGESTS file should be instead of using the first file name. Relying on
the ordering was subtle and easy to break.
Move from optparse to argparse. Move layout file and layout type to
global options with reasonable default values so every command doesn't
need to them. Adjust calling scripts to match.
For now layout type is being passed via an environment variable
DISK_LAYOUT_TYPE but this is a temporary situation.
Vagrant users are accustomed to much larger disk sizes so lets give it
to them. I'm leaving the others as-is since it is easier to grow than
shrink disks if anyone has a particular size they need.
Use the smaller base format for 'raw' disk images since these will
usually be dd'd to a block device to create AMIs and what not. For
images using qcow2 and vmdk stick with the larger vm size.
This change removes /usr/sbin/write_gpt.sh from images which we have no
use for. This allows us to drop the indirection of writing partition
tables by first writing out a script to call. Now cgpt.py can call cgpt
directly to initialize the partition layout. This opens the way for
further improvements to how disk images are created.
This one is more automagical and sets up ssh keys from ssh-agent and the
user's home directory by default. Also adds an option for setting the
ssh port so it can be something other than 2222. Script should be
sufficiently portable, tested in bash, dash, and ash.
The recommended command using the config file was triggering a massive
memory leak in qemu because it was adding both the default virtual
hardware nic as well as the virtio nic. This could be worked around by
adding something like -net none or moving all the -net commands from the
file to the command line but eh. Clearly qemu config files are used and
tested by nobody else so lets just use a trusty script instead.
Vagrant will need the virtualbox ovf plus its own Vagrantfile config.
After this we will need an optional "package" step to this script to
take these files and bundle them into a .box tarball. This could also be
used to switch from .ovf+vmdk for plain virtualbox images to a bundled
.ova archive which combines the two.
Trying to include version info by adding the directory name to VM image
names didn't work and a better solution is for build_image to write out
a version.txt file. This should also fix an issue where uploading from
image_to_vm.sh didn't always go to the same location as the images
uploaded from build_image did.
Enable sparse files for all dd and cp commands and replace some dd
commands that are really better off being truncate commands.
While in the neighborhood there were a number of useless sudo commands
for things that just happen to be in sbin. Call them directly instead.
Last minute bug slipped in because of a line I commented out since the
current coreos kernel doesn't support virtio block devices (that change
coming soon). Qemu doesn't tolerate any spaces before # in comments.
The old script was heading towards spaghetti code realm. This breaks up
all the image variations such as hybrid MBR, OEM packages, etc into
configuration options and small functions that actually do the work.
All this is in the new vm_image_util.sh library but the command line
parsing and overall procedure remains in image_to_vm.sh
As part of this we gain support for putting some qemu options in a
config file as well as Xen virtual machines using pygrub and pvgrub.
Lots of generally unused options have been removed to simplify things
and keep output file names consistent.
