Experimentation suggests Alibaba Cloud API calls are extremely
unreliable, with a failure rate around 1%. It is therefore necessary
to allow for retrying basically every API call.
Some API calls (e.g. DescribeImages or ModifyImageAttribute) are
naturally idempotent and so safe to retry. Some non-idempotent API
calls (e.g. CopyImage) support explicit idempotence tokens. The
remaining API calls may simply fail on a retry, if the original
request happened to succeed but failed to return a response.
We could write convoluted retry logic around the non-idempotent calls,
but this would substantially increase the complexity of the already
unnecessarily complex code. For now, we assume that retrying
non-idempotent requests is probably more likely to fix transient
failures than to cause additional problems.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The CopyImage API call does work, but is unacceptably slow due to rate
limiting. Importing a full set of images to all regions can take
several hours (and is likely to fail at some point due to transient
errors in making API calls).
Resort to a mixture of strategies to get images imported to all
regions:
- For regions with working OSS that are not blocked by Chinese state
censorship laws, upload the image files to an OSS bucket and then
import the images.
- For regions with working OSS that are blocked by Chinese state
censorship laws but that have working FC, use a temporary FC
function to copy the image files from the uncensored OSS buckets
and then import the images. Attempt downloads from a variety of
uncensored buckets, since cross-region OSS traffic tends to
experience a failure rate of around 10% of requests.
- For regions that have working OSS but are blocked by Chinese state
censorship laws and do not have working FC, or for regions that
don't even have working OSS, resort to using CopyImage to copy the
previously imported images from another region. Spread the
imports across as many source regions as possible to minimise the
effect of the CopyImage rate limiting.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Spinning up ECS instances is supported in all ECS regions (unlike
Function Compute), but turns out to be unacceptably unreliable since
Alibaba Cloud has a very irritating tendency to fail to launch ECS
instances for a variety of spurious and unpredictable reasons.
Rewrite the censorship bypass mechanism to use the (extremely slow)
CopyImage API call to copy an imported image from an uncensored region
to a censored region.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Function Compute is unsupported in several Alibaba Cloud regions.
Rewrite the censorship bypass mechanism to access OSS buckets using a
temporary ECS instance instead of a temporary Function Compute
function.
Importing images now requires that the account has been prepared using
the "ali-setup" script, which creates the necessary role, VPCs, and
vSwitches to allow ECS instances to be launched in each region.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Importing images into Alibaba Cloud currently relies upon using a
temporary Function Compute function to work around Chinese state
censorship laws that prevent direct access to OSS bucket contents in
mainland China regions.
Unfortunately, Alibaba Cloud regions are extremely asymmetric in terms
of feature support. (For example, some regions do not even support
IPv6 networking.) Several mainland China regions do not support
Function Compute, and so this workaround is not available for those
regions.
A possible alternative censorship workaround is to create temporary
ECS virtual machine instances instead of temporary Function Compute
functions. This requires the existence of a role that can be used by
ECS instances to access OSS. We cannot use the AliyunFcDefaultRole
that is currently used by Function Compute, since this role cannot be
assumed by ECS instances.
Creating roles is a privileged operation, and it would be sensible to
assume that the image importer (which may be running as part of a
GitHub Actions workflow) may not have permission to itself create a
suitable temporary role. The censorship bypass role must therefore be
set up once in advance by a suitably privileged user.
Add the ability to create a suitable censorship bypass role to the
Alibaba Cloud setup utility.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Creating ad hoc instances in Alibaba Cloud is extremely cumbersome and
tedious due to the need to specify an explicit vSwitch and security
group, with no defaults being available.
Add a utility that will create a VPC within each region, a vSwitch
within each zone within each region, and a security group within each
region.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Update the descriptive text for the disk log console tools to remove
references to INT13, since these now work for both BIOS and UEFI disk
log consoles.
Leave the script names as {aws,gce,ali}-int13con, to avoid breaking
any existing tooling that might use these names.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Allow the UEFI CPU architecture to be detected for the partitioned
disk images generated by genfsimg as of commit 2c84b68 ("[build] Use a
partition table in generated USB disk images").
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add support for a disk log partition console, using the same on-disk
structures as for the BIOS INT13 console.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Split out the generic portions of the INT13 disk log console support
to a separate file that can be shared between BIOS and UEFI platforms.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The name "int13" is intrinsically specific to a BIOS environment.
Generalise the build configuration option CONSOLE_INT13 to
CONSOLE_DISKLOG, in preparation for adding EFI disk log console
support.
Existing configurations using CONSOLE_INT13 will continue to work.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The workaround used for UEFI in commit 926816c ("[efi] Pad transmit
buffer length to work around vendor driver bugs") is also applicable
to the BIOS UNDI driver.
Apply the same workaround of padding the transmit I/O buffers to the
minimum Ethernet frame length before passing them to the underlying
UNDI driver's transmit function.
Reported-by: Alexander Patrakov <patrakov@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Commit cb95b5b ("[efi] Veto the Dhcp6Dxe driver on all platforms")
vetoed the Dhcp6Dxe driver to work around the bug described at
https://github.com/tianocore/edk2/issues/10506 that results in
EfiDhcp6Stop() getting stuck in a tight loop waiting for an event that
will never occur.
Since we now call UnloadImage() at TPL_APPLICATION, we no longer
trigger the bug in Dhcp6Dxe, and so the veto may be removed.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
As of commit c3376f8 ("[efi] Drop to external TPL for calls to
ConnectController()"), the veto mechanism will drop to TPL_APPLICATION
for calls to DisconnectController().
Match this behaviour for calls to UnloadImage(), since that is likely
to result in calls to DisconnectController(). For example, any EDK2
driver using NetLibDefaultUnload() as its unload handler will call
DisconnectController() to disconnect itself from all handles.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
On Ubuntu/Debian, syslinux-common installs mbr.bin to
/usr/lib/syslinux/mbr/mbr.bin. This path is not currently searched by
find_syslinux_file(), causing USB disk image generation to fail with
"could not find mbr.bin".
Add /usr/lib/syslinux/mbr, /usr/share/syslinux/mbr, and
/usr/local/share/syslinux/mbr to the search paths.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For UEFI, the USB disk image is constructed from the built EFI binary
(e.g. bin-x86_64-efi/ipxe.efi) by genfsimg, which does not itself have
any way to access the build configuration. We therefore need a way to
annotate the binary such that genfsimg can determine whether or not to
include a log partition within the USB disk image.
The "OEM ID" and "OEM information" fields within the PE header can be
used for this, since they are easily accessed and serve no other
purpose. We define bit 0 of "OEM information" as a flag indicating
that a log partition should be included. If this bit is set, genfsimg
will create a log partition with a layout matching that of the BIOS
build (i.e. using partition 3 and at an offset of 16kB from the start
of the disk).
The PE header is constructed by elf2efi.c, which takes as an input the
linked ELF form of the binary. We use an ELF .note section to allow
any linked-in object to communicate the log partition request through
to elf2efi.c, which then populates the OEM information field
accordingly.
We choose to use the same field locations within the BIOS bzImage
header, since this allows genfsimg to use the same logic for both BIOS
and UEFI binaries. In a BIOS build, there is no external processing
equivalent to elf2efi.c, and so we construct the field value directly
using absolute symbols and explicit relocation records.
(Note that the bzImage header is relevant only when using genfsimg to
construct a combined BIOS/UEFI image. In the common case of building
a BIOS-only image such as bin/ipxe.usb, the partition table is
manually constructed by usbdisk.S and genfsimg is not involved.)
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The syslinux function check_fat_bootsect() performs some sanity checks
to ensure that the filesystem type string (e.g. "FAT12") is correct
for the total number of clusters in the FAT. There is unfortunately a
bug in its calculation of the number of sectors occupied by the root
directory, which causes it to underestimate the number of sectors by a
factor of 32.
When the total number of clusters is close to the FAT12 limit of 4096,
this bug can cause syslinux to erroneously report that the filesystem
has "more than 4084 clusters but claims FAT12".
Work around this bug by selecting an explicit cluster size in order to
avoid potentially problematic cluster counts. We default to using 4kB
clusters, doubling to 8kB if using 4kB would result in a total cluster
count near 4096 (the FAT12 limit) or near 65536 (the FAT16 limit).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The calculations around the FAT filesystem layout currently use a
mixture of kilobytes and sector counts. Switch to using sector counts
throughout the calculation, to make the code easier to read.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The USB disk image constructed by util/genfsimg is currently a raw FAT
filesystem, with no containing partition. This makes it incompatible
with the use of CONSOLE_INT13, since there is no way to add a
dedicated log partition without a partition table.
Add a partition table when building a non-ISO image, using the mbr.bin
provided by syslinux (since we are already using syslinux to invoke
the ipxe.lkrn within the FAT filesystem).
The BIOS .usb targets are built using a manually constructed partition
table with C/H/S geometry x/64/32. Match this geometry to minimise
the differences between genfsimg and non-genfsimg USB disk images.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
We use mformat to ensure that the FAT filesystem starts as empty.
However, formatting the filesystem can still leave old data blocks
present (though unreferenced) within the disk image.
Truncate the image to a zero length before extending, to ensure that
no stale content is retained.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Following the examples of aws-int13con and gce-int13con, add a utility
that can be used to read the INT13 console log from a used iPXE boot
disk in Alibaba Cloud Elastic Compute Service (ECS).
We cannot reliably access the used iPXE boot disk (or a snapshot
created from it) since OSS buckets in mainland China cannot be
accessed due to Chinese laws. We therefore create a snapshot and
attach this snapshot as a data disk to a temporary Linux instance, as
we do in Google Compute Engine.
Unlike in Google Compute Engine, we cannot reliably capture serial
port output from the temporary Linux instance. Issuing the relevant
GetInstanceConsoleOutput API call will cause the output to be captured
once and (unpredictably) cached. Without knowing in advance precisely
when the output is complete, we cannot use this approach to capture
the relevant part of the output.
We therefore use an Alibaba Cloud Linux image that includes the Cloud
Assistant Agent. This allows us to use the RunCommand API call to run
a command on the instance and capture the output, all done via the
control plane so that we are not dependent on having direct network
access to the temporary instance.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Include additional condition to invoke short command logic when
firmware indicates it is required. Replace 100ms delay with wmb() to
ensure DMA buffer is ready when short command is invoked.
Signed-off-by: Joseph Wong <joseph.wong@broadcom.com>
Following the examples of aws-import and gce-import, add a utility
that can be used to upload an iPXE disk image to Alibaba Cloud Elastic
Compute Service (ECS) as a bootable image.
The iPXE disk image is first uploaded to a temporary Object Storage
Service (OSS) bucket and then imported as an ECS image. The temporary
bucket is deleted after use.
As with Google Compute Engine, an appropriate image family name is
identified automatically: "ipxe" for BIOS images, "ipxe-uefi-x86-64"
for x86_64 UEFI images, and "ipxe-uefi-arm64" for AArch64 UEFI images.
This allows the latest image within each family to be launched within
needing to know the precise image name.
Copies of the images are uploaded to all selected regions. One major
complication is that OSS buckets in mainland China can be created but
cannot be accessed due to Chinese laws, which require an ICP filing
for any bucket hosted in mainland China. We work around this
restriction by first uploading the image to a region outside mainland
China and then using a temporary Function Compute function running in
each region to copy the images to the OSS bucket via the internal OSS
endpoints, which are not subject to the same restrictions.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The undionly.kpxe binary does not need the full PCI bus support.
However, the overwhelming majority of UNDI devices are PCI-based and
we already end up dragging in PCI configuration space support in order
to be able to test for devices with broken interrupts.
Dragging in the PCI configuration allows the PCI settings mechanism to
also be present, which is often useful for end users. The total cost
is around 200 bytes in the final binary, which is acceptable for a
generally very useful feature.
Users wanting to minimise the binary size can choose to explicitly
disable PCI_SETTINGS via config/settings.h.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Since October 2025, the Microsoft UEFI Signing Requirements have
included a clause stating that "submissions must contain a valid
signed SPDX SBOM in a custom '.sbom' PE section". A list of required
fields is provided, and a link is given to "the Microsoft SBOM tool to
aid SBOM generation". So far, so promising.
The Microsoft SBOM tool has no support for handling a .sbom PE
section. There is no published document that specifies what is
supposed to appear within this PE section. An educated guess is that
it should probably contain the raw JSON data in the same format that
the Microsoft SBOM tool produces.
The list of required fields does not map to identifiable fields within
the JSON. In particular:
- "file name / software"
This might be the top-level "name" field. It's hard to tell. The
SPDX SBOM specification is not particularly informative either: the
only definition it appears to give for "name" is "This field
identifies the name of an Element as designated by the creator",
which is a spectacularly useless definition.
- "software version / component generation (shim)"
This may refer to the "packages[].versionInfo" field. There is no
obvious relevance for the words "component", "generation", or
"shim". The proximity of "generation" and "shim" suggests that this
might be related in some way to the SBAT security generation, which
is absolutely not the same thing as the software version.
- "vendor / company name (this must exactly match the verified company
name in the submitter's EV certificate on the Microsoft HDC partner
center account)"
This is clearly written as though it has some significance for the
UEFI signing submission process. Unfortunately there is no obvious
map to any defined SBOM field. An educated guess is that this might
be referring to "packages[].supplier", since experiments show that
the Microsoft SBOM tool will fail validation unless this field is
present.
- "product-name"
This might also be the top-level "name" field. There is no
indication given as to how this might differ from "file name /
software".
- "OEM Name" and "OEM ID"
These seem to be terms made up on the spur of the moment. The
three-letter sequence "OEM" does not appear anywhere within the
codebase of the Microsoft SBOM tool.
In the absence of any meaningful specification, we choose not to
engage in good faith with this requirement. Instead, we construct a
best guess at the contents of a .sbom section that has some chance of
being accepted by the UEFI signing submission process. We assume that
anything that passes "sbom-tool validate" will probably be accepted,
with the only actual check being that the supplier name must match the
registered EV code signing certificate.
To anyone who actually cares about the arguably valuable benefits of
having a software bill of materials: please stop creating junk
requirements. If you want people to actually make the effort to
produce useful SBOM data, then make it clear what data you want.
Provide unambiguous specifications. Provide example files. Provide
tools that actually do the job they are claimed to do. Don't just
throw out another piece of "MUST HAS THING BECAUSE IS MORE SECURITY"
garbage and call it a day.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add a workflow to build and import the official iPXE images for Google
Cloud. As with the AWS import, treat this as a workflow that must be
triggered manually.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The storage client is currently constructed with the project inferred
from the environment, rather than using the project specified via the
command line arguments.
Fix by passing the project name to the storage client constructor.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Commit 19dffdc ("[efi] Allow for creating devices with no EFI parent
device") relaxed the restriction on attempting to create SNP devices
when no EFI parent device is available, with the result that the test
network devices created when running the IPv4 tests are now registered
as SNP devices.
Since the dummy EFI parent device path is fixed and the test network
device MAC addresses are empty, the SNP devices end up with identical
constructed device paths and registration of the second and subsequent
devices will fail since device paths must be unique.
Fix by assigning MAC addresses to the test network devices.
Reported-by: Miao Wang <shankerwangmiao@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add a workflow to build and import the official iPXE images for AWS
EC2. Treat this as a workflow that must be triggered manually, since
importing is prone to failure for reasons unrelated to the state of
the codebase (e.g. the creation of new regions, or an explosion at a
data centre) and so should not result in CI failures being reported
against specific commits.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Most TPL manipulation is handled by efi_raise_tpl()/efi_restore_tpl()
pairs. The exceptions are the places where we need to temporarily
drop to a lower TPL in order to allow a timer interrupt to occur.
These currently assume that they are called only from code that is
already running at the internal TPL (generally TPL_CALLBACK). This
assumption is not always correct. In particular, the call from
_efi_start() to efi_driver_reconnect_all() takes place after the SNP
devices have been released and so will be running at the external TPL.
Create an efi_drop_tpl()/efi_undrop_tpl() pair to abstract away the
temporary lowering of the TPL, and ensure that the TPL is always
raised back to its original level rather than being unconditionally
raised to the internal TPL.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When we fall back to using our own loaded image's device handle
(instead of the most recently opened SNP device handle), we may find
that the device handle is no longer valid since we have disconnected
the driver that originally provided it.
Check for existence of the device path protocol on the identified
parent handle, and choose not to attempt to set a parent-child
relationship if the parent handle appears to no longer be valid.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When we fall back to using our own loaded image's device handle
(instead of the most recently opened SNP device handle), we may find
that some protocols (e.g. EFI_SIMPLE_FILE_SYSTEM_PROTOCOL) are already
present on this handle.
Fix by creating a child device handle with an added Uri() device path
component, and installing EFI_SIMPLE_FILE_SYSTEM_PROTOCOL and others
onto this handle instead of onto the identified parent device handle.
This also provides a way for us to communicate the image URI to a
chainloaded iPXE, so that that iPXE can set its current working URI
appropriately.
A side effect of this change is that the EFI_SIMPLE_NETWORK_PROTOCOL
will be found on the parent of the loaded image's device handle,
rather than directly on the loaded image's device handle. This will
not cause problems for a chainloaded iPXE, since that will already use
LocateDevicePath() to find EFI_SIMPLE_NETWORK_PROTOCOL (or the
EFI_MANAGED_NETWORK_SERVICE_BINDING_PROTOCOL created by MnpDxe) and so
will already find the instance on the parent device handle. If other
UEFI executables are found to exist that do require the protocols to
be installed directly on the loaded image's device handle, then we
could potentially install copies of these protocol instances on the
device handle.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
We need a device handle from which to nominally load an EFI image. We
currently rely on using the most recently opened network device's SNP
device handle, in the same way that we use the most recently opened
network device when loading a BIOS PXE NBP image. If there is no most
recently opened network device, then we cannot execute an EFI image.
We use three aspects of the SNP device handle: the handle itself
(giving us something on which to install protocols), the associated
device path (giving us a base path from which to construct the new
image's file path), and the associated network device (giving us an
interface for the PXE base code protocol installation).
Make the network device optional by simply choosing not to install the
PXE base code protocols when no network device is defined. This
allows us to fall back to using our own loaded image's device handle
and device path for the other two purposes.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When chainloaded from another iPXE, there will be both a virtual
filesystem and a managed network protocol available through which we
could attempt to load autoexec.ipxe.
Try both of these, with the virtual filesystem attempted first so that
an autoexec.ipxe that was explicitly downloaded by the chainloading
iPXE will have the highest priority.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
We currently expect to find either a cached DHCP packet (from a UEFI
PXE boot) or a URI device path (from a UEFI HTTP boot), but not both
simultaneously. When both are present, the cached DHCP packet will
currently override any current working URI that was previously derived
from a URI device path.
Treat the URI device path as being more informative than the cached
DHCP packet by swapping the order in which these are processed.
Leave the boot option device path as being a lower priority than a
cached DHCP packet, since the boot option device path may well refer
to an earlier boot stage.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
There is no viable way to link to the list of sponsorship recipients.
Add the organization itself as a sponsorship recipient, solely in
order to enable the use of https://github.com/sponsors/ipxe as a
central link for sponsorship information.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For a UEFI HTTP boot, we set the current working URI based on the
loaded image device path. The autoexec.ipxe script will be fetched
from the same directory as the iPXE binary itself.
For a BIOS or UEFI PXE boot, we do not explicitly set a current
working URI, but rely on the fact that registering the cached DHCP
settings block will cause the TFTP code to set the current working URI
to "tftp://${next-server}/". The autoexec.ipxe script will therefore
be fetched from the default directory (which is most probably the root
directory) of the TFTP server.
When using a UEFI shim, the shim will always fetch iPXE from the same
directory as the shim itself. This leads to a somewhat unintuitive
requirement for a UEFI PXE boot: the shim and iPXE must be placed in
the same directory, but the corresponding autoexec.ipxe script must be
placed in the root directory.
As with the loaded image device path for a UEFI HTTP boot, the
existence of a cached DHCP packet gives us a way to construct the URI
of our own binary. We can therefore choose to use this to set the
current working URI, so that the autoexec.ipxe script may be placed in
the same directory as the iPXE binary itself. This is the least
surprising location, and avoids the need for lengthy explanations in
documentation.
Choose to set the current working URI at the point that the cached
DHCP packet is recorded, rather than the point at which it is applied
and registered as a settings block. This avoids some awkward corner
cases (such as failing to find a matching network device for the
DHCPACK), and naturally ensures that we retrieve the next-server
address and filename from the same DHCP packet. We rely on the order
in which cached DHCP packets are recorded to impose a priority
ordering: later packets (e.g. PxeBSACK) will override earlier ones.
To avoid breaking existing setups that do place the autoexec.ipxe
script in the root directory, we modify the fetching logic to first
attempt to retrieve autoexec.ipxe from the current working URI, then
from the root directory of that URI.
As with commit a69afd7 ("[tftp] Use TFTP server URI only if no other
working URI is set"), this is technically a breaking change in
behaviour, but the new behaviour is almost certainly less surprising
than the existing behaviour. Scripts that rely on the current working
URI being set to the root of the TFTP server can use absolute URIs
(i.e. add an initial slash): this is more explicit and will work on
iPXE builds both before and after this change.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Pass both parts of the generated release notes through pandoc, to
ensure some consistency in terms of link styles and line lengths.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add support for loading iPXE via a UEFI shim in ISO and USB images.
Since the iPXE shim's default loader filename is currently "ipxe.efi"
for all CPU architectures, at most one architecture within an image
may use a shim. (This limitation should be removed in the next signed
release of the iPXE shim.)
Signed-off-by: Michael Brown <mcb30@ipxe.org>
It is unintuitive to have to include an "ifopen" at the start of an
autoexec.ipxe script. Commit efe8126 ("[cachedhcp] Automatically open
network device matching cached DHCPACK") causes the chainloaded device
to be opened automatically, using the cached DHCPACK to identify the
chainloaded device.
In the case of a UEFI HTTP(S) boot, the firmware does not provide
access to the DHCPACK and we are forced to instead extract the very
limited amount of information encoded into the loaded image's device
path.
Mark the device matching the loaded image's device path to be opened
automatically, so that the chainloaded device will be opened in the
same way for both TFTP and HTTP(S) boots.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
