High Performance Computing (HPC), as a category of computing, is becoming strategically important to Australia’s future as workloads such as AI Large Language Model training, earth observation, weather and climate modelling, and other massively parallel processing workloads with very large datasets emerge across portfolios and within Commonwealth entities.
When determining if HPC facilities, including specialist cloud services, can and should support a government activity, various factors must be considered and balanced, as follows:
Identify needs and approach in advance of selecting HPC facilities
When determining suitability of HPC facilities entities should:
- consider whether their specific needs match the specialist abilities of HPC facilities, such as the need for massively parallel computation and fast processing of very large volume datasets
- develop and include written statements on the following balance of interests:
-
Value for money
The performance and reliability of computing platforms are continually improving. Commonwealth entities should explore the best value for money approach when considering the need for HPC resources. Lower cost computing resources may be sufficiently fit-for-purpose to meet organisational needs.
-
Australia’s sovereign capability
Certain uses of HPC are critical to Australia’s national interests and sovereign capability, such as:
- defence capabilities
- research capabilities
- biological and health-related capabilities
- climate and weather modelling
- certain types of industrial design
Some aspects of cybersecurity also require HPC facilities. An emerging aspect of HPC sovereign capability is the development and training of AI Large Language Models (LLM) and other AI models with Australian content, values and cultural contexts. The potential need for Australia to research and build its own, foundational LLM rather than relying on fine-tuning of existing, overseas-controlled models is a current discussion topic in the research community.
Over-reliance on foreign HPC capability and capacity is not in Australia’s national interest and Commonwealth entities need to consider the whole-of-government perspective and the opportunities to partner with Australia’s research community to meet their HPC needs.
Improved, secure access and sharing of HPC resources for both researchers and government is becoming an imperative. This will include access to Quantum computing facilities as these become available.
There is a general distinction between HPC resources required for the research & development of models, including AI LLM training, and the deployment of those models for inference uses in operational settings, which does not necessarily require HPC facilities or sovereign capacity.
-
Australia’s research competitiveness
Research is becoming increasingly reliant on computational capacity, and there is a growing correlation between researchers’ access to HPC resources and the global competitiveness of that research. For Australia’s research competitiveness going forward, Commonwealth entities should consider the positive effects of supporting Australian HPC research facilities through utilisation of those facilities where appropriate.
-
Australia’s trade interests
To support Australian industry and ensure Australia’s continuing participation in global supply chain and value chain activities associated with HPC, Commonwealth entities should support Australian owned and operated HPC facilities wherever possible.
Access HPC resources utilising shared facilities available to Commonwealth entities
Several HPC facilities and resources have been developed across government, including within non-corporate and corporate Commonwealth entities.
Entities should:
- consider accessing existing HPC resources available to the APS via:
- HPC facilities directly owned by non-corporate Commonwealth entities, including:
- the Bureau of Meteorology’s weather forecasting-oriented supercomputing facilities
- defence hosted HPC resources - for entities in the Defence portfolio, please contact architecture@dta.gov.au for details
- Commonwealth-funded HPC facilities in the research environment, including:
- the CSIRO’s Commonwealth owned and operated facilities and National Collaborative Research Infrastructure Strategy (NCRIS) funded research facilities
- the Australian National University’s (ANU’s) National Computing Infrastructure (NCI), including ‘Gadi’ and other supercomputing facilities
- several ‘tier 2’ supercomputing research facilities, owned and operated by universities that are not Commonwealth entities, but available through various access schemes.
- Whole-of-Government procurement arrangements with industry, including:
- DTA-administered ICT procurement marketplaces and whole-of-government contract arrangements, including both cloud-oriented and on-premises HPC products and services.
In considering existing HPC resources available to the Commonwealth, entities should:
- map their unique requirements against the attributes of these facilities
- investigate potential for utilisation of these facilities prior to considering other procurement
- where existing facilities are not suitable, clearly document why this is the case, as part of any proposals to utilise non-government-owned, or develop bespoke, HPC facilities
- if utilising cloud providers for HPC functionality, procure via BuyICT and the whole-of-government contracts.
Align to guidelines and standards
Entities must:
- comply with the Hosting Certification Framework to ensure cloud-oriented HPC services are hosted in certified tenancies/data centres
- apply the principles and requirements of the Protective Security Policy Framework throughout their organisation, including in the context of High Performance Computing.
Entities should:
- apply a risk-based approach to cyber security, where it comes to HPC facilities, to align with the Information Security Manual (ISM).
Adhere to reuse principles
The Australian Government Architecture provides information for entities on Reuse.
When utilising existing HPC facilities, entities should:
- document their use case, including specifics such as data types and volumes
- define success measures prior to utilisation
- record as a pattern the process of utilisation
- assess and document outcomes against these success measures
- make documentation available to other entities to enable future reuse.
When investing in new HPC facilities, entities should:
- ensure these are built to provide maximum value to the Australian Government, including potential provision of shared access or re-use for other Commonwealth entities
- clearly document planning and design artefacts, processes, specifications, risk assessments and investment decisions
- enable, as much as is appropriate within legislative and practical constraints, future opportunities to share or re-use facilities across the APS
- meet the requirements of the Digital and ICT Reuse Policy.