Adelaide SA | Hybrid (Edinburgh & CBD)
NV1 Security Clearance a requirement for job site.
Permanent | Defence Program
Emanate Technology is partnering with a leading Defence organisation to recruit a Senior Data Engineer to support a critical program within the Department of Defence (DSTG environment).
This role will take a successful datalake proof of concept and evolve it into a production-ready, scalable solution, supporting advanced capability across radar and FPGA data systems.
The Opportunity: You will join a collaborative team of 5 engineers, working closely with a client-side team of 4, to deliver a next-generation data platform in a highly secure and technical environment.
This is a hands-on role focused on data orchestration, pipeline engineering, and large-scale data processing, with long-term opportunity to grow as a Data SME within the organisation.
Key Responsibilities
- Design and build scalable data pipelines and orchestration frameworks
- Transition a data lake POC into a production-grade platform
- Work with complex radar and FPGA data sets
- Implement partitioning strategies and data workflows
- Develop and maintain pipeline-as-code environments
- Manage data dependencies, scheduling, and event-driven workflows
- Optimise performance, reliability, and maintainability
- Collaborate closely with engineering and Defence stakeholders
Core Technical Skills
(Must Have) Data Orchestration & data lake Pipeline expereince.
- Experience with Dagster(ideal) , NiFi or similar frameworks
- Strong understanding of pipeline-as-code environments
- Experience with backfills, scheduling, and event-driven workflows
- Ability to manage complex pipeline dependencies at scale
- Essential Python development for production-grade pipelines
- Experience with PySpark(ideal) , Pandas or Polars
- Understanding of batch, streaming and incremental processing
- Experience with data validation and schema management
- Experience with distributed query engines (Trino, BigQuery)
- Knowledge of data lake technologies (Delta Lake)
- Experience with object storage (S3, MinIO, Ceph)
- Understanding of data lineage, metadata and governance
- Experience with ML / LLM integration into data workflows
- Exposure to Docker, Kubernetes, CI/CD pipelines
- Infrastructure automation (Ansible)
- Full-stack exposure
Environment & Benefits
- Work on a cutting-edge Defence data capability
- Hybrid role across Edinburgh (DSTG) and Adelaide CBD
- Permanent role with long-term program stability
- Opportunity to grow into a Data SME
- NV1 Clearance is mandatory
- NV2 highly regarded
We are an inclusive employer committed to fostering a diverse and accessible workplace. We encourage applications from Aboriginal and Torres Strait Islander peoples, people with disabilities, LGBTQIA+ individuals, people of all ages, and those from culturally and linguistically diverse backgrounds.
