Data Engineer
Fastmarkets3 months ago
Sofia, BulgariaMid Level
Responsibilities
- Design, develop, and maintain scalable data pipelines from a wide variety of sources.
- Use dbt and Jinja templating to transform data across cleansed, conformed, and presentation layers.
- Build custom Python connectors and use connector-based ingestion tools.
- Curate enterprise data models for reporting, analytics, and data science use cases.
- Implement automated data quality checks, validations, exception handling, notifications, and alerts.
- Use Dagster as the primary orchestration and observability platform for data pipelines.
- Manage CI/CD workflows with GitHub, GitHub Actions, and Kubernetes.
- Implement zero-copy cloning and containerized on-demand development environments.
- Implement and review RBAC policies in Snowflake and related platforms.
- Promote DataOps, code-first engineering practices, and improvements to data processes and systems.
- Use Anthropic Claude and GitHub Copilot for documentation, code generation, code review, and delivery acceleration.
Requirements
- Advanced SQL expertise, including CTEs, stored procedures, and query tuning across multiple SQL dialects.
- Mandatory hands-on Snowflake experience as a cloud data warehouse, including Snowpark for Python transformations.
- Experience with dbt, Jinja templating, and ELT/ETL ingestion tools such as Fivetran or Airbyte.
- Experience building Python data pipelines and using Pandas and Great Expectations for data quality.
- Experience with Python-based orchestration platforms such as Dagster, Apache Airflow, or Prefect.
- Experience with API integration, JSON parsing and transformation, Git version control, and CI/CD workflows.
- Experience with GitHub Actions or Azure DevOps and cloud services in Azure, AWS, or GCP; Azure is preferred.
- Understanding of data modeling approaches including 3NF, star schema, Kimball/Inmon, and OBT; Data Vault 2.0 is desirable.
- Strong analytical, problem-solving, communication, collaboration, flexibility, and attention-to-detail skills.
- Desirable experience with Kubernetes, container orchestration, Terraform or Pulumi, Snowflake Cortex AI, Snowflake native apps, Azure Service Bus, Kafka, Salesforce, Marketo, and AI-assisted development workflows.
- A related technology degree, or at least 3 years of experience in a similar role with professional certifications; SnowPro Certification is a nice-to-have.
Benefits
- Fastmarkets Rewards provides core benefits and a flexible allowance that can be tailored to lifestyle, wellbeing, and financial priorities.
- Benefits vary by country and role and support the company’s global workforce.
- Fastmarkets is committed to an inclusive workplace and can provide interview accommodations upon request.
Tech Stack
AirbyteApache AirflowApache KafkaAWSAzuredbtGitGitHub ActionsGoogle Cloud PlatformKubernetesPandasPythonSnowflakeSQLTerraform
Categories
Data Engineering
About Fastmarkets
Fastmarkets is a cross-commodity price reporting agency and market intelligence provider for metals, mining, forest products, agriculture, and energy transition markets. It sells benchmark prices, forecasts, analytics, and news via subscriptions and data feeds, and runs industry events. Founded in 1882 and headquartered in London, it is privately held by Epiris and incorporates legacy brands such as Metal Bulletin, RISI, and American Metal Market.