Content Pipeline Engineer
Black Canyon Consulting1 day ago
Responsibilities
- Design and implement reusable ingestion and processing pipelines for PDFs, scanned documents, images, forms, maps, and other unstructured content.
- Build OCR and multimodal document-processing workflows and extract text, tables, metadata, identifiers, and domain-specific attributes.
- Extract spatial information from plats and survey drawings, including legal descriptions, labels, bearings, and distances.
- Develop document parsing, classification, normalization, chunking, metadata-enrichment, provenance, and source-linkage processes.
- Build document and vector indexes, embeddings, semantic-search and retrieval pipelines, and reusable APIs for downstream applications and AI services.
- Implement agentic workflows for classification, extraction strategy selection, validation, exception handling, and quality assurance.
- Establish confidence thresholds and human-review processes for uncertain or ambiguous extractions.
- Collaborate with GIS engineers and Data Scientists on geospatial integration, evaluation datasets, and quality measurement.
- Build automated testing, monitoring, logging, observability, error handling, and quality controls for document-processing pipelines.
- Optimize pipelines for accuracy, throughput, repeatability, and cost.
Requirements
- Bachelor's degree in Computer Science, Software Engineering, Data Science, Information Systems, or a related technical field.
- 5+ years of software engineering, data engineering, search, document-processing, or related experience.
- Strong Python development experience and experience building production-grade data or content-processing pipelines.
- Hands-on experience processing PDFs, images, scanned documents, and semi-structured content with OCR and document-extraction technologies.
- Experience with document stores, search engines, or vector databases, including embeddings and semantic retrieval.
- Experience with document parsing, classification, chunking, metadata enrichment, and source-provenance preservation.
- Experience with LLM-enabled or agentic document-processing workflows for extraction, validation, routing, or exception handling.
- Experience building robust pipelines with automated testing, logging, monitoring, error handling, and quality controls.
- Experience evaluating OCR and extraction quality using measurable criteria and confidence thresholds.
- Strong problem-solving skills for complex and inconsistent document formats.
- Preferred experience with OpenSearch; AWS document-processing, AI, or machine-learning services; LangChain, LangGraph, or similar orchestration frameworks; multimodal or vision-language models; LLM-based document extraction; RAG or enterprise semantic search; maps, engineering drawings, plats, survey documents, or legal descriptions; geospatial extraction; U.S. Federal Government work; and permitting, environmental, energy, land-management, or regulatory documents.
Benefits
- Remote work arrangement.
- Medical, dental, and vision coverage.
- 401(k) plan with employer contribution.
- Paid holidays and vacation.
- Tuition reimbursement.
Categories
AI ApplicationsData Engineering
About Black Canyon Consulting
Black Canyon Consulting is a privately held, woman-owned IT and program management services firm that delivers contract support to U.S. federal research and health agencies. The company provides systems engineering, custom software, bioinformatics, and project management through government contracts and IDIQ vehicles; it was founded in 2017 and is headquartered in Fairfax, Virginia. Recent roles support the National Center for Biotechnology Information at the NIH/NLM.