3.1
3.1 Data Acquisition, Processing & Governance
Batch/stream ingestion, Lakehouse architecture (Iceberg/Delta), dbt modeling, metadata lineage, data contracts, and PII privacy governance.
Lakehouse Batch/Stream Ingestion Pipelines
High-throughput streaming/batch processing via Spark/Flink, Delta Lake/Iceberg/Hudi lakehouse formats, dbt SQL transformations, schema evolution, and multimodal OCR/PDF ingestion.
🏢 Companies
Databricks、Snowflake、Confluent、Fivetran、dbt Labs
🛠️ Tech Stack
Apache SparkApache FlinkApache IcebergDelta Lakedbt LabsSchema EvolutionMultimodal ParsingLakehouse
💼 Roles & Salary
Data Engineer、Analytics Engineer、Data Platform Engineer
💰 $175K - $360K / year (Data Engineering) | ¥450K - ¥1.1M / year
📚 Prerequisites: Advanced SQL & Query Optimization • Spark / Flink Distributed Computing • Lakehouse Formats (Iceberg/Delta) • Airflow / Dagster Orchestration
Data Governance, Lineage Tracking & Privacy Compliance
Enterprise data catalogs, column-level lineage tracking, Data Contracts, automated PII redaction/masking, consent compliance, and dataset version snapshotting.
🏢 Companies
Databricks、Collibra、Monte Carlo、BigID、Snowflake
🛠️ Tech Stack
Data CatalogLineageData ContractsPII RedactionData PrivacyGreat ExpectationsConsent Governance
💼 Roles & Salary
Data Governance Engineer、Data Platform Engineer、Data Quality Engineer
💰 $165K - $340K / year (Governance & Privacy) | ¥400K - ¥950K / year
📚 Prerequisites: Data Privacy Regs (GDPR/CCPA) • OpenLineage & Metadata Protocols • Data Quality SLA Monitoring • Data Masking & Security Policies