How to hire a data engineer

Hire for trustworthy data changes

A data engineer makes important data flows dependable from source to use. This example role owns subscription-event ingestion and the analytical tables finance and product use to understand paid subscriptions and recognized events. It must handle duplicates and late arrivals, explain a correction, and show which downstream metrics changed.

Before writing a role post, agree what event identity means, which timestamp controls each use, how corrections are represented, who approves a backfill, and who owns definitions such as active subscriber, gross bookings or recognized revenue. Monetary metrics need a named policy owner, currency basis, effective time and treatment of refunds, proration and reversals. Never let a pipeline convention become an undocumented financial definition.

Apache Beam describes event-time windows and watermarks as estimates, with explicit choices for early and late data. Its documentation also notes that the default windowing/trigger setup discards late data when allowed lateness is zero. That makes lateness policy a business and system decision, not an automatic property of streaming: Beam programming guide: triggers and late data.

↑ Back to sections

Separate engineering, analysis and metric authority

Data engineer, data analyst and data scientist work together, but their primary outcomes differ. A small team may combine tasks; state that openly and name the support, access and decision rights for each.

Set the boundary at each handoff: who defines customer/subscription state, who approves money rules, who can read raw identifiers, who owns downstream dashboards, and who approves a replay that changes published history. Engineering ownership includes making those boundaries visible and enforceable.

↑ Back to sections

Map requirements to evidence

Assess the data behavior the person will own. Tool familiarity can be learned when a reviewer and safe environment exist; replay safety, schema change judgment and access boundaries need evidence if the role carries them from day one.

A role centered on dashboards and decision support may be an analyst role. A role centered on inference and experiments may be data science. Include those outcomes only if they are truly part of this seat and the person will have the stakeholder access and time to do them.

PostgreSQL documents constraints as a way to enforce row relationships and uniqueness, and `INSERT ... ON CONFLICT` as a specific atomic insert/update mechanism. Neither alone defines correct event identity or business correction policy: PostgreSQL constraints and PostgreSQL INSERT.

↑ Back to sections

Worked brief: data engineer for subscription history

This sample assumes an existing subscription system, a relational source and a small analytics warehouse. Those are example premises, not a recommended architecture or a claim about common employers.

Blank brief to copy

Use this as a starting point and replace the example's assumptions. This page does not store your text.

Role and level:
First-quarter outcome:
First source and deliverable:
Event identity and timestamp owners:
Metric definitions and monetary policy owner:
Source, model and downstream consumers:
Replay/backfill authority and review path:
Day-one data, permissions and safe environment:
Schema evolution and compatibility expectations:
Privacy, retention and access approver:
Must-have capability -> assessment evidence:
Learnable tools -> reviewer/support:
First 30/60/90-day milestones and supplied support:
Out of scope:
Unknown policies to settle before posting:

The companion data engineer interview scorecard uses a late and duplicated subscription-event scenario. It tests data-contract and correction reasoning, not statistical modeling or dashboard polish.

↑ Back to sections

Assess one data-correction path fairly

Use the same synthetic event facts and staged changes for every candidate. Tell them the exercise is a discussion, not a SQL speed test, production migration or request to disclose prior-employer data. Invite clarifying questions and provide the same written scenario and time.

The core asks the candidate to define event identity and metric ownership, handle a duplicate and late event, respond to a schema change, then propose a bounded replay with lineage, access and before/after checks. Do not demand one product or streaming architecture. A sound answer may favor a batch correction if the product's freshness needs allow it, or a stream with explicit lateness and correction behavior where they do not.

Score observable decisions: what they preserve, who decides the business rule, how they constrain the correction, and how they know downstream outputs are right. If the candidate has not been given a metric definition or replay policy, credit them for making that missing authority explicit; do not expect them to invent finance policy.

Structured questions and common evaluation standards are supported as a process practice by the U.S. Office of Personnel Management's structured-interview guidance. This does not validate the scenario or provide jurisdiction-specific legal advice.

↑ Back to sections

Compare staffing paths and data authority

Choose based on whether the work is ongoing, whether decisions and access must remain with the organization, and whether there is an internal owner who can review definitions and corrections. A supplier's available hours are not equivalent to an employee's continuing ownership.

Before comparing options, align deliverables, start date, availability, included support, number of sources/consumers, review time, access level and handover. Keep cash and internal time separate; do not assign productivity multipliers to make unlike scopes look equal. The shared hiring budget worksheet records your own assumptions and does not estimate salary or supplier quality.

Reversal condition: if the job is a one-time connector with a stable contract and internal owner, consider a bounded engagement. If subscription definitions and corrections change every month and no internal owner exists, an external implementation alone will not create durable accountability.

↑ Back to sections

Set first-90-day milestones and support

If data access, the metric owner, test records or consumer review are unavailable, adjust the milestone and document the dependency. Do not judge a hire on an unresolved business policy or an inaccessible source system.

↑ Back to sections

Avoid predictable data hiring failures

• Calling every data role a data engineer. Decide whether the outcome is a reliable data product, business interpretation or statistical inference.
• Letting SQL define money. A query can run correctly while applying the wrong currency, effective-date, refund or recognition policy. Name the policy owner.
• Treating arrival order as truth. Late and duplicate events need explicit identity, event-time and correction behavior.
• Backfill by overwrite. Preview scope and impact, preserve needed history, bound the replay and verify before/after results with consumers.
• Schema changes without consumer review. Identify producers, downstream contracts and affected metrics before publishing a change.
• Broad read access for convenience. Define raw-data need, purpose, reviewer and access boundary before granting permission.

Preparation references: PostgreSQL interview guide, Python interview guide and, when cache or queue boundaries are in scope, Redis interview guide. The PostgreSQL vs MySQL comparison supports a database choice discussion; it is not a data-engineering qualification checklist.

This guide does not claim candidate supply, occupation-level pay, successful employer use or predictive validity. Its scenario is a discussion starting point. Have the relevant data and metric owners review it, and conduct a real dry run before using it in a consequential process. Continue to the data engineer scorecard.

↑ Back to sections

Choose your next step