I hope you are all fine. In this post I am going to share a quick tip on how to quickly test on your agent reasoning and answers. This is not written in stone but still works and it is an approach that works in many cases.

Consider a fictional promotion review. You give a conversational analytics tool a sales report, a margin calculation and a clear objective: decide whether to extend the promotion. One answer recommends expansion. Another recommends pausing. Both repeat the supplied numbers correctly.

Which answer should you trust? Checking the arithmetic won’t settle this disagreement. You need to examine the step between the evidence and the action. That’s the purpose of the proposed answer-change audit here: define what should affect the decision, preserve the supporting context, and investigate changes that don’t have an adequate explanation.

The numbers agree. Does the recommendation still follow?

In our fictional example, suppose the first response emphasizes sales growth while the second emphasizes margin pressure. Neither emphasis is automatically unreasonable. But how should we weigh them? Are we maximizing contribution profit, acquiring customers within an agreed budget, or clearing inventory?

Until we specify that requirement, the disagreement may tell us more about the question than the tool. The first useful check isn’t “Did the wording change?” It’s “Did we supply enough information to justify either action?”

For this proposed audit, numerical agreement and recommendation quality are separate review questions. An answer can reproduce the evidence faithfully without adequately explaining why the proposed action follows.

Define what should change before comparing answers

For the promotion review, separate three things: business evidence, decision requirements and presentation preferences. Evidence includes the supplied sales and cost information. Requirements include the objective and constraints. Presentation preferences concern how the explanation is delivered.

Write down expected sensitivity before running comparisons. A request for a shorter explanation should not, by itself, reverse the recommendation. A corrected cost estimate should trigger reconsideration of profitability. A newly stated budget constraint should enter the reasoning. Neither change necessarily requires the opposite action.

That last distinction matters. Responding to new evidence doesn’t have to mean reversing the decision. The answer might reasonably remain “extend the promotion,” but with a narrower scope or stronger qualification. Ask what reasoning should change, not just which final label you expect.

Run two comparisons, not one consistency test

Here is the proposed design. First, hold the supplied evidence and decision requirements fixed. Vary a presentation request you have judged irrelevant to the action for example, ask for a concise explanation instead of a detailed one. Review whether the conclusion, supporting evidence or uncertainty changes without justification.

Second, change a documented business input. In the fictional case, replace the original cost estimate with a corrected estimate. Check whether the response acknowledges the correction and follows its implications through the margin reasoning. Don’t require a changed conclusion when the revised evidence still supports the original action.

For each comparison, preserve the question, supplied context, source snapshot, response, run date and available model settings. Record the evidence period separately from the run date. Keep conversation history fixed where possible, and document what you cannot control.

Review factual claims, evidence selection, uncertainty and recommended action separately. Repeat the comparisons to characterize variation rather than treating one pair as definitive.

Useful adaptation is not a failure

But shouldn’t a finance lead and an operations manager get different answers? They may. In our fictional review, finance could ask about profitability while operations asks whether available stock can support an extension. Those are different decision requirements.

The proposed audit should allow that distinction. Different detail, caveats or actions can be justified when the objectives and constraints differ. What matters is whether the response explains the connection.

There’s an important limit here: fixing supplied evidence does not guarantee identical responses or establish why a difference occurred. This proposal offers no validated threshold for acceptable variation. An unexplained shift is a reason to investigate not a diagnosis of bias, model drift or any other specific cause.

Make the changed recommendation reviewable

Start with one recurring decision and a bounded pilot. Record what changed, what you expected to change, and whether the recommendation remains supported. Have the responsible analyst distinguish an acceptable presentation change from a missing requirement, a factual error or an unsupported change in the recommended action.

Before relying operationally on a changed recommendation, reconcile that shift against the evidence and decision requirements. If the explanation is still inadequate, seek further analysis rather than choosing whichever answer sounds more confident.

The target is justified responsiveness. A useful answer-change audit should challenge both an unsupported reversal and an answer that ignores relevant new evidence. Its practical purpose is modest: make the move from evidence to action reviewable before the business acts.

A worked example: should we extend the promotion?

Imagine a retailer deciding whether to extend a promotion for another week.

For this fictional example, the objective is explicit: extend only if the forecast contribution is at least €2,000.

Here is the supplied forecast:

Expected promotional sales: €20,000

Product costs: €12,000

Promotion-specific costs: €4,000

Forecast contribution: €4,000

We assume these are all the relevant costs for this simplified decision.

The AI recommends:

“Extend the promotion. Forecast contribution is €4,000, exceeding the €2,000 requirement.”

Now we run two comparisons, starting from that same baseline.

Comparison 1: change only the presentation

We keep the evidence, objective and conversation context fixed, but ask:

“Give me the recommendation in one sentence.”

A supported response would be:

“Extend the promotion: forecast contribution is €4,000, above the €2,000 requirement.”

The explanation became shorter. Nothing supplied justifies reversing the recommendation. If the AI now says “pause,” we should ask what supports that change.

Comparison 2: correct a business input

Return to the baseline. This time, we provide a correction:

“Promotion-specific costs are €7,000, not €4,000. Everything else remains unchanged.”

The calculation becomes:

€20,000 − €12,000 − €7,000 = €1,000 contribution.

A supported response would be:

“Do not extend under the current plan. The corrected costs reduce forecast contribution to €1,000, below the €2,000 requirement.”

Here, the recommendation changes for a traceable reason: a relevant input changed, and the revised result no longer meets the stated requirement.

But suppose the corrected costs were €5,000 instead. Contribution would be €3,000, still above the requirement. The AI should reconsider the calculation while reasonably keeping its recommendation to extend.

That is what we are checking: whether the response follows the evidence and decision rule. A changed answer is not automatically a failure, and an unchanged answer is not automatically a success.

For each comparison, save the inputs, prompt, response and available model settings. Repeat the comparison before drawing conclusions about a pattern. One unexplained reversal is something to investigate, it does not establish its cause.

Creating an isolated database environment is not the same as protecting the data inside it.

As AI agents become more involved in data engineering workflows, we’re increasingly giving them the ability to write SQL, modify schemas, run tests, and experiment with real datasets.

Database branching helps us isolate these changes from production. But there’s another question we often overlook: does the agent actually need access to all that production data in the first place?

In this article, I explore the distinction between what an experimental environment can change and what it should be allowed to see, using a recent Databricks architecture example to discuss data access, isolation, and governance in the age of AI agents.

You’re preparing an experimental database for a developer or AI agent. Before creating it, ask two questions: what can this task change, and what should it be allowed to see? The proposed distinction is simple: permission to experiment with isolated writes should not automatically authorize access to every production record.

That changes the provisioning conversation. “Can we create a branch?” is useful, but it isn’t the starting point. Start with “What information does this task actually need?” Then choose an environment that gives the work enough room without making broader visibility the default.

What branching solves and what it leaves open

In its October 31, 2026 account of Snorkel AI’s CI/CD architecture, Databricks describes database branches where pull-request tests can change data and schemas without affecting production or other branches.

The account also describes branch deletion with automatic expiration as a fallback. It includes private connectivity and separate identities for developers and pipelines. These are meaningful provisions in the described design.

Now separate two decisions. Preventing an experiment from changing production answers a mutation question. Deciding whether that experiment needs customer contact details answers a visibility question. Approve these separately, even when the same platform helps implement both.

The article does not specify field masking or external-output controls. That omission is not evidence that those controls are absent, and it does not establish a vulnerability.

Two permissions, one hypothetical transformation

Consider a hypothetical retailer delegating a SQL transformation that groups orders by product category. The requested output is category-level order totals. The task needs testing against relevant relationships and edge cases, but customer email addresses aren’t inputs to that calculation.

Start with the characteristics the test needs: orders linked to products, products linked to categories, missing classifications, and whatever cancellation rules the transformation must handle. Then decide how to represent them. Synthetic records, a restricted subset, masked fields or approved production records are candidates not interchangeable guarantees of realism.

For this task, permission to create intermediate tables could be broad while permission to inspect customer fields remains narrow. That’s the practical value of separating the decisions. You can give the transformation room to fail without assuming it needs unrestricted visibility.

Here’s a useful question for the team: what defect would access to an additional field help us detect? If nobody can explain the connection, leave that field outside the initial task scope. The decision is about supporting the work, not collecting access just in case.

But realistic data can be essential

There’s a reasonable objection here: preparing restricted test data takes work, and an artificial dataset may miss the problem. Databricks motivates its approach partly through real data shapes and the limitations of shared staging environments, fixtures and synthetic data.

So don’t turn minimization into a rule that makes investigation impossible. Ask what realism means for this particular problem. Is the suspected failure about unusual relationships, malformed values, historical combinations or application behavior? Define the condition before choosing the dataset.

Then test whether a restricted dataset preserves that condition. If it does not, document why broader access is necessary and who should authorize it. Full production visibility can remain an option without becoming the automatic consequence of creating a branch.

There’s a cost to this approach: someone must understand the task well enough to define its data needs. The proposed trade-off is to spend that effort explicitly rather than let convenience decide the exposure boundary.

Approve the task, not just the branch

Before provisioning, record permitted data, permitted mutations, task identity, expiry and allowed output destinations. Keep the record short enough to use, but specific enough that another person can understand why the access exists.

For the hypothetical retailer, that might mean category-relevant order fields, permission to create experimental tables, a dedicated task identity, a defined end point and results returned only through the approved review destination.

Why include outputs? Imagine the hypothetical task exports records into a debugging attachment. Deleting the database would leave the attachment untouched. The analytical point is about lifecycle: ending an environment’s lifetime should not be treated as proof that every resulting artifact has disappeared.

Databricks also proposes extending branching to AI-agent tasks, with changes reviewed before anything is applied to production. The source presents this as an extension of the pattern; it does not establish agent deployment outcomes.

Make production promotion a separate decision. A successful experiment should produce evidence for review, not automatic permission to deploy. Review the proposed change, its test evidence and its production implications independently of the original permission to experiment.

The recommendation isn’t to avoid production data or weaken experimental environments. It’s to stop bundling three decisions together: what the task may see, what it may change and what may eventually reach production.

For the next experimental SQL task, answer those questions before creating the branch. Give the work enough realism to be useful and enough freedom to explorebut make broader data visibility a justified choice, not a side effect of convenience.

References

1. Yasamin Tari and Kartik Mathur / Databricks — Lakebase branching for CI/CD: lessons from 100 PR databases (2026-10-31)

I’m bringing this space back to life, with more regular writing on data, analytics, data engineering and AI and the questions that come up when we try to make them useful in practice.

Working with data and teaching through DataConscious keeps bringing me back to a simple question: what does this actually mean for the person making a decision?

We’ll explore new developments, unpack papers and work through concrete examples, connecting technical choices with the business questions behind them.

If that sounds like your kind of conversation, welcome back or welcome aboard. There’s more to come.

Before putting a combined measure into an AI answer, check what it counts not just how the data arrives.

Imagine a fictional manufacturer asking a straightforward question: have the orders we shipped this month been invoiced? Finance has SAP records. Logistics has delivery events in another system. You need a combined answer so someone can decide which orders need attention.

Now suppose access between those systems becomes easier. That’s useful. But before relying on the answer, there’s another question to work through: what exactly have we combined? The argument here is simple: removing an extraction boundary should trigger a targeted reconciliation of business meaning, not make that reconciliation unnecessary.

What the announcement actually establishes

Databricks’ announcement published October 30, 2026 provides the immediate context.

Databricks describes sharing SAP data products through SAP BDC Connect for zero-copy access, avoiding additional extraction pipelines and copies. It also describes a foundation carrying SAP business semantics. These are vendor statements about the integration, not findings from independent testing.

Availability needs a careful reading. The announcement says SAP Databricks and SAP Business Data Cloud Connect for Databricks are available. That does not mean every expanded capability is deployed: the announcement also describes plans.

Specifically, the Genie–Joule integration is described as planned. It should not be presented as an established deployment.

The announcement also describes combining SAP data with other sources. That is where the question for an analytics team begins. The announcement supplies the integration context, it does not establish that a particular combined business measure has been reconciled.

A valid join can answer the wrong question

Return to the fictional manufacturer. Suppose an order line has several shipment records and several invoice records. If you join both sets directly to that line, each shipment can pair with each invoice. Sum invoice amounts across those combinations and you repeat amounts. The query runs. The total is still wrong for the question you wanted to answer.

The proposed modelling response is to establish the grain—the meaning of one row—before calculating the measure. Would an order-line summary answer the question? Or do you need an explicit allocation between shipped quantities and invoiced quantities? Start with the decision you need to support. A shared order identifier alone does not settle which relationship you need.

Now suppose you prevent all repeated counting. There’s still a separate issue: an order placed in one period, shipped in another and invoiced later belongs to different reporting periods depending on the question. An order-date sales report and a shipment-date invoicing check need not agree.

That is the important distinction in this hypothetical case: some differences are calculation errors; others are legitimate differences in scope. Neither is evidence of a defect in SAP or Databricks. The example shows why checking technical access and agreeing on the combined measure are separate jobs.

Doesn’t preserved business context solve this?

That’s a reasonable objection. If the source brings its business definitions with it, why reconstruct them? The recommendation is not to reconstruct them. Reuse those definitions and check their scope. A reconciliation exercise should investigate what the combination adds, not discard useful source context.

For the fictional manufacturer, the additional question is whether an external delivery event corresponds to an order line, a partial quantity or a completed shipment. Then you need to decide what evidence counts as invoicing that shipment. Those choices belong to the combined question, even if each source is internally well defined.

This is also where the recommendation has a limit. If the supplied definitions and relationships already resolve these questions, the check may be brief. The proposal is targeted reconciliation, not a mandatory semantic rebuild. Neither the effort required nor the resulting benefit has been measured here.

Reconcile one measure before widening the answer

Start with one representative measure and a short worksheet. Record the decision it serves, the source-report baseline, the grain of each input, the join relationships, included and excluded statuses, the reporting date and the aggregation rule. Add unresolved differences and a named owner who can settle the business interpretation.

For the fictional shipment question, compare a bounded set of order lines with the relevant source reports. Trace a partial shipment, a later invoice and a cancelled line. Can you explain each difference using the documented rules? The aim is not to force two totals to match when they describe different populations or periods.

Before operational use, resolve material ambiguities or explicitly limit the answer to the scope that has been checked. For example, the proposed answer might cover invoice coverage for completed shipments while excluding unresolved partial allocations. That’s a narrower promise, but someone can explain what it includes and what it leaves out.

Treat easier access as an opportunity to make the next question more precise: can someone explain why this combined result differs from the source report? The proposed business payoff is not a guaranteed productivity gain. It is a clearer basis for deciding what needs attention: operational performance, a modelling error or simply a change in what the measure counts.

This is a hypothetical example. But take a moment to consider the correction you’d choose.

Would you provide a clearer screenshot? Explain what “overdue” means? Ask it to count distinct invoice IDs? Check whether it included paid invoices? Recalculate the total?

Each intervention addresses a different problem. The final number doesn’t tell you which one happened.

Before choosing a correction, we need to separate the questions hidden inside the request: what the model recognised, what the business definition required, which unit it counted, which items it included and how it calculated the result.

That’s where the analyst’s judgment becomes useful.

What visual counting research can tell us

The Latent Space article by Richard MacManus provides an introduction to this work. The research evidence comes from the DecoyCount preprint, published on 26 October 2026, which describes a benchmark of 1,660 images. It compares neutral counting questions with targeted instructions that clarify the counting rule without revealing the answer. The image stays the same. The authors report higher accuracy with those targeted instructions.

DecoyCount, Figure 1: different visual traps can produce different wrong counts. Source: Zhang et al., DecoyCount, 2026, CC BY 4.0. User-supplied screenshot, unaltered.

That comparison offers a useful diagnostic starting point: change the instruction and observe what changes in the answer.

An improvement supports the interpretation that task specification mattered. It doesn’t establish that every object was recognised correctly or reveal the model’s internal process. Likewise, an unchanged wrong answer leaves several possible explanations open.

For an analyst, the next step is to look for observable evidence at each stage of the task.

What changes when the counting rule is clarified?

The chart separates accuracy under a neutral question from the additional accuracy obtained with a targeted cue. The striped extension is the reasoning-alignment gap, measured in percentage points. It is not a confidence score or the proportion of all errors explained by one cause.

DecoyCount, Figure 2: targeted guidance helps differently across failure families. Source: Zhang et al., DecoyCount, 2026, CC BY 4.0. User-supplied screenshot, unaltered.

The question for our workflow is which check helps on the cases we actually handle. A benchmark comparison can inform that question; it cannot choose the correction for our invoice report.

Five places to investigate

Here’s a practical framework for the invoice example. These five categories express the approach proposed in this article; they aren’t DecoyCount’s validated taxonomy. Several failures can occur together.

Recognition

Question to investigate: Were the dates, IDs and statuses read correctly?

Observable check: Compare extracted fields with the screenshot.

Possible intervention: Improve the image or extraction; verify unclear fields.

Business definition

Question to investigate: What qualifies as overdue?

Observable check: Agree on the reference date and treatment of payment status, disputes and exceptions.

Possible intervention: Document the rule and include it in the task.

Counting unit

Question to investigate: Are we counting rows, invoices or customers?

Observable check: Check identifiers and repeated entries.

Possible intervention: Specify the unit and how duplicates should be handled.

Selection

Question to investigate: Which items satisfy the agreed rule?

Observable check: Compare an inclusion list with independently checked labels.

Possible intervention: Correct the filtering instruction or logic.

Calculation

Question to investigate: Does the total follow from the accepted items?

Observable check: Recompute from the validated list.

Possible intervention: Use a deterministic calculation.

The distinction between definition and selection deserves attention.

If nobody has agreed whether disputed invoices count as overdue, the definition needs work. Once that rule is agreed, including an invoice that fails it becomes a selection error.

The counting unit introduces another question. Two rows might represent two invoices, or two line items belonging to one invoice. Even perfect reading and arithmetic won’t resolve that ambiguity without an identifier and a counting rule.

Same wrong total, different corrections

Let’s make the hypothetical example more specific.

For this exercise, the agreed rule is to count each distinct invoice ID once, include only invoices marked unpaid, and require the due date to be earlier than 16 October 2026. Assume every invoice in the screenshot has a clear paid or unpaid status.

Now consider two possible findings.

Finding one: a due date was read incorrectly

An unpaid invoice is due on 18 October. The extracted record says 15 October, so it enters the overdue list.

The selection decision follows the agreed rule using an incorrect input. Repeating the overdue definition won’t correct the date in that extracted record.

Check the source field. Try a clearer image or another extraction method, and compare the resulting value with the original. If the field remains unclear, flag it for human verification before calculating the total.

Finding two: a paid invoice was included

This time, the extracted due date and payment status match the screenshot. The invoice was due on 15 October, but its status is paid. It still appears in the overdue list.

Here, the observable discrepancy is the inclusion decision. A targeted clarification would be:

For this count, exclude invoices marked paid.

That instruction specifies a rule without supplying the expected total. Compare the affected invoice’s inclusion decision before and after the clarification.

If the decision changes appropriately, you have evidence that this instruction helped on this case. You still need to check the rest of the list.

In both branches, the total could be wrong by one. The evidence points toward different interventions.

Make the intermediate output inspectable

A useful diagnostic request is:

Return each visible row’s invoice ID, due date and payment status, with an include or exclude decision under the supplied rule. Mark unreadable fields as unclear. Identify repeated invoice IDs before reporting the distinct-invoice total.

This produces something you can compare with the source. Treat that table as another output to verify: a tidy table can contain an incorrect date or an unjustified inclusion decision.

Once the fields and included invoice IDs are validated, calculate the total with a spreadsheet formula or query. This separates checking the evidence from computing the result.

Check whether the correction holds

A corrected answer is a reason to investigate further. To evaluate a proposed correction, preserve the original image, prompt, response and model settings where available.

Then:

1. Establish the expected answer independently, including the accepted invoice IDs and rule.

2. Change one relevant instruction at a time so you can interpret the comparison.

3. Inspect which items changed, alongside the total. Two mistakes could cancel each other out.

4. Repeat comparisons where outputs vary, and test additional cases representative of the intended workflow.

Keep the expected answer outside the prompt used for evaluation. Otherwise, you risk testing whether the model follows your suggested answer.

These are proposed evaluation practices for an analytics workflow. The invoice example illustrates them; it doesn’t demonstrate measured performance.

Keep the benchmark’s limits in view

DecoyCount retains examples partly because models fail them, and its imagery includes edited or generated material. The authors explicitly caution that absolute accuracy doesn’t estimate performance across a regular population of images. Its scores therefore cannot establish how often AI will miscount your company’s invoices or dashboards. See the paper’s collection and validity checks.

The paper’s aggregate results are included below for reference. Read the multi-hop column cautiously: it uses a different valid paired subset and is not directly comparable with the neutral and cued columns. These are the study’s results, not measurements from our invoice example.

DecoyCount, Table 2: aggregate results under the paper’s evaluation conditions. Source: Zhang et al., DecoyCount, 2026, CC BY 4.0. User-supplied screenshot, unaltered. Open the linked paper to inspect the table at full size.

The application proposed here is narrower: investigate the discrepancy before selecting a correction, then test that correction on your own representative cases.

A useful working habit is to record four things: the expected unit, the inclusion rule, the observed discrepancy and the check intended to catch it.

That record connects the number to the decision someone will make with it. It also makes the analyst’s contribution concrete: explain which error the evidence supports, what remains uncertain and how the next check will help.

References

1. Zhang, Qiuyi (Richard); Xu, Zhen; Gu, Jing; Bang, Seojin; Das, Arnav Mohanty; Hu, Yaning; Kumar, Gautam; Lichtarge, Jared; Xue, Le; Catoto, Bettina; Datta, Akul; Luo, Daniel; Khan, Farha; Valentine, Marcella; Yang, Yinfei; and Dai, Andrew M. (26 October 2026). DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting. arXiv:2610.34855v1

2. MacManus, Richard (29 October 2026). Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning. Latent Space. Read the article.

A favorite project that started from a gap that I saw over and over again and I confirmed it through the first course I built, Data Analytics In Modern Corporate Business.

That's why I created its evolution, for a more senior audience, keeping the same philosophy. This is the next level.

What the participants loved about the previous course was that they didn't just see tools in isolation, but the real connection between business, data, pipelines, models, BI and decision-making.

Now, through a real lab environment, built on a modern architecture, on top tools, we can see in practice very important data analytics and data engineering concepts and give the participant the opportunity to experiment effortlessly in his own environment and understand in depth very important things about a corporate environment.

And it wouldn't be possible without AI. Under no circumstances would I have the full range of knowledge to build all this on my own. We are talking about another world now. And even if I had the knowledge, it would take me months morning and night.

The first premium course through this project will be the

The Senior Data Analyst Accelerator, which puts you in the real life of a senior analyst.

A hands-on cohort for mid-level analysts who want to keep up with developments, move towards senior-level thinking and influence the business.

On how to embrace AI and having the necessary supplies, unlock endless possibilities with the role of an analyst.

Who is no longer just an analyst.

• The first cohort will start in September

• It will have only 10 seats.

The reason places are limited is because I want the first cohort to be small, stable, quality and practical.

A small team of people who will work on a real modern analytics environment.

If you're a mid-level data analyst and feel like you want to better understand where the role is going, how tools are connected, how trust in data is built, and how a more senior analytics mindset works, you can join the waitlist.

I don't want the participants to spend their energy on "how to set up the environment" but on "how a modern data ecosystem really works".

That's why the course comes with:

➡ ready access to all platforms,

➡ usernames and passwords for the learning environment,

➡ fully configured repository,

➡ ready-made pieces of code,

➡ access from a browser,

➡ a set up corporate infrastructure to work on

👉 Register an interest here: https://labs.data-conscious.com/waitlist

I will personally contact those who are on the list.

Read more about the lab here: https://labs.data-conscious.com/

Read more about the course here: https://labs.data-conscious.com/daaccelerator

Hello folks,

In this post, I just want to mention how important it is to show who you are through the way you create. Never underestimate care, creativity and imagination. So I decided to share on Substack an older post about something personal and very close to my heart.
👉 A full, step-by-step walkthrough on 𝗛𝗼𝘄 𝘁𝗼 𝗕𝘂𝗶𝗹𝗱 𝘆𝗼𝘂𝗿 𝗖𝗩 𝘂𝘀𝗶𝗻𝗴 Tableau.
People underestimate how much a hiring manager appreciates effort, creativity and proof of skill.
Not because a fancy CV guarantees a job.
But because, for a hiring manager, it answers a deeper question:
‘𝗜𝗳 𝘁𝗵𝗶𝘀 𝗽𝗲𝗿𝘀𝗼𝗻 𝘁𝗿𝗲𝗮𝘁𝘀 𝘁𝗵𝗲𝗶𝗿 𝗼𝘄𝗻 𝘀𝘁𝗼𝗿𝘆 𝘄𝗶𝘁𝗵 𝘁𝗵𝗶𝘀 𝗺𝘂𝗰𝗵 𝗰𝗮𝗿𝗲… 𝗶𝗺𝗮𝗴𝗶𝗻𝗲 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗰𝗮𝗻 𝗱𝗼 𝗳𝗼𝗿 𝘁𝗵𝗲 𝗰𝗼𝗺𝗽𝗮𝗻𝘆.’
And I’ve seen this personally.
So many hiring managers opened the interview asking me about my Tableau CV.
It opened many doors.
Not because someone will hire you just for this.
But because if you were the one reviewing candidates you would absolutely 𝗴𝗶𝘃𝗲 𝗮 𝗳𝗲𝘄 𝗲𝘅𝘁𝗿𝗮 𝗽𝗼𝗶𝗻𝘁𝘀 𝘁𝗼 𝘁𝗵𝗲 𝗽𝗲𝗿𝘀𝗼𝗻 𝘄𝗵𝗼 𝗱𝗮𝗿𝗲𝗱 𝘁𝗼 𝗽𝗿𝗲𝘀𝗲𝗻𝘁 𝘁𝗵𝗲𝗶𝗿 𝘀𝗸𝗶𝗹𝗹𝘀 𝘄𝗶𝘁𝗵 𝗰𝗹𝗮𝗿𝗶𝘁𝘆, 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝘃𝗶𝘁𝘆, 𝗮𝗻𝗱 𝗶𝗻𝘁𝗲𝗻𝘁𝗶𝗼𝗻.
And that’s the whole point:
𝗧𝗼 𝘀𝗵𝗼𝘄 𝘄𝗵𝗼 𝘆𝗼𝘂 𝗮𝗿𝗲 𝘁𝗵𝗿𝗼𝘂𝗴𝗵 𝘁𝗵𝗲 𝘄𝗮𝘆 𝘆𝗼𝘂 𝗰𝗿𝗲𝗮𝘁𝗲.
If you want to follow the journey, learn, be inspired, or simply see how far creativity can go in analytics, you can find my newsletter and subscribe here:

Using Tableau (and Tableau Public for uploading) to showcase your portfolio demonstrates your expertise in data visualization, storytelling, and analytics, which are essential skills for the role. Incorporating Tableau dashboards or visualizations into your CV showcases your technical proficiency in using this tool.

• It highlights your ability to create interactive visualizations, and derive insights, which can be very appealing to potential employers.Using Tableau to create your CV stands out among traditional CVs.

• It can make your application memorable and unique, leaving a lasting impression on recruiters or hiring managers who are often inundated with conventional resumes.Tableau offers interactivity and engagement through its dashboards.

• Incorporating interactive elements in your CV allows the recruiter to interact with your data visualizations, providing them with a more immersive experience, which can set you apart from other candidates.

• Your Tableau-based CV can serve as a portfolio demonstrating your previous projects, data visualization skills, and the impact you’ve made in your previous roles.

• Using Tableau for your CV demonstrates your fantasy and inclination toward innovation and staying updated with the latest trends in data visualization and analytics.

While creating a CV with Tableau can be impressive, it’s essential to ensure that the visualizations are clear, easy to understand and relevant to the job you’re applying for.

Balance creativity with readability and relevance to make a compelling case for your candidacy as an analyst.

Below I’ll demonstrate Step-by-step the way I’ve created mine. Perhaps there’s space for improvement.

1. Data sources

Tableau Public accepts only extracts as data source type (live or extract).

• We have to create our personal data in cloud sheets (One Drive in our example, you can also use GSheets) and load them using the appropriate connector. You can choose the names and the type of content you want to include in your CV. I’ve created files for skills, experience, interests, personal data, etc., as seen below.

Experience:
Choose the names and the content you want to showcase for your experience. I’ve included the links for companies and universities.

Personal Data:
Here, we list our personal information. I chose email, location, nationality, and age. I also incorporated today’s date, birthday data in order to calculate the age integer (with div formula) every time the extract is refreshed.

Skills:
To provide the functionality of filtering the skills per company with dashboard actions, create the skills file including the name of the company in a tall data format.

• Create a separate data source for every Excel file. I chose to create a new Excel file and not a new sheet (in the same doc) for every table I need. Feel free to migrate all your data into a single document.

Content

The type of content you want to include is up to you and your preference. Of course, there are some basic guidelines and things to avoid. In my case, I chose to include:

1. Photo in the upper left corner along with a short text.

2. Personal details below the photo.

3. In the bottom left corner, we’ll find clickable logos for LinkedIn, GitHub, etc. The dashboard consists of 3 vizzes:

4. Education & Professional experience visualized with a Gantt chart. The Gantt chart includes details when hovering to showcase each role’s achievements and details.

5. Skills that can be filtered when the user clicks each role’s bar. The list will then filter out only the skills and technologies used in each role based on the company’s tech stack and projects.

6. Certifications along with clickable logos for each provider that redirects the user to the corresponding certification’s link.

Interaction

The workbook demonstrates some functionalities such as:

1. Redirecting to specific URLs when clicking icons for social links, certification links, personal data, etc. with Tableau Dashboard Actions.

1. As mentioned, filtering the skills and technologies used in each role based on the company’s tech stack and projects.

To use and match logos with providers, skills and in general column values, we use custom shapes.

Custom shapes in Tableau are fast and easy to use and provide the user unlimited control over mark shapes. Tableau shapes and controls can be found in the marks card to the right of the visualization window. There are plenty of options built into Tableau that can be found in the shape palette, but what if you want to use custom shapes for effect, branding, etc.?

There is a folder in My Documents called My Tableau Repository where you can find a shapes folder. Within this shapes folder, simply create a new folder for your shapes and name it an informative name. Save any custom shapes you would like to use in your visualization into this file.

To load your shapes, hit reload in your shapes palette and then select the new shapes folder from the drop-down menu.

Summary

Putting all these together you can have your Tableau CV and easily demonstrate smartly some of your Tableau experience.

Crafting your CV with Tableau offers a unique opportunity to showcase your BI skills creatively while adhering to essential CV and visualization guidelines. By leveraging Tableau’s dynamic features, you can artfully present your professional journey, skillset, and achievements in a visually compelling manner. Through imaginative yet structured visualizations, emphasize clarity, relevance, and a cohesive narrative. Ensure readability by using clear labels, concise descriptions, and intuitive design elements.

Balance creativity with professionalism, incorporating industry-standard CV sections such as experience, education, and skills, while infusing innovation through interactive elements and engaging storytelling.

You can view and download the workbook in Tableau Public.

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

We continue on the Data Career Compass post-series and today is the 4th one for the role of Head of Data Analytics/Business Intelligence. In the previous podcast episode for Analytics Engineer role I mentioned that it was the 4th episode. The truth is that Analytics Engineer was planned to be after this one, so this is the actual 4th blog post & podcast episode.

Today, we shift our focus to a higher level, the role of the Head of BI/Data. What does it really mean to lead a BI function at scale and how do you drive strategy, structure and decision-making across an organization? While I haven’t personally held a Head of BI position with BI Leads reporting to me, that’s exactly why I’m excited about this conversation because our guest brings that experience to the table.

I’m truly happy to reconnect with one of my former line managers, Christos Lontos -Head of BI at Skroutz (❤️). Christos has been a major influence, not just professionally but personally too (…and this makes me happier) and I’m thrilled to share his insights with you. That’s where I was inspired to write about Increasing the reputation of Business Intelligence teams within an organization. This where I’ve learned for the below ways of working for BI Teams.

In this episode of DataConscious – A Mindful Approach to Analytics, we sit down with Christos Lontos, Head of BI at Skroutz, to explore what it truly means to lead a modern BI function.

🎙 What does a Head of BI/Data really do? We explore how this role goes beyond reporting shaping data strategy, enabling decision-making and empowering teams across the organization.

🎙 His journey into the role: From software engineering to big-data projects and eventually leading the BI function at Skroutz, Christos shares how each step shaped his leadership style.

🎙 A day in the life: From strategic alignment and cross-functional partnerships to balancing fire drills and long term roadmaps, you’ll hear how he keeps his team focused and impactful.

🎙 Leading with intention: Christos shares how he builds relationships, earns trust across departments and fosters a team culture rooted in ownership and curiosity.

🎙 Hiring mindset: Thinker-doers over resume flash. He explains what makes a candidate stand out and how he builds a resilient, value driven BI team. ( I can confirm that ✅)

🎙 Honest leadership truths: Why mastering the fundamentals matters and why you should be ready to let go of them when the mission calls for it.

🎙 Advice for future Heads of BI: Grounded reflections on what he wishes he knew earlier and how to lead with empathy, adaptability, and purpose.

🎙 The next decade of AI: Predicting where we’ll be in 2035

🎙 What qualities does he look for when hiring data professionals?

🎙 How AI will affect the hiring of BI roles?

Christos highlighted that BI alignment is not about serving internal customers or responding to whoever requests something first. The Head of BI anchors the team on strategic initiatives often through OKRs to ensure BI resources create real business leverage.

The Head of BI sits at the crossroads of business strategy and data execution. It’s not a role about chasing dashboards or cleaning up everyone else’s mess. It’s about seeing the bigger picture, then building the systems, teams, and culture to bring it to life.

Yes, the job involves data. But not in the ‘just run this query’ way. Instead, the Head of Data defines how an entire organization uses data to think, act, and grow. In a startup, this might mean getting your hands dirty writing SQL, building dashboards, optimizing pipelines. In a more mature environment, like a large ecommerce company, the role starts to resemble a Chief Data Officer, shaping long-term architecture, driving governance, and coaching multiple data leaders across BI, engineering, and analytics.

Take Zalando for example. One of their engineering managers shared how the data team overhauled legacy systems using AWS standards and as a result, cut operational costs while delivering more, faster. It’s not about buzzwords it’s about real outcomes. Less friction. More speed. Smarter decisions. That’s the impact a good Head of Data can have. They’re not the smartest SQL writer in the room. They’re the person who connects the dots and clears the way.

Whether you’re growing into a leadership role or already leading a team, this episode offers a clear and thoughtful look at what it means to head a BI function that truly drives value.

🎧 Listen now & subscribe for more insights on mindful data practices!

Strategic Vision and Leadership

At its core, the Head of Data role is about strategy and leadership. You’re responsible for defining the data roadmap and ensuring it serves business objectives. This often involves:

• Data Strategy & Culture: Establishing a data driven culture across the organization. The Head of Data partners with C-level and business leaders to understand key goals and uses data to measure and drive them. They may introduce data governance, set data quality standards and evangelize self service analytics so teams can make informed decisions. At Skroutz, for instance, the leadership chose Amazon Redshift to ‘promote data democratization, empowering teams across the organization with seamless access to data, enabling faster insights’. A Head of Data would be the one championing that democratization, ensuring the data platform is accessible and transparent to marketing, product, finance and more.

  Christos described one of the toughest aspects of the role: delivering daily value while simultaneously building the systems, structures, and team needed for next year’s scale. This duality is unavoidable and mastering it defines senior data leadership. Build the car while riding the horse

• Building & Mentoring Teams: Unlike an analytics manager who might oversee a small reporting team, the Head of Data should supervise multiple leads (Analytics Engineers, Data Engineers, BI Managers, Data Scientists, etc.). They recruit and mentor these leads, helping them grow into strong specialists. In other words, the Head of Data is a people leader, ensuring each team has the right skills (cloud experts, SQL gurus, machine learning engineers) to execute on strategy.

  • Avoid the technical trap (don’t jump in and code).

  • Your team is your output (success is collective).

  • Expect to be misunderstood when driving change (friction means progress).

These can be a separate section:

• Cross-functional Collaboration: Heads of Data work across departments marketing, product, IT, operations to align data projects with business needs. They translate technical capabilities into business value (for example, turning raw clickstream data into a revenue forecast model for the retail team). Collaboration is key: an effective Head of Data partners with CTOs, product owners and business stakeholders to integrate data solutions smoothly. This might include rolling out a company wide BI dashboard, modernizing the data warehouse, or enabling new AI/ML products that directly impact revenue.

• Technology & Architecture Direction: A strategic Head of Data stays on top of the technology stack. They decide on architectures (data lakes, warehouses, real-time streams) and platforms (AWS, GCP, Azure) that will best serve the company’s scale and goals. In fact, companies leverage data infrastructures to combine transactional (e.g. SAP) and analytical data into near-real-time reports. The Head of Data would oversee such migrations or implementations, balancing performance, cost, and accessibility.

In practice, this means setting up a vision document or roadmap and then working backwards to pick initiatives. Will we migrate from on-premise SQL Server to Snowflake? Do we build a Kafka streaming layer for real-time user events? A Head of BI makes those calls along with engineering leaders and architects. They also measure the success of these projects in business terms (e.g. faster decision cycles, reduced storage cost, new revenue channels).

A Day in the Life: Leading and Deciding

On any given day, a Head of Data may:

• Meet with executives to review data driven KPIs (sales trends, churn rates, server usage) and adjust strategy.

• Oversee data team leads, reviewing progress on projects: Has the Snowflake migration reduced query times? Are dashboard refreshes failing? They solve people/process issues rather than writing the SQL themselves.

• Prioritize new initiatives: maybe the supply chain team needs a new predictive model, or marketing needs a campaign attribution dashboard. The Head of Data evaluates ROI and resource needs.

  As Christos notes, the Head of BI must balance responsiveness with discipline embracing ad-hoc queries to discover hidden needs while preventing them from disrupting long-term impact. The solution is pattern recognition and building self-service capabilities that absorb repetitive requests.

• Liaise with IT/security/governance: ensure compliance (GDPR data handling, access controls) and data quality audits are in place.

• Champion tools adoption: They might demo a new BI tool to stakeholders or agree on budgets for cloud services (e.g. migrate from Snowflake to AWS Redshift).

• Represent data at the executive table, translating technical progress into business terms and vice versa.

Importantly, while they understand the technologies (cloud, databases, machine learning) and might have written code earlier in their career, the Head of Data’s daily tasks are management and decision making. They delegate technical tasks to specialists.

For instance, a Data Lead might handle schema changes or build dashboards, but the Head of Data would say why those are needed, when and measure if they are delivering value (faster insights, increased revenue, cost savings).

Think of it this way: if a Data Engineer designs a pipeline, the Head of Data ensures that pipeline matches the company’s five year growth plan. If a Data Analyst delivers a report, the Head of Data asks how those insights will influence marketing or product strategy.

Key Skills and Career Path

The journey to Head of Data usually goes through roles like Data Analyst, Data Engineer, BI Manager or Data Science lead. Along the way, you build up expertise in data tools and grow soft skills. Here are some actionable steps and competencies for aspiring Heads of Data:

• Broaden Your Expertise: Master the full data stack, from ingestion (Kafka, APIs) to storage (Redshift, Snowflake, BigQuery) to transformation (dbt, SQL, Spark) to BI (Tableau, Looker). Understanding these tools helps you make informed choices and speak the language of your engineers.

• Develop Business Acumen: Learn how different departments use data. Can you read a finance report? Do you understand customer acquisition metrics? The best Heads of Data connect data projects to revenue, margins or customer experience.

• Build Leadership Experience: Lead small teams or projects to practice decision making. Mentor junior analysts/engineers.

• Communicate Clearly: Hone your ability to explain technical ideas to non-technical people. The Head of Data often pitches projects to executives or trains business teams on data tools.

• Stay Curious about Strategy: Always ask “why”. Why do we need this dashboard? How will this data product move the needle for the business? Heads of Data think in strategic terms, not just technical specs.

• Embrace Agile & Lean Principles: Modern data teams move quickly. Heads of Data often adopt agile methodologies, iterative delivery and OKRs/KPIs to measure success.

Finally, network with industry peers. Podcasts or communities like DataConscious often feature interviews with Heads of Data. For example, collaborating with colleagues (even if not named here) at Skroutz or other firms can reveal insights about balancing ecommerce metrics with data initiatives.

Learn from their experiences: how did they manage schema changes without slowing dev teams (as Skroutz did with Debezium/Kafka)? How did they guide major cloud migrations (as Zalando did with AWS services)? These stories can inform your own approach.

Remember, at the end of day you’ll probably end using Excel for one more (last) time.

The evolving role of BI in the Age of AI

Christos emphasized that AI literacy is rapidly becoming as essential for BI professionals as SQL literacy has been for the past decade. The shift underway is not merely technological but deeply cultural. Programming and analytical work, once dominated by traditionally trained engineers, are becoming accessible to broader groups of people with diverse backgrounds, mindsets, and values. This ‘destandardization’ of technology where historians, nurses, designers and domain experts can create software and intelligent agents through natural language will reshape how data products are built and how organizations operate.

According to Christos, BI leaders must be ready for a world where debugging resembles adjusting the behaviors or ‘personalities’ of AI agents rather than fixing code. In this landscape, structured thinking, clarity of communication, and the ability to collaborate with AI systems will matter even more than today. The Head of BI must prepare teams for this transformation, ensuring they can leverage AI as a performance multiplier while maintaining strong fundamentals in analytical reasoning, problem decomposition, and stakeholder communication.

Conclusion

The Head of Data is a mindful mix of strategist, technologist and leader. They ensure that data flows smoothly, decisions are evidence-based, and teams are empowered. In e-commerce and marketplaces like Amazon, Zalando and Skroutz, this role can dramatically influence growth: from enabling real-time analytics to cutting data costs and creating new data products (recommendations, forecasts, etc.)

If you’re aiming for this role, focus on both “big picture” thinking and the hands-on knowledge to back it up. Align data projects with business impact, build strong teams explore the the data stack. As companies scale, they need visionaries who can turn raw data into strategic advantage. A successful Head of Data lights the path from data to insight to action.

In a nutshell…

3 Interesting Posts to read

The Last Dev

The Last Dev ~ Cloud Updates #8

Alongside the format, I am also changing the name of this series. The idea is to express my personal opinion on some updates that I found interesting, and showcase some local events and articles…

Read more

a year ago · 3 likes · 2 comments · Konstantinos Siaterlis

Konstantinos Siaterlis demonstrates how to deploy a Minecraft server using AWS ECS Fargate and Terraform. The project involves setting up a Virtual Private Cloud (VPC) with public and private subnets, configuring a Network Load Balancer and integrating Elastic File System (EFS) for persistent storage. The tutorial emphasizes Infrastructure as Code (IaC) practices and provides a practical example of deploying containerized applications on AWS. The full code is available in the accompanying GitHub repository.

git push -f leadership

Built, Not Invited: The Story Behind LeadDev Athens

How LeadDev Athens Was Born…

Read more

a year ago · 3 likes · 3 comments · Christos Chatzis

‘Built, Not Invited: The Story Behind LeadDev Athens’ is a reflection on how real communities and initiatives are earned not handed. Christos Chatzis explains that what began modestly as a blog called git push -f leadership evolved through steady effort, shared belief, and active participation into LeadDev Athens. What matters, the post argues, isn’t being formally invited or given permission but rolling up sleeves, contributing with consistency and building momentum together.

Value from Data & AI

You're not supposed to be 'The boss'

“You own the data strategy.” “You’re responsible for driving data adoption.” “It’s your job to make this organisation data-driven…

Read more

a year ago · 3 likes · Nick Zervoudis

Value from Data & AI — ‘You’re Not Supposed to Be ‘The Boss’ argues that leading data or AI efforts successfully isn’t about asserting authority it’s about enabling influence, collaboration and delivering measurable value. warns against the ‘that’s-not-my-job’ attitude, urging data leaders to avoid siloed thinking and instead take ownership of both results and accountability. It calls out common pitfalls like building dashboards or analytics tools without connecting them to real business outcomes, practices that often leave data projects disconnected from actual value creation.

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

It’s been a while and I’ve missed you! I want to start with a warm welcome back. You might have noticed the long pause since my last post. Life took a busy turn as I relocated from Thessaloniki to Athens and stepped into a new role, which demanded my full focus for a while. Thank you for your patience during this. Now, I’m excited to resume our journey in data analytics conversations. In fact, we’re coming back strong with a regular schedule…you can expect new DataConscious posts every Wednesday or bi-weekly moving forward.

Let me reintroduce what DataConscious is all about. This publication (and the accompanying podcast) centers on mindful, impactful analytics practices. We cut through the noise and focus on what truly matters: driving business impact through better decisions. With that mission in mind, let’s get to today’s topic to bridge the gap between theory and job market.

Today’s post (and podcast episode) takes a short break from our ongoing Data Career Compass series. I want to shine a spotlight on something very close to my heart: the course Data Analytics in Modern Corporate Business, which I have the pleasure of creating for the International Hellenic University (IHU) and professor Dimitrios Varsamis.

Why now? Well, the second cycle of this course is kicking off on October 1, and applications are open. It’s the perfect moment to reflect on what the course offers and share stories from those who’ve been through it.

To do that, I’ve invited a special guest Lampros Kouremenos, Financial Analyst at Ikos Resorts, an alumnus of the first cycle to join me on the podcast and in this write-up. Lampros was a standout participant and I’m thrilled to share his journey from the classroom to real world data challenges.

🎙 His path into data analytics: From a finance background in Greece and the Netherlands, Lampros saw firsthand how data and automation transformed finance teams. His curiosity grew during his time at a fintech company, where he realized the power of data to streamline processes.

🎙 Why this course: Having tried isolated courses before, he was drawn to Data Analytics in Modern Corporate Business because it offered the whole chain of data from databases to visualization in a structured way, tailored for professionals not yet fluent in data.

🎙 Bridging theory to practice: Lampros highlighted how self-learning often left him lost in fragmented resources and jargon. The course helped connect the dots, teaching him how SQL, Python, Git, BigQuery, and visualization tools fit together in real-world pipelines.

🎙 Capstone experience: Building an end-to-end pipeline — PostgreSQL → Datastream → BigQuery → Python ETL → Tableau/Metabase was challenging but eye-opening. He described the pride of seeing his dashboards live as “like a baby is born.”

🎙 Applying it at work: Back in his finance role, he advocated for creating a data warehouse, proposed staging tables to detect anomalies, and began introducing more automation and BI reporting practices. The course gave him both the vision and confidence to push these ideas forward.

🎙 Advice for newcomers: Don’t pursue analytics just because others do pursue it because you enjoy it. Start with fundamentals (data types, databases, SQL), then move to warehousing, automation, and BI. Build a portfolio of projects, stay curious, and use AI as a complement not a replacement to understanding the fundamentals.

🎙 Where the field is going: Lampros believes AI will accelerate workflows. To truly add value, professionals must understand fundamentals and business needs areas where AI still struggles.

Bridging Theory and Practice: Inside the Course

Let’s start with a look at what the Data Analytics in Modern Corporate Business course is all about. I designed this program to help bridge the gap between academic theory and real-world practice in data analytics. Many aspiring analysts learn the fundamentals through university or online resources, but they often struggle with translating that knowledge into actual business impact. This course tackles that challenge head-on by providing essential skills and hands-on experience in a corporate data setting.

What does the course cover? In short, it walks participants through the modern data analytics stack from end to end. We start with the basics and progressively delve into the tools and techniques that data teams use in today’s companies. The curriculum is organized into five key modules, and along the way students engage in a realistic project that simulates a corporate environment. Here’s a quick rundown of the core components:

1. Data Analytics Fundamentals & Roles: Understanding the analytics landscape and the various roles (analysts, engineers, BI developers, etc.) sets the stage for everything. We discuss how data fits into business strategy and the common pain points in organizations so you know why you’re doing what you’re doing.

2. Working with Databases & SQL: Every analyst needs to speak SQL. We use PostgreSQL to teach relational database concepts and querying skills, so participants learn to pull insights from transactional data. This covers the core work of accessing and preparing data.

3. Version Control with Git: Modern data work is collaborative and code-driven. We introduce Git for version control and project collaboration, ensuring students adopt best practices in managing code and data pipelines. This is a key industry skill often missing in traditional coursework.

4. Data Warehousing with BigQuery: Handling large-scale data is a must in corporate settings. We cover data warehouse design and use Google BigQuery for scalable cloud analytics. Participants learn how to replicate and aggregate data in a warehouse and why data warehousing matters for business intelligence.

5. Data Visualization & BI Tools: Communicating data findings is half the battle. The course features Tableau Cloud and Metabase for dashboarding and visualization, teaching students to turn analyses into interactive, impactful visuals for decision-makers.

✅ 100% online & self-paced, study at your own rhythm

🎥 15+ hours of videos, walkthroughs & exercises

📦 Build your own data pipeline from PostgreSQL to BigQuery

📊 Create real dashboards in Tableau & Metabase

🎓 Receive a certificate of completion + 3.5 ECTS credits

💡 Designed for students, aspiring analysts and professionals who want to
bridge the gap between university theory and real corporate practice.

🌍 All content is in English and open to learners worldwide.

Capstone Project | Real Corporate Simulation

All the above skills form a hands-on capstone project that simulates a real-world business scenario. Participants are given a scenario: a company’s transactional data in PostgreSQL must be continuously replicated to BigQuery (using Google’s Datastream), then integrated and analyzed; code and configuration are managed in Git, and finally insights are delivered through Tableau and Metabase dashboards.

Each student builds this end-to-end solution, essentially experiencing the lifecycle of a data project in a modern corporate environment.

By covering this breadth of topics, the course aims to build confidence and real competence for those stepping into data roles. It’s not just theory; it’s doing the work. For example, when students configure a data pipeline from scratch in our capstone, they often have an “aha” moment – the fog of jargon and abstract concepts lifts, and they can see how things connect in practice. This comprehensive approach ensures that graduates have both the knowledge and the experience to contribute to data projects at work.

Once again, thank you for reading and for being part of the DataConscious community. It feels great to be writing to you again after the break.

I’m looking forward to sharing more insights, interviews and practical frameworks every Wednesday. If Lampros’s journey resonated with you, feel free to share this post with someone who might find it useful or drop a comment with your thoughts. Until next time – stay curious, stay conscious with your data!

Interesting Posts

git push -f leadership

LeadDev London (LDX3) 2025

It’s no secret that I’m a big fan of LeadDev in all its forms—for the incredible work it does for the engineering leadership community. From blogs and webinars to podcasts and conferences, LeadDev stands out as the premier brand that focuses on leadership in engineering, covering both technical and managerial tracks. This focus is especially important b…

Read more

a year ago · 4 likes · Christos Chatzis

Christos Chatzis shares his experience at LeadDev London (LDX3) 2025. It brought together over 2,000 engineering leaders in a festival-style event combining LeadDev, StaffPlus and LeadingEng into one multi-track experience. With 150+ sessions and speakers from companies like Salesforce, ThoughtWorks, and Softwire, the conference covered topics from AI-assisted development to building resilient teams and aligning engineering with business goals. Senior leaders also benefited from the exclusive DirectorPlus Pass for workshops and networking. LDX3 stood out as a premier gathering for engineering leadership, offering both strategic insights and strong community connections.

People In Software

After the Calendar Invite: Navigating Tech Layoffs with Resilience and Humanity

The notification arrives at 2:47 PM on a Friday. The calendar invite appears awkwardly formal in an otherwise casual culture. You already know what's coming. Maybe you've seen it happen to others. This time, it's your turn…

Read more

a year ago · 4 likes · George Dedes

George Dedes describes that tech layoffs are rarely about individual performance. They’re systemic, driven by shifting markets, cost-cutting and the rise of AI. This piece emphasizes the human side of layoffs: the shock, but also the strange gift of time for reflection and recalibration. It offers practical advice, financial preparation, continuous learning, networking, and empathetic leadership while reminding us that resilience and community are what truly carry us through uncertain times.

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Before we dive in, a quick note and a small apology. I know it’s been about a month since my last post. Life got busy and I hit pause to focus on something I felt truly needed building. That something is what this post is all about. Of course, Data Career Compass is to be continued and there will be five more post about data roles. So this Wednesday (and hopefully every Wednesday…) we are back on track with a new post.

I never planned to build my own app. In fact, for the longest time, I managed my mentoring with the usual patchwork of tools: Google Drive folders, email threads, random bookmarks, and the occasional spreadsheet. Every time I started with a new mentee, I found myself repeating the same setup. I’d create yet another Google Drive folder, collect links and docs from previous mentees, rewrite the same introductory notes and still have no real way to track how they were progressing. It got tiring, not the people, but the process. At some point, I had to ask myself:

“Why is this still so manual?“

So, I decided to change it. One weekend, I started vibe-coding with AI, to identify how hard/easy it was to create such a web app and deploy it without pain. And boom, I realised that I’ll have to dedicate some serious effort and pay some money, significantly less than building it without AI.

Finally, it’s quite easy for everyone and I managed to streamline my mentoring workflow. That little side project eventually turned into MentorHub. I built it for myself, to solve the exact pain points that had been frustrating me the last two years. MentorHub became the one place where I could do everything: build custom learning paths for each mentee, keep all our resources organized in one hub, track progress without fiddling with spreadsheets and actually see how someone is growing over time. In short, it took all those “manual” tasks off my plate so I could focus on what I love about mentoring…helping people grow.

What started as a personal project soon proved useful beyond just my own practice. I initially hadn’t planned on releasing MentorHub to the public. But after sharing the idea with a few fellow mentors, I realized I wasn’t alone in these struggles. The enthusiastic responses (“Where can I get this?”) made it clear that MentorHub addresses common mentoring headaches. So, at this point (and not forever) I’m now offering it to others for free so any mentor (formal or informal) can use it to simplify their workflow.

In this post, I want to give you a walkthrough of MentorHub: how it works, what problems it solves, and why it might just change the way you mentor.

A Quick Overview: What is MentorHub?

MentorHub is a web-based platform that brings all the pieces of the mentoring puzzle into one place. It’s designed as a collaboration hub where mentors create personalized learning content and mentees access structured guidance. In practice, that means MentorHub serves both sides of the mentoring relationship:

• Mentors can design a custom curriculum for each mentee, assign resources and tasks, schedule and log sessions (with notes on payments) and monitor progress at a glance.

• Mentees get a clear learning path laid out for them, a single portal for all the materials and tasks their mentor has assigned and a way to track their own progress through the curriculum.

Everything is organized by roles and topics. Think of it like having a template learning path for each type of mentee you mentor. For example, if you often mentor Data Analyst trainees, you can set up a “Data Analyst Learning Path” once and reuse it, instead of reinventing the wheel for each new person. MentorHub wasn’t built as a commercial product with all the flash. It was built by a mentor, for mentors, with a focus on solving real problems in a straightforward way. Let me take you through its key features. Of course there are bugs and stuff that needs to be improved.

The MentorHub Dashboard: All-in-One Mentoring Snapshot

MentorHub’s Dashboard gives mentors a bird’s-eye view of all mentoring activity, from active mentee counts to upcoming sessions (as shown above).

The first thing you see when you log into MentorHub is the Dashboard. I designed the dashboard to answer the question, “What’s going on with my mentees this week?” at a single glance. At the top, it highlights the basics: how many active mentees I have and the total number of sessions we’ve logged so far (a satisfying reminder of the impact over time). A "Session Activity" chart shows the flow of mentoring sessions over the past weeks, for example, I can see if I’ve been more active this month than last.

Just below the stats and chart, MentorHub lists Upcoming Sessions with each mentee. This has been a game changer for me: no more digging through my calendar or email threads to remember when the next call is. Each upcoming session is right there with the date, time, and topic and there’s even a one-click link to join the video meeting when it’s time. Next to each session, MentorHub also flags whether the session is marked as “Paid” or “Unpaid”, a subtle reminder for me if I need to follow up on a payment later.

On the dashboard I’ve also included a Recent Assignments feed (visible in the screenshot’s sidebar). This feed shows the latest learning resources or tasks I’ve assigned out. It’s a running log of content I’ve been adding for my mentees. In practice, this helps me recall “Oh right, I gave Alice that SQL tutorial last week”, useful context right before a session with her, or when planning what to assign next. The dashboard, in essence, centralizes the heartbeat of my mentoring practice: sessions completed and coming up, and content being worked on. It’s the hub that ties everything together.

Role-Based Learning Paths: Reusable, Structured Content

One of the aha moments that led to MentorHub was realizing how much of my mentoring content was reusable. Why was I manually compiling the same resources for every new “Junior Data Analyst” I mentored? MentorHub tackles this with role-based learning paths. In the app’s administration section, I can create a Role (say, “Data Analyst - Junior”) and define a structured curriculum for that role, broken down into Topics. Each topic is like a module or chapter in the learning journey and under each topic I can list specific Resources (articles, videos, PDFs, or any learning material).

For example, for the Data Analyst - Junior role, I might have topics like “Intro to Data Teams and Roles,” “Intro to SQL,” “Data Visualization Basics,” etc. Under “Intro to Data Teams and Roles,” I can attach resources such as a blog post on data team structures, an infographic on different data roles, or a PDF overview of agile methodologies in data. Building these content libraries takes a bit of upfront work, but it pays off immensely. The next time I onboard a mentee who wants to become a data analyst, I don’t have to start from scratch —> I simply assign them the pre-built learning path for that role and adjust as needed.

These learning paths are flexible and customizable. MentorHub lets me add or remove resources on the fly, or even tweak the sequence for a particular person. The content library lives in one place, neatly organized, so I “keep everything in one place” as I intended. No more hunting through old Google Drive folders or emails to find that one article I know I used before. By structuring content by roles and topics, MentorHub not only saves me time, it also ensures consistency, every mentee with the same goal gets the benefit of the best materials I’ve curated over time.

Assignments and Progress Tracking: No More Spreadsheets

Perhaps my favorite aspect of MentorHub (and one that my mentees appreciate too) is how it handles assignments and progress tracking. Once I’ve assigned a learning path or selected topics for a mentee, MentorHub generates a tailored list of tasks for them. On my side, I can see a Mentee Progress Dashboard for each person, which shows at a glance how far along they are. For each mentee, MentorHub displays key stats like the number of topics assigned, how many are completed, the total resources (e.g. articles, videos) I’ve given them and how many of those they've finished. It even boils this down into an overall completion percentage for that mentee’s curriculum.

When I click on a mentee’s profile in MentorHub, I get a detailed breakdown of their progress. All the topics we planned are listed, each with an expandable list of the resources inside. I can see checkmarks or status indicators next to each item. For example, it might show that out of 8 resources under “Intro to Databases,” 5 are completed and 3 are pending. This means I no longer need to maintain a separate spreadsheet or checklist to track what my mentee has done. Both the mentor and the mentee can update the status of an assignment (so if the mentee finishes reading an article, they can mark it done). The next time I prepare for a session, I can quickly review what they’ve completed and where they might be stuck.

All this tracking is not noticeable but incredibly useful. It gives transparency to the mentee’s journey: I can actually see how someone is growing over time. If I notice a mentee hasn’t completed the last few assignments, that’s a signal to check in: Was the material too challenging? Are they short on time this week? If someone completes many resources, I know they’re hungry and can offer them new challenges faster.

In the past, I often only discovered progress issues during sessions (or when a mentee went silent). With MentorHub’s progress dashboard, there are fewer surprises. It keeps both of us accountable and aligned, and it does so without me having to juggle external spreadsheets.

Scheduling Sessions and Tracking Payments Made Easy

Another area MentorHub streamlines is the scheduling of sessions. Before, coordinating a session meant an endless back-and-forth of messages: “Are you free Wednesday at 6?”, sending calendar invites, remembering to send a Zoom or Meet link, and so on. In MentorHub, I can schedule a session in a few clicks right within the app. I pick the mentee, set the date/time, add a brief description or agenda for what we’ll cover, and that’s it. The session then appears on both my dashboard and the mentee’s view as an upcoming event. When the time comes, we simply click “Join Meet” to hop on the call (I use Google Meet links by default, but any meeting URL can be plugged in). This has drastically reduced the friction in getting our sessions on the calendar.

MentorHub also keeps a log of all past sessions, which is useful for looking back on what we’ve covered and when. Each session entry can hold notes or a recap (so I can see what we did, or what to follow up on next time). And importantly for those of us who mentor professionally, the app has a simple payment tracking mechanism. When I record a session, I can mark whether it’s been paid or is still unpaid. On the sessions list (and on the dashboard upcoming list), I’ll see a label next to each session indicating its payment status (e.g. Unpaid for a future session that the mentee hasn’t paid for yet, or Paid for completed sessions that have been settled). This gentle reminder system means no more awkwardly forgetting if someone has outstanding session fees. It keeps my mentoring sideline feeling professional and organized without me keeping a separate ledger.

*Payments are getting filled manually, Stripe integration is not completed yet

Finally, the Mentees section in MentorHub ties together a lot of the above features. Here I see a roster of everyone I’m mentoring, each with a quick summary: their assigned role/learning path, how many sessions we’ve had, their current progress percentage and even a note on their session rate or plan. From this view, I can jump into a specific mentee’s detailed progress dashboard or quickly schedule a new session for them. The mentee cards give me an immediate sense of who might need attention for instance, if I see one mentee at 10% progress and another at 80%, I know who might need a nudge or an extra check-in this week.

🛎️ Notifications: Stay in the Loop, Effortlessly

MentorHub’s notification center functions as a central alert system for key mentoring activities. When a mentee completes an assignment say, “Fundamentals of Operating Systems I”, the platform sends you a notification. This small feature carries a big impact: it allows mentors to acknowledge learning progress as it happens, enabling timely feedback, encouragement, or follow-up questions.

Whether you're managing one mentee or ten, this system ensures you're always informed without being overwhelmed. It's another way MentorHub helps reduce cognitive load and keeps you meaningfully connected with your mentees' progress.

By the mentee’s side, they are always up to date if new content (resources) have been added to an existing Topic. (e.g SQL Course (resource) to Introduction to SQL (topic).

Conclusion: Mentoring, Upgraded (and How to Try MentorHub)

Building MentorHub has fundamentally improved my mentoring experience. It started as a personal fix for a repetitive process, but it grew into a platform that makes mentoring more enjoyable and effective for me and my mentees. The app consolidates all the disparate parts of mentoring into one coherent workflow: from content curation and assignment, to tracking and feedback, to scheduling and administration. Instead of spending my time on admin work, I now spend it where it counts, engaging with my mentees.

What’s most exciting to me is seeing other mentors take MentorHub and run with it. I initially built it for myself and wasn’t even sure if I’d share it widely.

But the response from the community convinced me to open it up. If you’re a mentor (or considering becoming one) and any of the frustrations I described sound familiar, I invite you to give MentorHub a try. It’s (still) free to use and I genuinely hope it can make your life easier. You can access the platform here:

→→→→→→ https://mentorhub.replit.app ←←←←←←

I’m continuing to refine the tool (it’s a labor of love in progress) and I’d love to hear your feedback. Feel free to reply to this post or reach out to me directly if you’re interested or have questions.

I’ll give you free lifetime access forever at this point.

Mentoring is a rewarding journey and with the right tools, it can be even more impactful for both you and your mentees.

Happy mentoring, and I hope MentorHub can help you as much as it’s helped me!

I would really love to hear your feedback.

Interesting Posts

The Last Dev

Using AWS Glue Jobs with Terraform

The road so far… *Carry on, my wayward son is playing in my mind*. In a previous post, we saw Amazon Athena and how to query data, such as CUR reports. Amazon Athena is a fantastic tool provided by AWS, a serverless managed Presto that allows you to query your unstructured and semi-structured data at a very low cost. In this post, we will continue our j…

Read more

a year ago · 1 like · Konstantinos Siaterlis

Konstantinos Siaterlis offers a concise, practical guide on how to use Terraform to provision and manage AWS Glue Jobs, including IAM roles, Glue connections, and job scripts. It walks through the infrastructure-as-code approach to deploying ETL pipelines on AWS. Ideal for data engineers looking to automate their Glue workflows.

git push -f leadership

Digital Archaeology for Engineering Leaders 🧱

How to Run Tech Due Diligence When You Land in a Team You Don’t Know ⚠️…

Read more

a year ago · 2 likes · 2 comments · Christos Chatzis

Christos Chatzis explores the concept of Digital Archaeology in engineering teams, uncovering past decisions, hidden context and legacy patterns buried in codebases and tools. It highlights why understanding historical technical choices is crucial for effective leadership and decision-making in modern engineering environments.

People In Software

The Art of Repair: Kintsugi and Growing Stronger Through Challenges

In Japan, artisans don't discard or hide the damage when precious pottery breaks. Instead, they practice kintsugi, carefully mending the pieces with gold-infused lacquer, transforming fractures into beautiful golden seams. The repaired piece emerges more valuable and unique than before, with its repair history visible and celebrated…

Read more

a year ago · 1 like · George Dedes

George Dedes draws inspiration from Kintsugi, the Japanese art of repairing broken pottery with gold, to reflect on personal and professional growth in software careers. It explores how embracing imperfection, setbacks and emotional resilience can lead to deeper strength, authenticity and meaningful development in tech.

Thanks for reading until here. I know this one was a bit big :)

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

In this blogpost we’ll explore the role of Analytics Engineer. Our podcast guest is a former colleague of mine, Panagiotis Apostolopoulos, Analytics Engineer at Hack The Box, a great guy and professional and I can admit that I cannot be biased with him because we are connected in many ways, from bugs, sql models and dirty infra work to conferences around the world. Thanks Panos for supporting DataConscious and let’s return to the role!

The Analytics Engineer is a new role at the intersection of data engineering and data analytics. It has emerged as a response to modern data workflow challenges and opportunities and it continues to evolve. At its core, the analytics engineer is responsible for ensuring the quality, reliability and usability of data across an organization. The role sits at the heart of the operation, connecting raw data infrastructure to trusted analytics ready data that business users can confidently explore.

In this episode of DataConscious – A Mindful Approach to Analytics, we explore the ever-evolving role of the Analytics Engineer. Our guest, Panos, brings clarity to this hybrid position that lives between data engineering and data analytics, offering a practical view on what it really means to enable data driven decision making from within.

🎙 What exactly is an Analytics Engineer? From owning data transformation pipelines to bridging business needs with technical solutions.

🎙 His path into the role: Shares his transition into the role.

🎙 A day in the life: You’ll hear how he manages dbt projects, curates business logic, ensures data quality and collaborates closely with both business stakeholders and data engineers.

🎙 Skills that stand out: From SQL and data modeling to the art of documentation and stakeholder communication, he reveals which capabilities have made the biggest difference in his effectiveness.

🎙 Where the field is going: As the role matures, we discuss the shifting expectations, the impact of modern data stacks and how analytics engineers are becoming cornerstone members of data teams.

🎙 Advice for newcomers: Panos offers insights on learning SQL deeply, embracing version control, thinking like a software engineer, and finding joy in iteration and testing.

Whether you're curious about this emerging role or aiming to pivot into it, this episode is packed with grounded insights and actionable advice. The Analytics Engineer isn't just a technical translator, it’s someone who ensures the business sees the data clearly, consistently and confidently.

Whether you're a data leader, engineer, BI professional or decision-maker, this episode will help you navigate the evolving landscape of modern data architectures.

🎧 Listen now & subscribe for more insights on mindful data practices!

(Please find the 2025 State of Analytics Engineering Report which Panos mentions in the podcast episode at the end of the post)

What Does an Analytics Engineer Do?

Analytics engineers wear many hats in a data team. Some of their core day-to-day responsibilities include:

• 🏗️ Building & Transforming Data Models: Analytics engineers spend a lot of time writing SQL queries to transform raw data into structured tables. They take messy, siloed data from various sources and model it into clean, unified datasets. This often means applying business rules (e.g. renaming fields, cleaning formats, calculating new metrics) so that the data makes sense for analysis. The goal is to create a single source of truth that analysts and business folks can trust instead of everyone fetching raw data.

• 🧪 Ensuring Data Quality (Testing): Because they treat data like a product, analytics engineers set up tests and monitoring to ensure data is accurate and reliable. For example, they might write tests to catch if yesterday’s sales data suddenly dropped to zero or if there are duplicate entries where there shouldn’t be. Using tools like dbt, they can build tests right into the transformation process (think of it like unit tests for data 🙌). If something looks off, they investigate and fix it before it misleads someone.

• 🚦 Monitoring Data Pipelines: Analytics engineers also keep an eye on data pipelines and workflows. They make sure all the automated jobs that move and transform data (often running overnight) have completed successfully. If a pipeline fails or data is delayed, the analytics engineer gets an alert and jumps in to troubleshoot. In this way, they ensure data is flowing from point A to point B on time and nothing critical breaks without them knowing.

• 🔧 Optimizing & Refactoring: With a foot in software engineering, an analytics engineer continuously improves the code and queries that shape the data. They review each other’s SQL code, refactor messy or slow-performing queries, and tune the data models for efficiency. This might mean rewriting a heavy query to run faster, or reorganizing a data model for easier use. Over time, these tweaks keep the data playground running smoothly and scalable as data grows.

• 📝 Documenting & Defining Data: Analytics engineers often act as data librarians for the organization. They document what each data model or table means, define business terms (e.g. what exactly counts as an “active user”?) and make sure everyone can find and understand the datasets available. This might involve writing clear descriptions in a data catalog or within their dbt project so that anyone (from a new analyst to the CEO) can grasp what the data represents.

🛠 Skills - Tools - Key Responsibilities

To accomplish all of the above, analytics engineers rely on a mix of modern tools and software best practices. Here are some of the key ones:

• SQL: SQL is the alphabet for an analytics engineer. It’s the main language they use to query and reshape data in the warehouse. Strong SQL skills are a must, as they’ll be writing complex joins, aggregations and data cleaning logic every day.

• dbt (Data Build Tool): dbt has become the go to tool for analytics engineering. It allows them to manage all those SQL transformations as modular code. With dbt they organize transformations into models (SQL files) and can run them in sequence ensuring dependencies are handled (e.g. build raw data into staging tables then into intermediate tables then final reports). dbt also enables writing tests, documenting models and even creating a lineage graph so you can see how data flows from raw sources to final analytics. It’s like an analytics engineer’s command center for building a robust analytics layer on top of the raw data.

• Version Control (Git) and Code Review: Just like software engineers, analytics engineers use Git or other version control to collaborate on code. All the SQL scripts and configuration (often in a dbt project) live in a repository. They create branches to add new features or fix issues and teammates review each other’s code changes. This process helps catch mistakes early and maintain a high quality for the data transformations (no more “quick and dirty script” living on someone’s laptop — everything is tracked 👍).

• Orchestration & Scheduling: Tools like Apache Airflow, dbt Cloud or other workflow orchestrators are used to schedule and run data pipelines. Analytics engineers set up these schedules so that, say, every morning the latest data is loaded and transformed by 7am. They also handle dependency management and alerts if a data source didn’t refresh or a transformation fails, the orchestrator will alert the team. Essentially, they make sure the whole system runs perfectly.

• Data Warehousing & Cloud Platforms: Analytics engineers work closely with data warehouses (like Snowflake, BigQuery, or Redshift) where all the data lives. They need to be familiar with how to optimize queries for these platforms and use their features (partitioning, indexing, etc.) effectively. They also often leverage cloud services for storage and processing. While deep infrastructure might be more the data engineer’s realm, analytics engineers should know enough to make the most of their warehouse and not, say, run a query that costs a fortune or slows everything down.

• Data Quality & Observability Tools: Beyond just writing tests in dbt, many analytics engineers implement additional tools to monitor data quality and pipeline health. This can include packages like dbt-expectations (for extended testing) or standalone tools like Great Expectations and Monte Carlo for data observability. These help catch anomalies or bad data early. Some teams also track metrics like how fresh the data is, or if any dashboards are broken, to proactively maintain trust in data. The following chart of 2025 State of Analytics Engineering Report illustrates it better.

How to proactively grow into an Analytics Engineer role?

So how can you become an analytics engineer, or sharpen the skills if you’re aiming for this role? Here are a few tips to get started:

• 👉 Master SQL fundamentals: SQL is your best friend. Practice writing queries across different scenarios – joining tables, filtering, aggregating, etc. You can start with any sample database (like PostgreSQL or even an online SQL tutorial) to get comfortable. The better you get at SQL the easier everything else becomes.

• 👉 Learn a transformation tool like dbt: Since dbt is so central to analytics engineering today, it’s worth getting hands-on experience with it. You can take a public dataset (for example, COVID-19 data or Spotify song data) and try building a mini dbt project. This will teach you how to structure models in layers (staging → intermediate → reporting), how to write tests, and how to use Jinja (dbt’s templating) for DRY (Don’t Repeat Yourself) code. There are many community tutorials and even a dbt online course for beginners – take advantage of those.

• 👉 Develop your data modeling mindset: Read up on the basics of data modeling (things like star schema vs. 3NF, fact and dimension tables) and practice by designing schemas for problems you’re interested in. Think about how raw data should be organized to answer real business questions. For example, if you love sports, imagine how you’d model player and game stats for analysis. Modeling is part art, part science —> the more you do it, the better you get.

• 👉 Embrace testing and data quality checks: Start treating your data work like software development. Learn about data quality tools and why they matter. This mindset of “trust but verify” will set you apart as someone who delivers reliable results, not just any results.

• 👉 Build a portfolio project: Nothing beats real-world practice. Create a project where you take raw data, load it (you could use an open-source pipeline tool or even manual CSV import), transform it with dbt or SQL, and then analyze it or build a small dashboard. Focus on showcasing your thinking: document the problem you tackled, how you modeled the data, how you ensured quality and what insights came out. Even if it’s a simple project, this is a great way to demonstrate to potential employers that you understand end-to-end analytics engineering. Plus, you’ll learn a ton by doing it.

• 👉 Sharpen soft skills (communication & business understanding): An often overlooked aspect of this role is the ability to communicate and understand context. Practice explaining your data findings or project decisions in plain language. This could be through blogging, or just talking to non-technical friends about your project. Also, train yourself to always ask “what is the business need here?” whenever you tackle a data task. The more you tune into the actual problem to solve (e.g. increasing user engagement, reducing costs, etc.) the more effective your solutions will be.

• 👉 Connect with the data community & keep learning: The analytics engineering space is evolving, so stay curious! Join communities like the dbt Slack or relevant forums/subreddits, follow data blogs or a Substack (😉), attend virtual meetups or local data events. Learning from others can spark ideas and keep you up-to-date on new tools and best practices. And don’t be afraid to ask questions or even contribute to open-source projects. Being active in the community can accelerate your growth and also connect you with mentors or job opportunities.

Remember, becoming a great analytics engineer is a journey. It blends technical stuff with a keen understanding of people and business. So keep experimenting, keep learning, and don’t hesitate to think outside the box. Every data problem you solve (or even attempt) is a step forward. Good luck on your analytics engineering adventure! 🚀

Maggie explains it also quite well…

📈 Career growth opportunities and salary expectations

Analytics engineers command impressive salaries, reflecting their pivotal role in bridging data engineering and analytics.

• Entry-Level Positions: Starting salaries range from $110,000 to $117,000 annually.

• Senior-Level Roles: Salaries can escalate to $190,188 per year, with top-tier professionals earning up to $627,000, particularly in high-demand markets or leading tech companies.

• Notable Example: At Netflix, analytics engineers have reported median salaries of $197,371, with compensation packages reaching up to $295,194.

These figures underscore the lucrative nature of the analytics engineering profession, especially for those who continuously enhance their skills and adapt to emerging technologies.

The role of an analytics engineer is not only financially rewarding but also offers diverse pathways for professional advancement. Professionals in this field can progress into roles such as:

• Data Architect: Designing and managing complex data systems.

• Analytics Manager: Leading teams to derive actionable insights from data.

• Product Data Lead: Overseeing data strategies for product development.

• Chief Data Officer (CDO): Setting the data vision and strategy at the organizational level.

The demand for analytics engineers spans various sectors including technology, finance, healthcare, and e-commerce. The projected growth rate for this profession is 18% over the next decade highlighting its critical importance in data-driven organizations.

In a nutshell…

Interesting Posts

The Last Dev

The Last Dev ~ Cloud Updates #8

Alongside the format, I am also changing the name of this series. The idea is to express my personal opinion on some updates that I found interesting, and showcase some local events and articles…

Read more

a year ago · 3 likes · 2 comments · Konstantinos Siaterlis

Konstantinos Siaterlis highlights significant advancements in Amazon's Q Developer tools, including enhanced CLI support for Model Customization Parameters (MCP) and an agentic coding experience within IDEs like VS Code and JetBrains enabling more intuitive and automated development workflows.

git push -f leadership

Don’t Just Build Great Software—Build a Reputation

In today’s competitive and connected tech world, building great products is only part of the equation. To truly thrive, companies must also focus on building visibility, credibility, and community presence. This is where promoting your company’s extroversion becomes essential. 🌍…

Read more

a year ago · 3 likes · 4 comments · Christos Chatzis

Christos Chatzis argues that great code isn’t enough, you need great people working well together. It highlights how trust and open communication are essential for teams to thrive. Ultimately, it's a reminder that strong software comes from strong relationships, not just strong technical skills.

Thanks for reading until here. I know this one was a bit big :)

References

https://www.getdbt.com/resources/state-of-analytics-engineering-2025

https://www.talent.com/salary?job=and+analytics+engineer

https://www.businessinsider.com/how-much-netflix-pays-employees-us-salary-levels

https://www.datacamp.com/blog/what-is-an-analytics-engineer-everything-you-need-to-know

https://discourse.getdbt.com/t/snowflake-dbt-how-to-use-different-role-in-model-view/8156

https://www.reddit.com/r/dataengineering/comments/1fvtlq9/what_is_analytical_engineer/

https://www.coursera.org/articles/analytics-engineer

https://www.getdbt.com/blog/what-is-analytics-engineering

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Imagine you have a huge digital warehouse filled with tons of boxes (your data).
At first, everything looks manageable.
You can pick up a box, see what's inside, and store it back.

But over time, the warehouse becomes chaotic:

• Boxes get bigger and heavier (more data).

• Some boxes change shape (schema changes).

• You don’t know which version of a box you’re looking at (no time travel).

• If you try moving things around, you might accidentally damage other boxes (no data consistency).

• And when too many people try to move things at the same time, everything becomes a mess (no transactions).

Regular data lakes basic storage like S3, GCS, or Azure Blob are like that warehouse with no rules, no labels, and no managers.

🔵 This is where Apache Iceberg and Delta Lake come in.

They are like adding a smart warehouse management system on top of your messy storage:

• 📦 They keep track of every change made to your data (like a version history).

• 🛡️ They guarantee safety when multiple people are reading and writing data at the same time (ACID transactions).

• 🕰️ They allow you to travel back in time to earlier versions of your data — just like looking at an older version of a document.

• 🔄 They handle changes gracefully, like updating the structure of your data without breaking everything.

If you’re building or managing a data lakehouse today, chances are you’ve already heard the names Delta Lake and Apache Iceberg being thrown around.

At first glance, they seem to promise the same things:
👉 ACID transactions on your data lake
👉 Schema evolution
👉 Time travel
👉 Reliable performance at massive scale

But once you dive in, you realize they're actually quite different and choosing the right one depends a lot on your setup, your team and your long term vision.

Let's walk through a simple comparison, not the marketing slides but what you actually need to know.

🧱 Their Origins (and Why It Matters)

• Delta Lake was born inside Databricks and later open sourced (though still heavily driven by Databricks’ vision).
  Think: best friends with Apache Spark, deeply optimized for it.

• Apache Iceberg started at Netflix, faced with petabyte-scale challenges.
  Later donated to the Apache Software Foundation, it now thrives under a truly open, engine agnostic philosophy.

🔵 Delta Lake: Spark-first, Databricks-powered
🧊 Iceberg: Multi-engine, community-led

What is Delta Lake?

Delta Lake, initially developed by Databricks and now an open source project under the Linux Foundation, was designed to add reliability and performance layers on top of existing data lakes, primarily using the Parquet file format. Its core strength lies in bringing ACID (Atomicity, Consistency, Isolation, Durability) transactions to data lake operations, ensuring data integrity even with concurrent reads and writes. Delta Lake achieves this through a transaction log (_delta_log) that records every change made to the table. Key features include seamless integration with Apache Spark, efficient data versioning enabling time travel (querying previous states of your data) and optimizations like data compaction.

If your organization is heavily invested in the Databricks or Apache Spark ecosystem, Delta Lake often presents a natural and tightly integrated choice. It excels in scenarios requiring robust consistency for write heavy workloads and real-time streaming analytics.

# --- Setup Spark with Delta (Needs Delta installed!) ---
import pyspark
from delta import *

builder = pyspark.sql.SparkSession.builder.appName("DeltaLakeDemo") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = configure_spark_with_delta_pip(builder).getOrCreate()

# --- Create a simple Delta table --- 
data = spark.range(0, 5)
data.write.format("delta").mode("overwrite").save("/tmp/my_delta_table")

# --- Read the table --- 
df_read = spark.read.format("delta").load("/tmp/my_delta_table")
df_read.show()
# +---+
# | id|
# +---+
# |  0|
# | ...|
# |  4|
# +---+

# --- Time Travel! Read the first version (if you overwrite later) ---
# df_v0 = spark.read.format("delta").option("versionAsOf", 0).load("/tmp/my_delta_table")
# df_v0.show()

What is Apache Iceberg?

Apache Iceberg originated at Netflix to address the challenges of managing massive, rapidly changing datasets in their data lake. Now governed by the Apache Software Foundation, Iceberg is designed as a truly open table format, emphasizing compatibility across various compute engines like Spark, Trino (formerly PrestoSQL), Flink, and others. Unlike Delta Lake's transaction log, Iceberg manages table state through metadata stored in manifest files, tracking individual data files within snapshots. This architecture supports features like full schema evolution (adding, dropping, renaming columns without rewriting data files), flexible partition evolution, and support for multiple file formats (Parquet, ORC, Avro). Iceberg also provides ACID guarantees and time travel capabilities.

Iceberg's strength lies in its openness and flexibility, making it a compelling option for organizations seeking to avoid vendor lock-in or needing to integrate with a diverse set of processing engines. Its robust schema and partition evolution capabilities are particularly beneficial for managing long-term datasets where structures might change over time.

-- --- Assuming you have a catalog 'demo' configured for Iceberg --- 

-- --- Create a table --- 
create table demo.my_iceberg_table (
  id bigint,
  data string,
  category string
)
partitioned by (category); -- Define partitioning

-- --- Insert some data --- 
insert into demo.my_iceberg_table values
(1, 'apple', 'fruit'),
(2, 'banana', 'fruit'),
(3, 'carrot', 'vegetable');

-- --- Read the data --- 
select * from demo.my_iceberg_table where category = 'fruit';
-- Returns apple and banana rows

-- --- Iceberg also supports time travel (syntax might vary slightly by catalog) ---
-- select * from demo.my_iceberg_table version as of <timestamp_or_snapshot_id>;

🛠️ How They Handle Your Data

Where They Fit Best

• Choose Delta Lake if:

  • Your team is heavily invested in Apache Spark or Databricks.

  • You prioritize write heavy operations and need native performance.

  • You’re okay with being slightly tied into the Databricks ecosystem.

• Choose Apache Iceberg if:

  • You want true engine independence: Spark, Flink, Trino, Presto... take your pick.

  • You expect your partitioning strategies to evolve as your data grows.

  • You prefer open, community driven technology over vendor driven.

Conclusion

If you're already working deep inside Databricks or Spark, Delta Lake will feel smoother.
If you're aiming for a more flexible, cloud-agnostic lakehouse or mixing multiple engines, Iceberg is your better long term bet.

Neither one is "better" universally. It all comes down to your specific use case.

Data architecture today isn't about picking "the best."
It's about picking what fits you best and being able to adapt as you grow.

I strongly suggest watching the video below:

References

• Delta Lake Documentation

• Apache Iceberg Documentation

• Apache Iceberg Vs. Delta Lake Vs. Apache Hudi! Data Lake Storage Solutions Compared!

• What is Delta Lake

• AWS re:Invent 2023 - Netflix’s journey to an Apache Iceberg–only data lake (NFX306)

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

The importance of high quality data cannot be overstated. Businesses rely on accurate and reliable data for decision-making, strategic planning and operational efficiency. However, poor data quality remains a significant issue and its financial impact on businesses is profound.

Not all business and data leaders truly anticipate the mechanics behind poor data quality investment. And it’s normal. It’s hidden cost/opportunity and demands to change the way that things already work. The market becomes mature but there’s still a lot of work. There are still many companies in different industries (especially non-tech) which have miles to cover.

Research indicates that poor data quality can cost businesses up to 20% of their revenue. This blog post delves into the hidden costs of poor data quality and underscores the necessity for businesses to invest in data quality management.

It’s more frequent to meet Data Quality frameworks being applied to complex system integrations and apps, in contrast with analytics teams which demand more tech maturity and budget allocation from the business.

Understanding Data Quality

But what do we mean by ‘Data Quality‘? Data quality refers to the condition of a set of values of qualitative or quantitative variables. There are several dimensions to data quality, including:

1. Accuracy: How closely data values match the true values.

2. Completeness: The degree to which all required data is present.

3. Consistency: The uniformity of data across different datasets.

4. Timeliness: How up to date the data is.

5. Validity: The extent to which data conforms to defined business rules and constraints.

6. Uniqueness: Ensuring that each entity is recorded only once in the dataset.

When data falls short in any of these dimensions, it can lead to substantial problems for a business.

I could say that throughout my experience I’ve seen half of companies take the above seriously and the other half not at all, while they really anticipated the importance. Yes, this is true.

The Financial Impact of Poor Data Quality

The financial impact of poor data quality manifest in various ways:

Operational Inefficiencies

Inaccurate or incomplete data can lead to significant inefficiencies in business operations. Employees may spend a considerable amount of time correcting errors, reconciling discrepancies, and verifying data accuracy. This not only reduces productivity but also diverts resources from more critical tasks.

Lost Revenue Opportunities

Businesses rely on data to identify and capitalize on revenue opportunities. Poor data quality can result in missed sales opportunities, incorrect pricing strategies, and flawed market analysis. For instance, if customer data is inaccurate, marketing campaigns may target the wrong audience, leading to lower conversion rates.

Increased Costs

Poor data quality can lead to increased costs in various areas. For example, inaccurate inventory data can result in overstocking or stockouts, both of which are costly for businesses. Additionally, errors in financial data can lead to compliance issues and potential fines.

Damaged Reputation

Errors and inconsistencies in customer data can lead to poor customer experiences, which in turn can damage a company’s reputation. Negative customer experiences can result in lost customers and a tarnished brand image, both of which have long term financial implications.

Poor Decision Making

Reliable data is important for informed decision making. Poor data quality can lead to flawed business strategies and decisions, which can have a big effect on all aspects of the business. Decisions based on inaccurate data can result in financial losses, missed opportunities and strategic missteps.

It’s both an alerting and monitoring issue. This is a dashboard example from Ryan Lowers.

The Path to High-Quality Data

To mitigate the costs associated with poor data quality, businesses must prioritize data quality management. Here are some steps to achieve high quality data:

Implement Data Governance

Establish a data governance framework that defines policies, procedures, and responsibilities for data management. This framework should ensure accountability and provide a structured approach to data quality.

Invest in Data Quality Tools

Leverage data quality tools that offer capabilities such as data profiling, cleansing, matching, and monitoring. These tools can help identify and rectify data quality issues in real time.

Educate and Train Employees

Ensure that employees understand the importance of data quality and are trained in best practices for data entry, management, and usage. Data quality should be a shared responsibility across the organization.

Conduct Regular Data Audits

Regularly audit data to identify and address quality issues. This proactive approach can help maintain high data quality over time.

Establish Data Quality Metrics

Define and monitor key data quality metrics that align with business objectives. These metrics should provide insights into the state of data quality and guide improvement efforts. Life will become easier and more enjoyable if you are part of the team.

Conclusion

The cost of poor data quality is a significant burden for businesses, impacting revenue, efficiency, and reputation. By investing in robust data quality management practices, businesses can turn data into a valuable asset that drives growth and success. Ensuring high quality data is not just a technical necessity.

It is a strategic imperative that can provide a competitive edge in the modern business landscape.

By understanding the financial impact of poor data quality and taking proactive steps to address it, businesses can safeguard their revenue and build a strong foundation for future growth.

References

MIT Sloan Review

Attacama

Cluedin

180ops

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

If data engineering is about constructing roads and bridges for data, data architecture is urban planning ensuring everything is designed for long term growth and usability.

In this episode of DataConscious – A Mindful Approach to Analytics, we dive into the world of Data Architecture. Our guest, Christina Kalimeri, provides all the necessary details on:

🎙 What it’s like to architect data solutions and integrate systems.

🎙 Her journey into the role: How she got started and what she learned along the way.
🎙 What her day to day actually looks like: Real talk about responsibilities, challenges, and wins. What the role actually does (beyond the job title)
🎙 The skills that truly matter: What she use daily vs. what’s often overhyped.
🎙 Career growth and industry trends: Where the role is headed and what aspiring professionals should know.
🎙 Honest advice for newcomers: What she wishes she had known before stepping into the role. How to transition into the role effectively.

Christina has extensive experience:

1. Identifying what needs to be done, in terms of business goals, in complex payments architectures.

2. In migrating core business solutions/applications within each other or in a centralised Data Warehouse. For her, trust me, migrating into a Data Warehouse is the fun part of the job.

3. Evaluating business analysis and collaborating with Banks, Fintech companies, Public Governance Authorities and several other vendors in huge scale projects.

4. Designing, developing and optimizing data systems: She creates robust database solutions by designing conceptual and logical data models, flowcharts, and optimized schemas.

5. Supporting software and application development: Analyzes structural requirements for new software and applications, integrate new components and enhance system performance through testing and troubleshooting.

Whether you're a data leader, engineer, BI professional, or decision-maker, this episode will help you navigate the evolving landscape of modern data architectures.

🎧 Listen now & subscribe for more insights on mindful data practices!

What Does a Data Architect Do?

A Data Architect’s job is to create a blueprint for an organization’s data infrastructure. They define:

• Where data is stored (databases, data lakes, data warehouses)

• How data moves (ETL/ELT processes, APIs, streaming pipelines)

• Who can access it (security policies, governance frameworks)

• How it scales (handling increasing data volumes, performance optimization)

Their work ensures that engineers, analysts, and business teams can use data effectively without running into bottlenecks, inconsistencies, or security risks.

Two main types of Data Architects

1️⃣ Those who design infrastructure for reporting and analytics needs – Their focus is on building data warehouses, data lakes, and ETL pipelines to support BI dashboards, machine learning models, and analytical queries.

2️⃣ Those who design infrastructure for real-time data flow between applications – Their job is to ensure multiple systems can communicate efficiently in real time. For example, if Systems 1, 2, and 3 need to exchange data, the architect will design a solution to facilitate this. This often involves an intermediary layer (e.g., an event-driven architecture using Kafka or a data hub) that collects, processes, and redistributes data among all systems to maintain synchronization.

These two approaches require different skill sets, but both are essential depending on business needs.

Real-World Example: E-Commerce Platform

Imagine an e-commerce company that collects user behavior, sales transactions, and product inventory data. The Data Architect would design:

✅ A real-time event streaming system (Kafka, Kinesis) to track customer actions on the website.
✅ A cloud-based data warehouse (Snowflake, BigQuery, Redshift) to store transaction data for analysis.
✅ A data governance policy ensuring only authorized teams can access sensitive payment data.
✅ A scalable architecture that supports high traffic during peak sales (Black Friday).

🛠 Skills - Tools - Key Responsibilities

Hard Skills

1. Designing Data Infrastructure

• Choosing between data lakes (S3, Delta Lake) vs. data warehouses (Snowflake, BigQuery, Redshift)

• Deciding between batch processing (dbt, Airflow) vs. real-time processing (Kafka, Flink)

• Managing cloud environments (AWS, Azure, GCP)

2. Setting Data Standards & Governance

• Defining data models (Star/Snowflake/Data Vault/3NF schema for warehouses)

• Implementing access control (RBAC, IAM roles)

• Ensuring compliance with GDPR, HIPAA, SOC 2

3. Optimizing Performance & Cost

• Indexing & partitioning strategies for faster queries

• Data lifecycle management (archiving old data to reduce costs, hot/cold storage)

• Query optimization (Snowflake’s automatic clustering, BigQuery’s partition pruning, Databricks Spark techniques)

4. Collaborating with Teams

• Working with Data Engineers to ensure pipelines match the architecture

• Assisting Data Scientists by structuring data for ML models

• Supporting Business Intelligence teams with well organized datasets

Soft Skills

• Strategic Thinking: Ability to align data infrastructure with business needs.

• Communication: Clear articulation of data architecture decisions to both technical and non-technical stakeholders.

• Problem-Solving: Ability to anticipate and address data-related challenges before they arise.

• Collaboration: Working effectively with cross-functional teams.

Challenges Faced by Data Architects

1️⃣ Balancing Flexibility & Standardization: Companies want flexible data solutions, but too much flexibility leads to chaos. Architects must set clear standards while allowing teams to innovate.

2️⃣ Managing Data Chaos: As organizations adopt data lakes, warehouses, real-time streaming, and microservices, keeping track of all data assets is challenging. Metadata management tools like Data Catalogs (Select star, Unity Catalog etc) help.

3️⃣ Cost Optimization: Cloud storage and compute costs can skyrocket if not managed properly. Architects must balance on-demand vs. reserved instances, cold vs. hot storage, and query efficiency.

4️⃣ Scaling for Growth: A system that works for 10 TB of data may not work at 500 TB. Architects need to plan for horizontal scaling (distributed databases) and vertical scaling (query optimization, caching).

📈 Career growth opportunities and salary expectations

Data Architects are in high demand as organizations increasingly rely on data for decision making. According to industry reports, salaries for Data Architects vary based on experience and location, but they generally range from $120,000 to $180,000 per year in the US, with senior roles exceeding $200,000.

How This Role is Evolving

🚀 Data Mesh & Decentralization: Companies are moving away from monolithic data warehouses and embracing domain driven architecture, where teams own their data.

⚡ Real-Time & Event-Driven Data: More companies are replacing batch processing with real-time analytics. Kafka, Flink are gaining traction.

🛠 AI-Driven Automation: Architects are leveraging automated data modeling, governance, and cost optimization tools to reduce manual work. Tools: Monte Carlo (Data Observability), dbt Cloud, Dataplex

Who Should Consider This Role?

A Data Architect is NOT an entry level role. It requires YEARS of experience in data engineering, database administration, or cloud architecture. Strong skills in SQL, data modeling, and cloud platforms are essential.

This role is ideal for:
✅ Senior Data Engineers who enjoy designing long-term solutions
✅ Database Administrators looking to expand into cloud and big data systems
✅ Cloud Architects who want to specialize in data infrastructure

If you enjoy solving complex data problems, designing scalable systems, and working at the intersection of technology and business, this could be the right career move.

🎙 What the Data Pros Say

Beyond the technical breakdowns, you’ll hear from experienced professionals in each role. Through our ‘What the Data Pros Say’ section, we’ll bring you firsthand insights from people actively working in the field. Thanks to everyone for contributing :)

Starting my career as a mechanical engineer in the manufacturing industry, I was intrigued by the value of leveraging data and the hidden, untapped potential it could provide. Wanted to learn more about the magic power that data could offer I transitioned, at first, into software and data engineering roles and then eventually evolved into a data architect.
Data architect role is surely demanding, requiring a wide variety of skills. However, this also ensures that it never gets boring, as it involves a diverse range of responsibilities. Some things learned from my experience are:

• Technical excellence, continuous learning, and visionary thinking are essential,

  as decisions must be made regarding data engineering solutions, cloud infrastructure, and the implementation of data governance.

• As the data architect has to communicate with a number of technical teams like cloud engineers, cyber security or business teams that provide requirements, crucial for the success of the role are communication skills and stakeholder management.

• Basic element of the role is providing leadership to data engineers so emotional intelligence and motivation are necessary for the success of the enitre data team.

Long story short:

A well designed data architecture is the difference between a scalable, efficient data ecosystem and a slow, unreliable mess. Data Architects shape how organizations use data, and their decisions impact everything from business intelligence to AI-driven products.

In a nutshell…

Additional material:

• https://www.snowflake.com/trending/data-architects/

• https://www.simplilearn.com/data-architect-article

• https://www.reddit.com/r/dataengineering/comments/1d64ht9/what_does_a_data_architect_actually_do/

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

When you’re working with databases, keeping data accurate, reliable, and consistent is non negotiable. That’s where ACID transactions come in. ACID stands for Atomicity, Consistency, Isolation, and Durability, 4 key principles that ensure databases handle transactions correctly and keep your data safe from chaos. Let’s break them down with some real world examples and explore what happens when ACID principles aren’t followed.

Breaking Down ACID Properties

1. Atomicity: The All or Nothing Rule

Think of atomicity as the "no half-measures" rule. A transaction is either fully completed or not done at all, no in between states allowed.

Real-World Example: Online Shopping Cart

Imagine you're shopping online, and you’ve added a few items to your cart. When you check out, multiple things happen in the background:

1. Your credit card is charged

2. The store updates its inventory

3. An order confirmation is sent to you.

Now, what if the payment goes through but the order isn’t recorded? Nightmare. Atomicity prevents this by ensuring that if one step fails, the whole transaction is rolled back, so you don’t end up charged for an order that never existed.

2. Consistency: Keep it normal

Consistency ensures that data follows all the rules and stays valid before and after a transaction. No weird anomalies or broken constraints.

Real-World Example: Bank Withdrawal

Imagine Bob has $100 in her bank account and tries to withdraw $200. The banking system enforces a rule that an account cannot have a negative balance.

Here's what should happen:

1. The system checks Bob’s account balance.

2. Since Bob has only $100, the transaction violates the consistency rule (an account cannot go negative).

3. The transaction is canceled, and Bob's balance remains unchanged.

If the system did not enforce consistency, it might allow Bob to withdraw more money than she has, leading to data corruption, financial mismatches, or even bank failures.

Consistency ensures that databases operate within predefined rules, preventing logical errors and maintaining trust in the system.

3. Isolation: Preventing Transactions from Colliding

Isolation makes sure that multiple transactions don’t get in each other’s way.

Without it, you could get scenarios like double bookings or incorrect balances.

Real-World Example: Airline Ticket Booking

Picture this: two people are trying to book the last available seat on a flight at the exact same moment. If the system doesn’t handle transactions properly, both might end up thinking they got the seat, leading to an overbooking mess. Isolation ensures that only one of those transactions goes through first, avoiding conflicts.

4. Durability: Making Data Stick

Once a transaction is completed, its changes must be permanent, even if the system crashes right after.

Real-World Example: E-commerce Order Processing

You’ve placed an order, got a confirmation email, and now you’re waiting for your package. But what if a system crash happens right after your order was processed? If durability wasn’t enforced, you might lose your order in the void. Thankfully, databases commit transactions to permanent storage, ensuring that once your order is placed, it’s there for good.

When ACID Fails: The $440 Million Mistake

Not following ACID principles can lead to disastrous consequences and Knight Capital’s $440 million meltdown in 2012 is a serious example.

What Went Wrong?

A faulty software update caused the system to execute a bunch of unintended stock trades at lightning speed. Because ACID principles weren’t properly enforced:

• Atomicity failed: Partial trades went through without rollbacks.

• Consistency failed: The system allowed invalid transactions to be executed.

• Isolation failed: Transactions clashed with each other, causing pricing chaos.

The Aftermath?

Knight Capital lost $440 million in just 45 minutes, forcing them into bankruptcy. One tiny ACID failure, and boom. Game over.

Final Thoughts

ACID transactions might sound technical, but they’re absolutely critical in keeping databases safe and reliable. Whether it’s banking, e-commerce, or real-time systems, following these principles ensures that data remains accurate, consistent, and trustworthy.

Have you ever run into messy database issues at work? Share your experience in the comments!

📌 Enjoyed this post? Subscribe to DataConscious for more deep dives into database design, data analytics, and mindful approaches to data integrity.

Great analysis starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Thanks for reading DataConscious – A mindful approach to analytics! Subscribe for free to receive new posts and support this effort

Welcome back! I hope you found the previous deep dive into SQL optimization both informative and inspiring. Now, as we continue our journey into the world of SQL mastery, this second part of the series will take us even further. 🚀

Remember, optimizing your SQL queries isn’t just about making them run faster, it's about empowering you and your team to work more effectively together. Consistency in your coding style can foster collaboration and streamline the review process, which is why we can’t overlook the importance of following best practices.

Join me as we delve into specific strategies with practical examples and insights that will elevate your SQL skills to pro levels. Let’s supercharge your sql code management and make your code work for you, not against you!

Like I mentioned in the previous post:

By following these tips you will make your code reviewer life MUCH easier.

5. Use proper naming convention

Overall, it is important to use a consistent naming convention and to choose names that are descriptive, clear, and easy to understand. This makes it easier for users to work with the database and ensures that queries and joins are accurate and efficient.

Instead of this

Do this

Notice how the table and column names are all abbreviated and use a non-standard naming convention. The table name is in singular form, which could be confusing if there are multiple entities in the table. The column names use abbreviations, which may not be immediately clear to users who are not familiar with the system. This can make it difficult to understand the relationships between tables and can lead to confusion and errors in querying the database.

6. Avoid select *

Please stop doing that. Reduce the number of columns returned in the select statement to only those that are necessary for the query. This is a good practice for several reasons, especially in a columnar database. In a columnar database, each column is stored separately from the other columns, which means that retrieving data from a large number of columns can be slower than retrieving data from a smaller number of columns. by selecting * we increase the amount of data that needs to be transferred across the network and processed by the application. This can result in higher memory usage and longer processing times, which can negatively impact query performance. Furthermore, we reduce the clutter in the select statement and make it easier to see what the query is doing.

Instead of this

Do this

7. Use proper indentation and white spaces

Using proper indentation and spacing can help to make SQL code more readable and easier to maintain. It’s important to follow best practices and to keep the code well-organized to prevent errors and improve efficiency.

Instead of this

Do this

The ORDER clause is uppercase on purpose to remind you, how ugly it is to code in uppercase :P

8. Use exists instead of in

The primary advantage of using exists over in lies in the way each construct processes the query. With exists, the SQL engine evaluates the subquery and stops as soon as it finds the first record that matches the criteria. This characteristic makes exists particularly valuable when dealing with larger datasets, as it can significantly reduce processing time.

In contrast, the in clause compiles a list of values from the subquery results before it even begins to evaluate the outer query. This means that if the subquery returns a large set of values, the database must first gather all these values before starting the comparison. The combination of these steps can make in less efficient, especially as the size of the dataset increases.

In simple terms, you can think of exists like a librarian searching for a specific book on a shelf. As soon as they find the book they're looking for, they stop searching and confirm its presence. On the other hand, using in is like having the librarian first pull out all the books on that shelf, regardless of any particular title they might be looking for, resulting in extra work and potential delays.

Instead of this

Do this

As we wrap up this two-part series on optimizing SQL queries, I hope you've gained valuable insights that will elevate your skills and streamline your SQL coding practices. The journey to mastering SQL is not just about writing efficient queries. It’s also about clarity, consistency and collaboration within your team.

By implementing the strategies we've discussed, you'll find that your SQL code becomes not only more efficient but also significantly more readable and maintainable. This will not only make your own work easier but also simplify the review process for your colleagues.

Remember, great analysis begins with mindful coding practices. Keep these tips in your toolkit, and you'll be well on your way to writing SQL like a pro.

Thank you for being part of the DataConscious community!

Great analysis starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Thanks for reading DataConscious – A mindful approach to analytics! Subscribe for free to receive new posts and support this effort

We're hitting pause on the Data Role series for the next 1–2 weeks as I dive deep into producing the upcoming DataConscious podcast episodes and let me tell you, it's been an intense but exciting process! 🎙️

But don't worry, the content won’t stop! During this break, I'll be sharing two value packed posts focused on SQL optimization and pro level SQL techniques. Whether you're looking to write more efficient queries or level up your SQL game, these insights will help you code like a pro. Stay tuned! Powerful SQL tips are coming your way! 🚀

SQL query optimization is critical for advanced users who work with large, complex databases. Even small improvements in query performance can translate to significant gains in productivity and cost savings.

In some cases, poorly optimized queries can even bring a system to its knees.But it’s also a matter of code readability and collaboration. When building SQL pipelines with your teammates you need to be consistent on the way you work. You cannot have half of the scripts coded in style A and the rest in style B.

Finally, by following these tips you will make your code reviewer life MUCH easier.

At its core, query optimization involves finding the most efficient way to retrieve data from a database. This often requires a deep understanding of the database structure, as well as the underlying hardware and software environment. Optimizing queries involves a variety of techniques.In this post, we will explore some of these advanced techniques in greater detail, providing examples and best practices for optimizing SQL queries.

1. Use lowercase code

The truth is that for many years, I used to code in uppercase SQL. This finally changed, when I had to adapt myself in a professional setting that codes lowercase. I got used to it immediately and I never looked back.

Despite the fact that uppercase is more commonly used for SQL keywords, as it can make the code easier to read and differentiate between keywords and identifiers, unfortunately, it is not ergonomic. Lowercase style is less “noisy” and more natural. You do not have to focus on pressing CAPS LOCK but in the SQL side of things. Finally, readability is getting the same when you are getting used to lowercase.

Instead of this

Do this

Consistency is key when it comes to choosing between uppercase and lowercase keywords. If you’re working on a team or with a codebase that uses lowercase keywords, it’s important to maintain that style for the sake of consistency and clarity.
Ultimately, the choice between uppercase and lowercase SQL keywords comes down to personal preference and the style guidelines of your team or organization.

2. Use leading commas

In SQL, leading commas and trailing commas refer to the placement of commas in a list of items within a SQL statement. Let’s break down these terms:

• Leading commas refer to commas that appear at the beginning of each element in a list

• Trailing commas refer to commas that appear at the end of each element in a list

Of course, there is no single standard for how to write SQL code. Some people use uppercase keywords, while some lowercase. Some use trailing commas and others leading commas.

Trailing commas style is easier to read for most people. One advantage of using leading commas is that the last item doesn’t differ from the others, but on the other hand the first item differs.

To my experience, leading commas style is easier for debugging as for errors that will occur due to missing commas, it will be faster to check the lines from start instead of their end and fix the appropriate line.

The most crucial thing here is to adapt ourselves in the organisation that we belong and stick to the team’s coding style. It’s more useful to have pieces of code with a consistent look, rather applying your personal preferences.

Instead of this

Do this

aha moment: Imagine that a comma is missing. Which style suits you better for debugging?

3. Refer table names in columns

In a database with multiple tables, it is possible that two or more tables have columns with the same name. If we don’t specify the table name when selecting columns, the database may not know which column we are referring to, leading to errors or incorrect results.

Instead of this

Do this

It makes the query more readable and easier to understand. When we refer to columns without specifying the table name, it can be difficult to determine which table the column comes from, particularly if the query involves multiple tables.
by including the table name in the select statement we avoid conflicts when joining tables. It is possible for two or more tables to have columns with the same name.

4. Use Common Table Expressions (CTEs)

CTEs can make the query more readable and easier to understand. Nested queries can become very complex and difficult to read, especially if there are several levels of nesting. It can also improve query performance. When we use a nested query, the database management system has to execute the inner query before it can execute the outer query. This can result in slower query performance, especially if the inner query involves a large amount of data or a complex calculation.
They can be reused in multiple queries. When we use a nested query, we can only use the result set of the inner query once. If we need to use the same result set in multiple queries, we have to execute the inner query multiple times, which can result in slower query performance.

• Breaking down complex queries into smaller, results in more manageable and readable parts. This can greatly improve the clarity and maintainability of SQL code.

• Code Reusability Results can be referenced multiple times within a single query.

• Easier Debugging and Testing It becomes simpler to isolate and debug specific parts of a query.

• Code Documentation Properly named CTEss can serve as self-documenting code, providing descriptive names for temporary result sets and making the intention of the query clearer.

Instead of this

Do this

These queries achieve the same result but they are more clear and easier to read and understand. They are also faster, especially if there are many orders and/or customers, because it is not executing subqueries for each row in the main query. It breaks the query down into smaller, more manageable parts, and simplifies the logic, making it easier to read and optimize.

Stay tuned!

Great analysis starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Thanks for reading DataConscious – A mindful approach to analytics! Subscribe for free to receive new posts and support this effort

Let’s start…

1. You’ve taken courses, built a portfolio, maybe even gotten a certification.
2. You’ve applied to dozens (or hundreds) of jobs labeled “Entry-Level.”
3. And yet… every role asks for 2-3 years of experience.

What happened to actual entry level jobs?

❌ The Harsh Reality:

Companies don’t want to train junior data professionals anymore. It’s that simple. And it hurts.

Instead, they:
✅ Automate tasks that juniors used to do
✅ Hire experienced analysts who can deliver value immediately.
✅ Expect candidates to already have real world experience.

💡 What Can You Do?

If the jobs aren’t coming to you, you need to make yourself hireable in a different way.

✅ Stop Applying —> Start Building
• Create projects that solve real business problems, not just Kaggle datasets.
• Document your work: Write case studies, post dashboards, explain your thought process.
• Show that you can drive impact, not just write SQL or Python.

✅ Leverage Alternative Pathways
• Freelance on Upwork or Fiverr to gain hands on experience.
• Contribute to open-source projects, nonprofits, or startups.
• Join hackathons, competitions, or online communities to collaborate on real work.

✅ Network Like Your Career Depends on It (Because It Does)
• Connect with hiring managers and industry professionals.
• Engage with LinkedIn posts, comment with insights, and be visible.
• Ask for informational interviews people hire who they know.
• Find a mentor.

🔥 The Truth:

Companies aren’t eliminating entry level roles. They’re just raising the bar for what “entry level” means. Every data person who I have discussions with, validates it.

If you’re waiting for a company to take a chance on you, you might be waiting forever. Make them notice you instead.

What’s your biggest struggle in breaking into data?

Great analysis starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Thanks for reading DataConscious – A mindful approach to analytics! Subscribe for free to receive new posts and support this effort

In this episode of DataConscious – A Mindful Approach to Analytics, we dive into the world of Data Engineering and the game-changing concept of Data Mesh. Our guest, Konstantinos Siaterlis (bsky), Senior Cloud Engineering Tech Lead at Mondelēz International and AWS Hero, shares his deep expertise in building scalable, reliable cloud infrastructure and leading Data Mesh transformations.

Below you can find his Substack publication.

The Last Dev

Writing about DevOps, Cloud Engineering, Data Engineering and more!

By Konstantinos Siaterlis

Join us as we explore:
🔹 The real role of a Data Engineer – beyond the job title
🔹 Essential vs. overhyped skills in the industry
🔹 Career growth, trends, and the future of data platforms
🔹 How Data Mesh redefines data ownership and scalability

Whether you're a data leader, engineer, BI professional, or decision-maker, this episode will help you navigate the evolving landscape of modern data architectures.

🎧 Listen now & subscribe for more insights on mindful data practices!

What the role actually does (beyond the job title)

When you start as a data engineer, you expect to work with cutting edge, state of the art tools like Hadoop, Spark, and Kafka daily. Reality? Many companies still rely on legacy systems, often using drag and drop ETL tools like SSIS instead of complex coding solutions. I have seen many disappointed data engineers working on stuff that they really do not want to. But you know what? It’s always part of the job.

Core Responsibilities

1. Building & Maintaining Pipelines: While new pipelines are part of the job, much of the work involves migrating existing ones to newer platforms for cost savings and performance improvements FOR THE BUSINESS.

2. Ensuring Data Accessibility: Data engineers create structured, reliable data storage that analysts and data scientists can easily access. Often, companies provide SQL based interfaces to simplify usage.

3. Keeping Systems Running: Managing data infrastructure isn’t just about innovation, it’s about maintaining stability. Scaling storage, improving processing speeds and ensuring smooth data flow are critical daily tasks.

4. Bridging the Gap Between Raw Data and Insights: Engineers clean, organize and integrate data, making it usable for other teams. Without this foundation analytics and AI efforts WILL FAIL.

The Reality Check

Unlike data scientists, data engineers don’t get the spotlight. Their work is behind the scenes, ensuring that business decisions with data are even possible. It’s not the “sexiest job” but it’s essential. And while it may lack public/organisational recognition, the financial rewards are there! Salaries often surpass expectations as demand for skilled engineers grows.

🛠 The skills and tools required to succeed

Data engineering is an ever evolving field that requires expertise in a variety of tools and concepts. Technologies that were critical a few years ago have been replaced by more efficient solutions and modern data engineers must stay updated to remain competitive.

Skills

A strong data engineering skill set includes:

• SQL & Python: SQL for querying and transforming data, Python for automation, scripting, and working with libraries like Pandas and PySpark.

• Cloud Technologies: Expertise in AWS, Google Cloud, Azure, and Databricks for scalable data solutions.

• Workflow Orchestration: Using tools like Apache Airflow, Prefect, Dagster to automate and manage data pipelines.

• Data Modeling: Designing efficient data structures:

  • 3rd Normal Form (3NF): Optimized for OLTP systems with structured relationships.

  • Data Vault: Flexible and scalable modeling for historical tracking.

  • One Big Table (OBT): Simplified models for analytics and reporting.

• Business Acumen (my favourite): So simple as that, technical skills aren’t enough! Without business acumen, a data engineer is just a pipeline builder. Understanding company goals and industry challenges is what separates an average engineer from an indispensable one. STICK WITH ME: It's not about moving data, it’s about delivering impact. Engineers who don’t align their work with business priorities waste time and resources. Those who do? They drive revenue, streamline operations, and make data a competitive advantage. How should your manager ask for budget → and then reward you with a salary raise, if the business value is not explainable?

Tools

The modern data stack consists of:

• Storage & Processing:

  • Databricks: Unified data analytics and AI platform, built on Apache Spark for large scale data processing.

  • Presto and Trino: Powerful distributed SQL engines designed for querying massive datasets across multiple sources. Originally built at Facebook, Presto was later forked into Trino for improved performance and scalability.

  • Apache Iceberg & Delta Lake: Ensuring schema evolution, ACID transactions and efficient lakehouse management.

  • BigQuery, Snowflake, Redshift: Scalable cloud data warehouses.

• Data Ingestion & Integration:

  • Airbyte & Fivetran: Low code ELT tools for seamless data movement.

  • Kafka & Debezium: Real time data streaming and Change Data Capture (CDC) solutions.

• Data Transformation:

  • dbt (Data Build Tool): Managing and defining analytics ready datasets.

Data Engineering Concepts

Key concepts for handling and structuring data effectively:

• Change Data Capture (CDC): Tracking and processing real time database changes.

• Slowly Changing Dimensions (SCD): Managing historical data changes for accurate reporting. SCD is like keeping a historical record of the data, while CDC is like getting real time updates on any changes made to that data.

• ETL vs. ELT: Traditional ETL (Extract, Transform, Load) vs. modern ELT (Extract, Load, Transform) for scalability.

• Batch vs. Streaming Processing: Choosing between scheduled batch jobs and real time event driven pipelines.

• Lakehouse Architecture: Combining the best of data lakes and warehouses with tools like Databricks & Delta Lake.

📈 Career growth opportunities and salary expectations

Data is the new gold of the 21st century (sic). Some of the largest companies in the world process terabytes of information daily and data engineering has become one of the highest demand jobs globally. A quick search on LinkedIn reveals thousands of job postings for data engineers, with an average salary of $130,000 per year in the U.S. The demand is immense, yet the market remains relatively niche.

The Power of Now in Career Growth

Success thrives in the present moment. Every opportunity whether from friends, clients, or unexpected encounters exists now and the key is to seize it without hesitation. When a chance to contribute arises, say yes without overanalyzing. The present moment is where momentum is built when things are working, don’t pause to overthink. Lean into the flow, embrace the now and let it carry you forward.

Laying the Foundation: Building a Strong Data Career Portfolio

To truly capitalize on opportunities, it’s essential to establish a solid foundation:

• Master the fundamentals: Python, SQL, and command line proficiency are the backbone of data engineering.

• Say yes to skill-building opportunities: Engage in projects that push your limits and broaden your expertise.

• Reduce risk: Expand your network and diversify your skill set.

Many people feel overwhelmed when starting their data engineering journey because of the sheer number of tools and buzzwords Spark, Hadoop, Hive, Airflow and so on. Unlike software engineering, where roles are well-defined, data engineering is more ambiguous.

However, a quick glance at job descriptions on LinkedIn reveals a pattern:

1. Expertise in relational SQL

2. Proficiency in Python

3. Experience with workflow scheduling tools like Apache Airflow

These foundational skills are critical for getting started. Once you gain confidence in Python, SQL, and command line (CLI) operations, you’ll be ready to tackle data storage, orchestration, and workflow automation which are the key components of modern data systems.

It’s important to focus on concepts rather than tools. Technologies evolve rapidly by the time you master one, a newer version will emerge. Instead, learn core principles like distributed computing and scalable data architectures.

The Mindset for Long-Term Success

Success in data engineering or any field demands practice through projects and building a strong portfolio. The hardest part of starting out is feeling lost due to the overwhelming number of resources. Stick to one high quality resource and master it before moving on.

Instead of reading documentation endlessly, start building. Experiment, make mistakes, and refine your understanding through hands on projects.

Take calculated risks, optimize for long term value, and continuously seek ways to own your career trajectory. The best opportunities come not from job applications but from the things you create and share with the world.

🔁 How to transition into the role effectively

🔹 Junior Engineers → Start with SQL (it's everywhere), get comfortable with Python, and understand ETL vs. ELT. Learn how to work with JSON, XML, YAML, and the basics of data modeling (star schema, Data Vault, One Big Table, 3NF etc).

Oh, and don’t forget Git…you’ll be using it a lot!

🔹 Senior Engineers → Master data modeling, CDC, data streaming, and infrastructure. You should be able to navigate CI/CD, automation, and cloud platforms effortlessly. Most importantly, you’re no longer just executing, you’re making key technical decisions and mentoring others.

🔹 Lead Engineers → You’re setting the architecture vision, ensuring security & performance, and influencing the entire org’s data strategy. You’re not just coding anymore, you’re leading, guiding, and making sure everything scales and breaks rarely.

Remember: You don’t need to know it all at once. Stay curious, stay adaptable, and keep learning.

🎙 What the Data Pros Say

Beyond the technical breakdowns, you’ll hear from experienced professionals in each role. Through our ‘What the Data Pros Say’ section, we’ll bring you firsthand insights from people actively working in the field. Thanks to everyone for contributing guys :)

Like many, I initially jumped on the data science hype train, eager to build cool machine learning models. It felt like being a wizard, conjuring insights from raw data! But I quickly hit a wall – my magical models were failing because the data was a mess. It was like trying to bake a cake with sand instead of flour. That’s when I discovered the real magic: data engineering. It’s not as flashy, sure, but it’s the bedrock of everything. Data engineering is like building the solid foundation and sturdy frame of a house before you start decorating with fancy (data science) furniture. Without it, the whole thing collapses. It’s the unsung hero, making sure the data is clean, reliable, and ready for its close-up.

My journey into data engineering began in physics, where I fell in love with solving problems using data and programming. From analyzing data produced by experiments to building simulations in C. That passion led me here. Here are a few thing I learned as a data engineer, the past years:

• Master the basics: They’re timeless and most are technology-agnostic.

• Know your tools: They’re your leverage in building efficient systems. This includes everything from the text editor you use to the last tool in your current stack.

• Embrace change: This field evolves fast, and loving to learn is non-negotiable.

• Learn to communicate with non-technical people: Translating business needs into data solutions is key. Let me tell you a secret, sometimes the best solution might not even require a technical implementation.

• Share your knowledge: Teaching others, documenting processes, and fostering a culture of learning can amplify your impact far beyond your own work.

  Remember, “If you want to go fast, go alone. If you want to go far, go together.” Stay curious, keep learning, and enjoy the journey. The data world is full of opportunities and challenges!

I came into data engineering through the backdoor, studying AI I quickly realized that no matter how powerful an algorithm is, it’s useless if the data feeding it is unreliable, messy, or slow to arrive. Without clean, reliable, and well-structured data, even the best analysts and data scientists are stuck fighting fires instead of driving impact. That’s where data engineering comes in. It’s not just about moving data from point A to point B but about enabling teams to trust their data, make better decisions, and focus on solving real problems. Whether it’s building scalable cloud pipelines or ensuring smooth data flows across an organization, data engineering is what turns raw information into real business value. And that’s what makes it so rewarding

Long story short:

It’s not as flashy, sure, but it’s the bedrock of everything.

Translating business needs into data solutions is key.

It’s not just about moving data from point A to point B but about enabling teams to trust their data, make better decisions, and focus on solving real problems.

In a nutshell…

Additional material:

• https://www.datamesh-architecture.com/

• https://www.datamesh-architecture.com/data-product-canvas

• https://www.datamesh-governance.com/

Great engineering starts with great awareness. Be mindful, be DataConscious.

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Let's HONESTLY admit that job titles don’t tell the full story…

As I was thinking about the content for DataConscious while running on the treadmill at the gym, an idea struck me…I should start the newsletter journey with the basics. First things first.

That’s how I decided to create a series of posts where we’ll deep dive into each role breaking them down in detail. Each post will also be accompanied by a podcast episode featuring industry professionals who will share their valuable insights and experience.

🔹 A Data Analyst in one company crunches numbers in Excel, in another, they’re writing complex SQL queries and managing data models.

🔹 An Analytics Engineer might be fine tuning data tests in one role, but in another, they’re the architect behind the entire ETL pipeline.

🔹 A Data Scientist in one company builds machine learning models, in another, they’re mostly creating reports and explaining metrics.

🔹 A Data Engineer in one job is all about Spark and real time pipelines, while elsewhere, they’re maintaining old school ETL jobs in SQL.

🔹 A Head of Data in one organization sets strategy and builds teams, in another, they’re hands-on coding and still cleaning up legacy reports.

That’s why I’m launching The Data Career Compass, a post + Podcast series breaking down every data role, featuring real industry pros sharing what they actually do, the tools that matter, and how they got there.

The goal? To bring clarity to how the data landscape is structured today. We’ll create a dedicated card for each role, covering everything you need to know.

🎙 The podcast

I really like podcasts and the idea to start a podcast series was always stuck into my mind. The truth is I was not sure about the topic. And boom, here it is!

To bring these roles to life, each post in this series will be paired with a podcast episode featuring industry professionals who have firsthand much of experience in the role we're exploring.

Why? Because job descriptions only tell part of the story, the real insights come from the people living the job every day.

In each episode, we’ll dive into:
🎙 Their journey into the role: How they got started and what they’ve learned along the way.
🎙 What their day-to-day actually looks like: Real talk about responsibilities, challenges, and wins.
🎙 The skills that truly matter: What they use daily vs. what’s often overhyped.
🎙 Career growth and industry trends: Where the role is headed and what aspiring professionals should know.
🎙 Honest advice for newcomers: What they wish they had known before stepping into the role.

Real stories, real experiences, and real advice from people who have been where you want to go.

Whether you're trying to choose a career path or level up in your current role, these conversations will give you insider knowledge that you won’t find in a typical job post.

*There will also a podcast for the latest post Increasing the reputation of Business Intelligence teams within an organization from a BI Leader…🤫

🔊 Subscribe, tune in, and let’s navigate the data world…one role at a time.

📌 What the role actually does (beyond the job title)

I remember when I moved to Athens for my first job as a Data Scientist. I was excited but also 80% unsure about what my role truly entailed. And guess what? That’s completely normal. No one knows exactly what to expect when they start out and if you feel that way, don’t worry.

The reason? Companies shape roles differently based on their needs, teams, and tech stack. There’s no universal standard.

For example, in one company, a Data Analyst might work only with processed data from a Data Warehouse, while in another, they might also be responsible for building data pipelines.

Is one approach more correct than the other? Not necessarily. It all comes down to what aligns with your goals and interests.

Similarly, an Analytics Engineer in one company might focus solely on writing tests, in another, they might handle everything from ETL to dashboarding, and in a third, they might be dedicated entirely to data quality checks.

That’s why we’ll hear directly from industry professionals performing these roles because real world insights are invaluable. Trust me, you won’t want to miss this!

🛠 The skills and tools required to succeed

Conversations around tools are never-ending.

• Tableau vs. Power BI

• Azure Synapse vs. AWS Redshift vs. Google BigQuery vs. Snowflake

• Snowflake vs. Databricks

• Python vs. R (though we know who the winner is here 😉)

• Prefect vs. Mage

…and the list goes on.

But from my perspective, there’s no absolute winner, not without context. A tool is only as good as the problem it solves and the environment it operates in.

That’s why, in this series, we’ll break down the key tools and tech stacks used in each role, so you can understand what’s truly essential and avoid wasting time on tools you don’t need.

📈 Career growth opportunities and salary expectations

• What does the career path for this role look like?

• Role Growth in the industry

• Where have professionals ended up after 3, 5, or 10 years in this role?

• What are the salary expectations for entry, mid, and senior levels across different locations?

• How easy (or difficult) is it to advance in this role?

We’ll dive into real career trajectories to give you a clear understanding of where each role can lead you.

🚀 How to transition into the role effectively

Whether you're a student, a career switcher, or someone already in the data space looking for a shift, we’ll lay out practical steps to help you land the role you want.

This includes:
✅ Recommended courses
✅ Certifications worth pursuing
✅ Real-world projects to showcase your skills
✅ Portfolio-building strategies

The goal? To make your transition smooth, intentional, and effective.

🎙 What the Data Pros Say

This is my favorite part of the series!

Beyond the technical breakdowns, you’ll hear from experienced professionals in each role. Through our ‘What the Data Pros Say’ section, we’ll bring you firsthand insights from people actively working in the field:

📢 What they wish they knew before starting
📢 The biggest challenges they face daily
📢 Their advice for aspiring data professionals

If you want to give your perspective on the role you are in, I really want to hear from you, comment the post, or drop me a message!

Who is this series for?

🔹 If you’re exploring data careers, this series will give you clarity and direction.
🔹 If you’re already in one role but considering a switch, you’ll gain insights into alternative paths.
🔹 If you’re a business leader hiring for data roles, you’ll better understand what to look for in candidates.

Great analysis starts with great awareness. Be mindful, be DataConscious.

References

❤️ Liked this article? Make sure to push the button

💬 Community needs your voice, do you have something to add? Feel free to

🔁 Know someone that would find this helpful? Feel free to share

Share

Having worked in diverse teams across multiple industries, I have always been curious about the role and impact of BI and Data teams—how to measure their performance, enhance their influence, and elevate their reputation within an organization. Achieving this requires a strong foundation in data democratization and data literacy, ensuring that insights are accessible, trusted, and effectively used across all levels of the business.

Business intelligence (BI) teams play an important role in helping organizations make informed decisions and achieve their goals. However, establishing and maintaining a strong BI team can be challenging.

Creating a high-performing Business Intelligence (BI) team isn’t just about filling seats—it’s about strategically selecting experts whose skills align with the organization’s unique data challenges. Each role serves a distinct purpose, and overlooking these critical positions can leave gaps that undermine data-driven decision-making. It’s critical for the hiring and management teams to choose not onl;y the correct people but also the correct roles for the team. I’ll try to group up some roles and will not dive into much detail on how they differ in large/huge scales. Here’s how to align hiring with business needs:

1. BI Analyst / Data Analyst
When business leaders clamor for answers but drown in spreadsheets and DWH tables, a Data Analyst bridges the gap. They transform messy data into clear KPIs, dashboards, and ad-hoc reports, ensuring stakeholders base decisions on facts—not hunches. Companies lacking analysts often rely on error-prone manual processes, resulting in misaligned strategies and missed opportunities. They can deliver .CSVs, Dashboards, ad-hoc queries to non-technical people, they can communicate efficiently with stakeholders. Stuff that a data engineer is not that willing to do.

2. Data Engineer
This is currently the most in-demand role. Companies have realized that hiring data analysts and data scientists alone is not enough to drive business success. This is because, in most cases, integrating data from various sources requires working with APIs, building complex data pipelines to transform data into the required format, and orchestrating these pipelines efficiently—none of which are trivial tasks.

Data quality checks are also critical. While analysts may be able to handle some of these tasks, the ideal candidate is someone deeply involved in the entire data pipeline, from front-end ingestion to back-end storage and processing. A great data engineer is not only technically skilled but also highly curious, constantly seeking better ways to optimize data workflows and ensure seamless integration across systems.

3. Data Scientist / ML Engineer
If your business is stuck reacting to trends instead of anticipating them, a Data Scientist unlocks the next frontier. They deploy machine learning and statistical models to predict outcomes, automate processes, and uncover hidden opportunities. Skipping this role means settling for hindsight—not foresight—leaving competitive advantages untapped. For example, in e-commerce a data scientist can develop recommendation engines that personalize customer experiences, increasing sales and retention. In finance, they can create fraud detection models that analyze transaction patterns in real time to prevent losses. In healthcare, they enable predictive analytics to identify at-risk patients and recommend early interventions.

Why Role-Specific Hiring Matters
A well-structured BI team operates like a finely tuned engine: every part has a purpose. Hiring a BI Engineer without a Data Analyst leaves insights buried in pipelines. Employing a Data Scientist without a BI Developer means predictive models never reach decision-makers. The right mix & match of roles ensures data isn’t just collected—it’s translated into actionable intelligence that drives growth, efficiency, and innovation. It’s important to align hiring with strategy.

Clearly define the team's mandate and responsibilities

It is important that the BI team's mandate and responsibilities are clearly defined and understood by all stakeholders. This will help the team focus on its core objectives and deliver value to the organization. It can play significant role to communicate efficiently the mission of the team within the organization.

The BI team's mandate might be to act as a data hub and support initiatives regardless of teams invloved. Alternatively, it can support specific team(s) because of complex data or lack of analysis skills.

Build strong relationships with other teams

BI teams often work with a variety of teams within an organization, such as sales, marketing, and operations. Building strong relationships with these teams can help the BI team understand their needs and priorities, and ensure that their work is aligned with the organization's goals.

• Clearly communicate the value of the business intelligence team's work: Make sure other teams understand how the insights and data provided by the team can help them achieve their goals and make better decisions.

• Be responsive to other teams' needs: Show that the team is responsive to the needs of other teams by quickly addressing their requests for data and insights.

• Share knowledge: Share the knowledge and skills of the team with other teams. This will help other teams become more self-sufficient in using data and insights to support their work.

• Be transparent: Share the team's process and methodologies to gather, analyze, and present data. This will help other teams understand the context and limitations of the data they are working with.

• Encourage feedback: Encourage other teams to give feedback on the work of the BI team. This will help to improve and better serve the needs of the organization.

• Hold regular meetings: Hold regular meetings between your team and other teams to discuss upcoming projects, share progress and results, and address any concerns or issues.

• Show results and celebrate: Share the results and impact of the business intelligence team's work with the rest of the organization. This will help other teams see the value of the work that the BI team does and how we work together to achieve shared goals.

Communication channels like Slack and Microsoft Teams are really useful to communicate goals, strategy and defined boundaries on how the BI/Data team works. This can significantly reduce the number of questions that your team will receive. You really do not want your team to be DOOMED of useless questions and dashboard requests from everyone. In any case, your data will end up in an excel file, so you have to protect the team and choose wisely when this is necessary or not. Does this sound familiar to you?

Continuous learning

BI teams should be committed to staying up-to-date with the latest technologies and best practices. Encouraging team members to attend conferences, workshops, and training sessions can help the team stay at the forefront of the field.

• Encourage cross-training: Encourage members to learn about the work of other teams, and vice versa. This will help build understanding and appreciation for the work of different teams and the skills they bring to the table.

  E.g Establish Quarterly meeting with the teams that you collaborate the most.

• Create opportunities for learning: Set up regular training sessions, workshops or seminars where BI staff can share their expertise with other teams, or where other teams can share their knowledge with them.

  E.g Establish cycles of training (6 Months) not only for the BI Tool but also for the organization’s data.

• Create a mentorship program: Set up a mentorship program where the BI team can mentor less experienced members of other teams, and vice versa. This will help build trust and understanding between team members.

  E.g Match data team’s members with a corresponding analyst/stakeholder which works for another dept/team.

• Encourage experimentation: Encourage the team to experiment with new technologies, techniques, and methods. This will help everyone learn and grow.

  E.g Give opportunities to line staff members with experimentation and POCs in tools or MVPs in projects.

• Share best practices: Share best practices and lessons learned either between team members or other teams. This will help everyone learn from the successes and failures of others.

  E.g Establish an internal mechanism and mentality which indicates that sharing is caring. Surpass the mentality that sharing is ‘showing off‘ yourself.

• Make use of modern learning technologies: Encourage the use of online resources and digital learning platforms to access training, tutorials and other educational materials.

  E.g Paid learning platforms, DataTalks club etc. Always stay up to date with tech and trends.

• Celebrate success: Recognize and celebrate the successes of both BI and other teams and how they have grown and developed through learning. Retro sessions are a great way to measure the progress.

  E.g Take accountability but don’t forget to celebrate. Have drinks with your team to celebrate success.

The 1% principle is a simple yet powerful concept that has the potential to drive incredible growth in our businesses without requiring a monumental effort. It all comes down to consistently doing just 1% more than expected or required of us.

Deliver value

The most effective way to increase the reputation of a BI team is to consistently deliver value to the organization. This means providing insights and recommendations that drive meaningful business outcomes.

• Lead by example: Lead by example and demonstrate the importance of data-driven decision making and collaboration in your own work.

  E.g Highlight team’s contribution in a company’s significant project. Spread the word. You made an impact. 

• Understand the business: Take the time to understand the business goals, objectives, and strategies of the organization. This will help you to provide insights and data that are directly relevant to the business.

  E.g Learn the business. Everything from everyone. Be curious. The team is there to help the business and not to work on academia datasets like IRIS.

• Provide actionable insights: Provide insights and data that are actionable, meaning they can be used to make decisions and take actions that will drive the business forward.

  E.g Match insight with actionable. Otherwise, it’s not an insight. Quarter over Quarter sales decreased by 10%, the team discovered that email open rate decreased by 30%. We should explore the reason behind it and try to lift it. 

• Be timely: Provide insights and data in a timely manner, so that they can be used to support decision-making and inform the direction of the business.

  E.g Speed vs Quality is always the case. This is not a research paper, but you also have to do it correct. Ensure data quality and deliver. No, you don’t need this super fancy visualization or that 15 page pdf document. There’s an entire suite of KPIs for data team performance that includes metrics such as time of request to insight, insight to action etc.

• Communicate effectively: Communicate the insights and data provided by the BI team in a clear and concise manner, so that they can be easily understood and acted upon by decision-makers.

  E.g One of the most underrated skills is data communication. You may code like a pro hacker, creating the best analyses or ML model. If you cannot communicate to the C-Level suite or the Board Of Directors, the impact of this work, it may ends up the Project graveyard. As you climb the ladder, soft skills make the difference. Describe the beginning, the middle and the end. 

  Use the right tools: Use the right tools and technologies to collect, analyze, and present data. This will help to deliver insights and data in an efficient and effective manner.

  E.g Experiment. Your software engineers may prefer to code in cloud and Terraform needing 1 week and 10 hours per month for maintenance and development, while databricks can deliver value in 5 minutes with a cheap solution.

• Be flexible: Be flexible and responsive to changing business needs and priorities. In this way, you can deliver value to the business in a timely and effective manner.

  E.g Things change. The world changes. Adapt and adopt. You may need to change your tech stack or the priorities. Listen actively, process, act. You may need to kill your project, yes. In project management, kill switches and culling exercises are essential tools for maintaining efficiency and minimizing risks. A kill switch is a predefined trigger that prompts a reassessment of a project's viability, preventing unnecessary investment in failing initiatives. For example, in a software development project facing persistent delays and budget overruns, invoking the kill switch allows stakeholders to halt further losses and redirect resources.

• Measure and report on the results: Measure and report on the results of the BI team's work in terms of how it has helped the business to achieve its goals and objectives.

  E-commerce Sales Performance Dashboard: Developed a real-time dashboard tracking sales, customer acquisition costs, and product performance.

  • Identified underperforming products, leading to targeted promotions that increased sales by 12%.

  • Optimized pricing strategies, improving revenue per customer by 8%.

  Automating Financial Reporting: Built an automated financial reporting pipeline, reducing the need for manual data extraction and reconciliation.

  • Saved 200+ labor hours per month, allowing finance teams to focus on analysis rather than data gathering.

  • Reduced reporting errors by 30%, improving decision accuracy.

  Churn Prediction Model in a Subscription Business: Developed a machine learning model to identify customers likely to churn based on usage patterns and feedback.

  • Implemented targeted retention campaigns, reducing churn from 10% to 7% in six months.

  • Increased lifetime customer value by 15% through personalized engagement strategies.
    

  Reducing Inventory Costs with Predictive Analytics: Built a demand forecasting model to optimize stock levels across warehouses.

  • Reduced excess inventory by 20%, cutting storage costs.

  • Improved stock availability, reducing lost sales due to stockouts by 10%.

  Workforce Productivity Dashboard in Customer Support: Developed a dashboard monitoring ticket resolution times, agent workload, and customer satisfaction.

  • Optimized staffing schedules, reducing average resolution time from 5 hours to 3 hours.

  • Increased customer satisfaction scores by 10 points due to faster responses.

Great analysis starts with great awareness. Be mindful, be DataConscious.

Liked this article? Make sure to ❤️ click the like button.

Community needs your voice, do you have something to add? Feel free to comment.

Know someone that would find this helpful? Feel free to share this post.