Subscribe now

We will develop each concept step by step, connecting the underlying mathematics to an implementation we can inspect and experiment with. We will demonstrate the algorithms in both Python, using MLX and NumPy, and R, using base R. We will share the concepts and examples across all three implementations, so you can follow the approach that fits your background or compare implementations across tools. Along the way, we will write the calculations that train each model and produce predictions, using numerical building blocks instead of ready-made model-fitting functions.

Why build from scratch?

Machine learning libraries can hide much of what happens between providing data and receiving a prediction. By building an algorithm from scratch, we can inspect its core components: how it represents data, measures error, learns its parameters, and produces predictions. This gives us a clearer view of how the mathematics translates into the algorithm’s behavior.

That visibility helps us understand what affects the algorithm’s performance and which changes may improve its results. We can also examine its assumptions, identify the types of problems it handles effectively, and recognize the conditions in which it may perform poorly or fail.

Building from scratch does not mean recreating every numerical operation. We will use building blocks such as arrays, matrix multiplication, and equation solvers. However, we will implement the logic that trains each model and produces its predictions instead of delegating those steps to ready-made model-fitting functions.

From data to a prediction

Consider a dataset of apartments. Each apartment is an observation. Its floor area is an input feature, and its monthly rent is the target we want to predict. A model describes a relationship between the input and the prediction.

A simple model predicts rent by multiplying area by a slope and adding an intercept. The slope and intercept are its parameters. With an illustrative slope of 20 and intercept of 500, an area of 50 square meters gives a predicted monthly rent of 1,500 currency units: 20 × 50 + 500. This calculation applies chosen parameters; it does not tell us how to choose them.

Training, or fitting, uses data to determine parameters. A loss function quantifies prediction errors and gives us an objective for fitting. Prediction applies the fitted model to inputs. Evaluation checks how useful its results are, including on observations held out from fitting. Small training errors alone do not establish that it will predict new rents well.

Learning from inputs paired with targets is supervised learning. Predicting a numerical value is regression; predicting a category, such as whether an email is spam, is classification. Unsupervised learning looks for structure without supplied target labels. Clustering apartments by their features is one example. These distinctions are covered in Google’s introduction to machine learning.

One article, three implementation tracks

Each coding tutorial will pair one shared article with three implementations, so we can follow one track without learning three tools at once.

• Python with NumPy: NumPy provides arrays and numerical operations, making this a natural starting point for readers familiar with Python.

• Python with MLX: MLX is Apple ML Research’s array framework and provides another Python path for exploring the same algorithms.

• Base R: Base R provides built-in vectors, matrices, and functions for readers familiar with R.

For setup, start with the NumPy installation guide, MLX installation requirements, and R downloads. The next tutorial, Vectors, Matrices, and Linear Algebra, will provide the chosen track’s setup and execution steps. Installing the other tracks is unnecessary.

The companion repository will hold the supporting code. A notebook combines explanations, executable code cells, and outputs; an annotated script presents code with explanatory comments. We will read the article, then run the chosen implementation from the beginning. Tracks will use shared fixtures, stored inputs for reproducible checks, and compare results within documented numerical tolerances.

The learning path

We will begin with the foundations: vectors, matrices, loss functions, gradients, and data preparation. With those pieces in place, we will build regression models and see how methods such as gradient descent learn their parameters. We will then move into classification before exploring trees, ensembles, and a margin-based classifier. Clustering and dimensionality reduction will introduce unsupervised learning. Finally, we will bring these ideas together in a small neural network and compare representative models on suitable tasks.

Along the way, we will introduce the mathematics when it becomes useful and focus on one model or problem at a time. A separate, future Statistics with Tensors series will cover large-scale batched statistics.

Summary

We now have a reading guide: one shared explanation, a chosen implementation track, and explicit fitting and prediction logic built from numerical primitives. Training, prediction, and evaluation will give us a common way to examine each model.

In the next tutorial, Vectors, Matrices, and Linear Algebra, we will set up one track and use small arrays and matrices to express the numerical operations needed by later models.

• Open Source of the Week - DuckPad

• New learning resources - Kimi K3.1 Pro, agent harnesses, OpenCode 2.0, local AI agents on Ubuntu, and prefill-only decision models

• Book of the week - AI-Assisted Python for Nonprogrammers by Reuven M. Lerner

The newsletter is available on LinkedIn and Substack.

Open Source of the Week

This week’s focus is on the DuckPad project.

DuckPad is an open-source SQL workspace powered by DuckDB 2.0. It turns data sources, joins, reusable components, and SQL transformations into visual pipelines you can explore and version. Rather than treating a result as the end of the workflow, DuckPad lets users select a cell and trace the source values that contributed to it. You can store projects in Git, develop on independent branches, compare them visually, and review changes before merging. The project runs on your own infrastructure and supports environment-specific connections and team access controls.

Project repo: https://github.com/hugues31/duckpad

Key Features

• Visual pipelines — connect data sources, joins, and reusable components in a graphical workspace.

• Integrated SQL editor — use guided forms or write SQL directly when defining transformations.

• Cell-level lineage — trace a result cell back to the source values that contributed to it.

• Git versioning — save projects, work on separate branches, and compare pipeline changes visually.

• Merge review — inspect proposed changes before applying them to a project.

• Self-hosted operation — run DuckPad on your own infrastructure and configure connections by environment.

• Project access — manage access for teams working in the same DuckPad deployment.

More details are available in the project documentation.

License: MIT

Subscribe now

New Learning Resources

Here are some new learning resources that I came across this week.

The Best Open Model Right Now? (Kimi K3.1 Pro)

This video from NeuralNine examines Kimi K3.1 Pro through published benchmarks and three coding tasks: a CNN training visualizer, thumbnail asset generation, and a MOBA game prototype.

What Is an Agent Harness? How AI Coding Agents Work

The following visual explainer from AI Systems with Andy breaks an agent harness into its loop, tools, context, memory, and controls. It also covers the execution cycle, context management, permissions, agents versus workflows, and a minimal Python implementation.

Harness² Explained: Self-Improving Agents Without Training

This video from Prompt Engineering examines Harness², a research framework that enables an agent to modify its prompts, skills, scripts, subagents, and control-loop hooks for each task without retraining the model. It explains the Propose, Probe, and Compose process, its experience library, comparisons with coding agents, and the approach’s limitations.

OpenCode 2.0: Biggest Update Ever (Free + Local Models)

This video from Leon van Zyl tests OpenCode 2.0 by building a bill splitter application. It covers free models, sessions shared across the terminal and desktop app, background agents, MCP code mode, Git worktrees, custom plugins, and running local models with Ollama.

Jev Course – Prefill-Only Inference & Decision Models from Scratch

This course from freeCodeCamp builds Jev-style decision models with PyTorch and Hugging Face. It covers encoder- and decoder-based architectures, custom causal attention masks, KV-cache reuse, teacher-student distillation, and reinforcement learning for structured choices, scores, and Boolean outputs.

Building a Local AI Agent in Workshop on Ubuntu

The following tutorial from Canonical Ubuntu builds a local coding agent with Canonical Workshop and the Qwen3-Coder inference snap. It demonstrates an isolated development environment, a tunnel to the locally hosted model, and coding tasks that run without an internet connection.

Agent Harness for Learning Things

This video by Eero Alvar presents LearnHarness, an agent-based system for personalized learning. It demonstrates how the system probes existing knowledge, plans a dependency path, teaches and practices in small steps, and stores propositions and reasoning chains in a persistent knowledge graph.

Book of the Week

This week’s focus is on a new book — AI-Assisted Python for Nonprogrammers by Reuven M. Lerner. The book introduces Python to readers without previous programming experience while using AI chatbots as learning partners. It builds from the concepts behind computers and programming through core Python syntax, data structures, functions, modules, and packages. Focused exercises use approaches such as strategy sessions, debugging prompts, interviews, and reverse Socratic questioning to reinforce the material. The final chapters connect these foundations to AI-assisted and agentic coding tools while emphasizing the programming knowledge needed to evaluate and maintain generated code.

Topics Covered

• Python foundations — programming concepts, installation, IDEs, and JupyterLite.

• Values and variables — data types, assignment, input, and formatted strings.

• Control flow — conditions, comparisons, Boolean logic, and loops.

• Core data structures — strings, lists, tuples, and dictionaries.

• Files — reading, writing, parsing, and converting text into data.

• Functions — parameters, return values, documentation, scope, and arguments.

• Modules and namespaces — imports, aliases, and Python’s standard library.

• Package management — evaluating PyPI packages and managing projects with uv.

• AI-assisted coding — pair programming with chatbots and GitHub Copilot.

• Agentic coding — working with Claude Code and OpenCode while considering code quality and maintenance.

This book is ideal for beginners and nonprogrammers who want to learn Python fundamentals while developing practical habits for working with AI coding tools.

The book is available on O’Reilly and Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Pixeltable

• New learning resources - Qwen RL experiments, DeepSeek V4.1 Flash 1017, terminal development setup

• Book of the week - Data Engineering for Multimodal AI by Vasundra Srinivasan

The newsletter is available on LinkedIn and Substack.

Subscribe now

Open Source of the Week

Pixeltable is an open-source Python project for building AI applications with images, video, audio, documents, and structured data. It combines a multimodal database, data processing, vector search, and API serving. Transformations and model calls are defined as computed columns, with their results stored alongside the source data. Pixeltable tracks column dependencies and processes new rows as they arrive. The goal is to reduce the code needed to keep media, metadata, model outputs, and search indexes in sync.

Project repo: https://github.com/pixeltable/pixeltable

Key Features

• Multimodal tables — work with images, video, audio, and documents alongside ordinary columns.

• Computed columns — define transformations and model calls as part of a table.

• Incremental processing — process new data and recompute affected results without repeating unrelated work.

• Vector search — declare embedding indexes that the table maintains as data changes.

• Media processing — build workflows for document chunks, video frames, and audio transcripts.

• Versioning — inspect table history and revert changes.

• API serving — declare HTTP routes alongside tables using FastAPI integration.

Here is an example by Alison Presmanes Hill, PhD builds a photo search application with Pixeltable using six retrieval approaches. It starts with keyword search, then adds LLM query rewriting, hybrid search with BLIP captions and CLIP embeddings, automatic captioning and indexing of new images, metadata filtering, and comparison of two embedding models. The walkthrough also covers query decomposition and serving the application over HTTP. It demonstrates how the retrieval strategy can evolve while Pixeltable manages the underlying data and indexes.

More details are available in the project documentation.

License: Apache-2.0

New Learning Resources

Here are some new learning resources that I came across this week.

Tuning LLM with RL

This video from Prompt Engineering tests whether reinforcement learning can move Qwen3.5-4B closer to Qwen3.5-27B on held-out tasks. It covers LoRA training, GRPO, strict checkers for reward hacking, task selection, and combining model distillation with reinforcement learning.

DeepSeek V4.1 Flash 1017 Explained in 15 Minutes

The following video by Bruno Vega examines DeepSeek V4.1 Flash 1017, including its effort-conditioned output-length penalty, benchmark caveats, token costs, and local inference implications. It also discusses DeepSeek Harness and the Search & Fetch API.

Great Terminal Setup

This walkthrough from DevOps Toolbox presents the free and open-source tools that make up the creator’s terminal workflow. It covers the terminal and multiplexer, Workmux, shell configuration, coding agents, navigation, command history, prompts, the IDE, file exploration, and development environment setup.

Book of the Week

This week’s focus is on a new book — Data Engineering for Multimodal AI by Vasundra Srinivasan. The book explores the infrastructure behind multimodal AI, from data architecture and ingestion to retrieval and production operations. It treats assembling useful context from different data types as a data engineering problem. Each chapter includes a practical playbook that readers can extend, connecting the concepts to an implementation. The focus is on building and operating the pipelines that supply multimodal models with reliable data, with examples spanning cloud-native and on-premises environments.

Topics Covered

• Data architecture — semantic layers, data contracts, and multimodal storage patterns.

• Ingestion and transformation — validation and processing across data modalities.

• Feature engineering — feature extraction, fusion, and multimodal feature stores.

• Storage and retrieval — vector search, knowledge graphs, and hybrid retrieval.

• Streaming pipelines — event alignment, adaptive processing, and edge computing.

• MLOps — versioning, testing, monitoring, and automated retraining.

• Scaling and serving — distributed processing, inference deployment, and cost management.

• Governance and security — lineage, privacy, and responsible AI practices.

This book is ideal for data engineers, ML engineers, and MLOps practitioners who want to build and maintain data infrastructure for multimodal AI applications.

The book is available on O’Reilly and Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

question → prompt → model-generated SQL → DuckDB → result

The complete workflow has several setup steps, but the agent at its center is only a few lines of Python. Keeping this first implementation small will help us see exactly what each component contributes before we add more context, safety controls, and reliability mechanisms later in the series.

This tutorial series is based on my LinkedIn Learning course, Build with AI: SQL Agents with Large Language Models. If you prefer video-based learning, please check out the course on LinkedIn Learning.

By the end of this tutorial, we will know how to:

• Connect to an existing DuckDB database through Ibis.

• Extract table and column information for model context.

• Send system instructions and a user question through the OpenAI Responses API.

• Extract SQL from the model response.

• Execute the generated SQL against DuckDB.

• Combine these steps into a small reusable SQL AI agent.

A reproducible example of the code in this tutorial is available in the supporting Jupyter notebook.

Subscribe now

Prerequisites

To run the code in this tutorial, we will need a configured Python environment and a DuckDB database. The following tutorials cover these prerequisites:

• Python environment: Follow Set up a Python environment for SQL AI agents to create the environment and install the dependencies declared in docker/requirements.txt.

• DuckDB settings: Follow Prepare data for a SQL AI agent with DuckDB and Ibis to create the sql-ai-agent.duckdb database and its air_traffic table.

The tutorial also requires two environment variables. DUCKDB_DATABASES_PATH points to the directory containing the database file, and OPENAI_API_KEY contains an OpenAI API key:

export DUCKDB_DATABASES_PATH="$HOME/databases/duckdb"
export OPENAI_API_KEY="your_api_key_here"

Note: This tutorial uses the OpenAI API for the model request, but the overall workflow is not tied to one provider. We can adapt it to Google Gemini, Anthropic, or another provider by replacing the provider-specific client, credentials, and request and response handling. The surrounding steps—assembling context, executing the generated SQL in DuckDB, and returning the result—remain the same.

Keep the API key in the environment rather than writing it directly in the notebook. With the environment variables set and the database ready, we can assemble the first end-to-end agent.

Understand the minimal SQL AI agent workflow

At a high level, the workflow begins with a user question. Before sending it to the large language model (LLM), we combine the question with useful context that can help the model generate the correct SQL query. As we discussed in Context in SQL AI Agents: The Three Layers Behind Reliable Answers, clearer and more complete context reduces ambiguity and gives the model fewer details to guess, making its response more predictable.

First, the model needs the table schema. The schema contains the available column names and their data types, such as integer, text, Boolean, or date. This information helps the model refer to the correct columns and use them according to their types. We will also tell the model that the database uses DuckDB and instruct it to return only the SQL query.

These are simple ways to derive context from our database. In later tutorials, we will explore more advanced methods for generating and enriching that context.

Once the prompt is ready, we send it to the LLM. The model returns a SQL query, which our Python code extracts and sends to DuckDB. DuckDB executes the query and returns the resulting table to Python, where we can present it to the user. Figure 1 illustrates this workflow, including how the table schema and user question enter separate prompts before the model request.

We begin by importing the modules used throughout the notebook:

import os
import re

from pathlib import Path

import ibis
from openai import OpenAI

We use os to read environment variables, Path to construct the database path, re for a small response-parsing helper, Ibis to connect to DuckDB, and the OpenAI Python SDK to send the model request.

Connect to DuckDB and extract the schema

The previous tutorial created a persistent sql-ai-agent.duckdb file. Rather than loading and preparing the source CSV again, we reconnect to that database and reuse its air_traffic table.

The following code reads DUCKDB_DATABASES_PATH, constructs the database file path, and raises a clear error if the directory variable or file is missing:

database_directory = os.environ.get("DUCKDB_DATABASES_PATH")
if not database_directory:
    raise EnvironmentError(
        "Set DUCKDB_DATABASES_PATH to the directory used in the previous tutorial."
    )

database_path = Path(database_directory).expanduser() / "sql-ai-agent.duckdb"
if not database_path.is_file():
    raise FileNotFoundError(
        f"Could not find {database_path}. Run 03a_duckdb_settings.ipynb first."
    )

Next, we connect Ibis to the DuckDB file and confirm that the expected table is available:

connection = ibis.duckdb.connect(database_path)
table_name = "air_traffic"

if table_name not in connection.list_tables():
    raise RuntimeError(
        f"Could not find the {table_name!r} table. Run the previous tutorial first."
    )

print(f"Connected to {database_path.name}")
print(f"Available table: {table_name}")

The verified notebook returns:

Connected to sql-ai-agent.duckdb
Available table: air_traffic

The model cannot inspect this connection directly, so the table name alone is not enough. It also needs the available column names and their data types. We can retrieve those details from DuckDB’s information_schema.columns view.

The query selects the column name and DuckDB data type for air_traffic. The ordinal_position ordering preserves the table’s column order:

schema_sql = f"""
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = '{table_name}'
ORDER BY ordinal_position
"""

schema_details = connection.sql(schema_sql).execute()
schema_context = ",\n".join(
    f'    "{row.column_name}" {row.data_type}'
    for row in schema_details.itertuples(index=False)
)

print(schema_context)

This produces a compact schema description that we can insert into the model instructions:

    "Year" BIGINT,
    "Date" TIMESTAMP,
    "Operating Airline" VARCHAR,
    "Operating Airline IATA Code" VARCHAR,
    "Published Airline" VARCHAR,
    "Published Airline IATA Code" VARCHAR,
    "GEO Summary" VARCHAR,
    "GEO Region" VARCHAR,
    "Activity Type Code" VARCHAR,
    "Price Category Code" VARCHAR,
    "Terminal" VARCHAR,
    "Boarding Area" VARCHAR,
    "Passenger Count" BIGINT

At this point, our Python process knows where to execute a query, and we have a text representation of the schema for the model. The schema identifies the available fields and their types, but it does not describe distinct values, business definitions, or relationships with other tables. Those are separate context problems that we will address later in the series.

Set up OpenAI and build the prompts

The OpenAI client needs an API key before it can send a request. We use os.getenv() to load OPENAI_API_KEY, check that it is available, and pass it to OpenAI():

openai_api_key = os.getenv("OPENAI_API_KEY")
if not openai_api_key:
    raise EnvironmentError("Set OPENAI_API_KEY before running the notebook.")

client = OpenAI(api_key=openai_api_key)
model = "gpt-5.4-mini"

Loading the key from the environment keeps the credential out of the notebook source and its saved outputs. We store the selected model name separately so it is easy to identify or change without editing the request code.

Before building the prompts, it helps to separate their roles. The system prompt contains the reusable instructions and context that define the model’s task: generate DuckDB SQL, use the supplied schema, and follow the required output format. The user prompt contains the specific natural-language question we want to translate into SQL. Keeping them separate lets us reuse the same instructions and schema context with different questions.

With the Responses API, we pass the system-level guidance through instructions and the user prompt through input. The instructions take priority over the input, so they establish the rules the model should apply to the question. The OpenAI prompt-engineering guide describes these roles in more detail.

Next, we define the instructions that describe the model’s task:

system_prompt = f"""
Translate the user's question into one DuckDB SQL query.
Return only the SQL query as plain text, without Markdown or an explanation.
Use only the table and columns shown below.
Enclose table and column names in double quotes.
Write a SELECT query that reads data; do not modify the database.

CREATE TABLE "{table_name}" (
{schema_context}
);
""".strip()

The prompt supplies five pieces of context:

• The task is to translate a question into SQL.

• The target dialect is DuckDB SQL.

• The output should contain only one plain-text query.

• The query should read rather than modify data.

• The model may use only the supplied table and columns.

We represent the schema with a familiar CREATE TABLE shape because it keeps the table name, field names, and data types together. This string is context for generation; it does not create or change the real table.

For the first question, we ask for one straightforward aggregation:

question = "What was the total passenger count in 2024?"
user_prompt = f"Write a SQL query that answers: {question}"

The instruction to write a SELECT query guides the model, but it is not an enforced database rule. We will return to that limitation after the query runs.

Generate, extract, and execute the SQL

We now have the client, model name, system instructions, and user question. The next call sends them to the Responses API:

response = client.responses.create(
    model=model,
    instructions=system_prompt,
    input=user_prompt,
)

model_output = response.output_text
print(model_output)

The SDK’s output_text helper collects the text output into one string. In the verified run, the model returned:

SELECT SUM("Passenger Count") AS "total passenger count"
FROM "air_traffic"
WHERE "Year" = 2024;

The query uses the schema context correctly: it selects Passenger Count from air_traffic, filters Year to 2024, and uses SUM to calculate the total.

Model output is nondeterministic, so another successful run may use different spacing or a different alias while expressing the same query. The system instructions ask for plain SQL, but a model may still wrap the query in a Markdown fence. We add a small helper to handle that presentation difference:

def extract_sql(text):
    fenced_sql = re.search(
        r"```(?:sql)?\s*(.*?)```",
        text,
        flags=re.IGNORECASE | re.DOTALL,
    )
    sql = fenced_sql.group(1) if fenced_sql else text
    sql = sql.strip()

    if not sql:
        raise ValueError("The model did not return a SQL query.")

    return sql


generated_sql = extract_sql(model_output)
print(generated_sql)

extract_sql() looks for an optional fence beginning with either three backticks or three backticks followed by sql. When a fence exists, the regular expression returns its contents. Otherwise, the helper uses the model text as-is. It then removes surrounding whitespace and rejects an empty response.

This helper is a parser, not a SQL validator. It does not check whether the text contains one statement, whether that statement is read-only, which tables it references, or how much work the query could perform.

With the text extracted, we pass the query to the same Ibis connection and ask DuckDB to execute it:

result = connection.sql(generated_sql).execute()
result

The verified data snapshot returns:

  total passenger count
0              52210939

The total passenger count recorded for 2024 is 52,210,939. At this point, the complete path works: the question reached the model, the model returned SQL, and DuckDB returned a result.

Combine the steps into a reusable SQL AI agent

The previous sections separated each operation so we could inspect it. We can now wrap the model request, SQL extraction, and database execution in one function:

def sql_ai_agent(question):
    response = client.responses.create(
        model=model,
        instructions=system_prompt,
        input=f"Write a SQL query that answers: {question}",
    )
    sql = extract_sql(response.output_text)
    result = connection.sql(sql).execute()

    separator = "-" * 72
    print(f"Question:\n{question}\n")
    print(f"Generated SQL:\n{separator}\n{sql}\n{separator}")
    print("\nResult:")
    print(result.to_string(index=False))

    return sql, result

The printing makes the notebook output easy to inspect, while the returned (sql, result) tuple remains available for later code. The three lines at the center of the function are the minimal agent loop:

response = client.responses.create(...)
sql = extract_sql(response.output_text)
result = connection.sql(sql).execute()

The connection, schema, system instructions, and client are prepared once. For each question, the function then asks the model for SQL, prepares the text, and sends the query to DuckDB.

We run the complete workflow with the same question:

output = sql_ai_agent(question=question)

The verified notebook prints:

Question:
What was the total passenger count in 2024?

Generated SQL:
------------------------------------------------------------------------
SELECT SUM("Passenger Count") AS "total_passenger_count"
FROM "air_traffic"
WHERE "Year" = 2024;
------------------------------------------------------------------------

Result:
total_passenger_count
             52210939

That’s it. We have built a working SQL AI agent from scratch with a small Python function.

The implementation is intentionally minimal. The prompt includes only basic schema context, and extract_sql() handles formatting without validating the query. Most importantly, the function sends model-generated SQL directly to the database. Asking the model to return a SELECT statement does not prevent it from returning something else, and the current connection does not provide an independently enforced read-only boundary.

That makes this implementation useful for understanding the complete workflow, but not suitable as a production safety pattern. Later tutorials will add richer context, a stronger output contract, SQL validation, bounded repair, read-only permissions, and evaluation.

Summary

In this tutorial, we built the first complete SQL AI agent in the series. We reconnected to the persistent DuckDB database created in the prerequisite, confirmed that air_traffic was available, and extracted its ordered column names and data types from information_schema.columns.

We then initialized the OpenAI client with an API key loaded from the environment. We created system instructions containing the DuckDB task and table schema, paired them with a natural-language question, and sent both through the Responses API. The model returned a SQL aggregation, which we extracted and executed through Ibis. DuckDB returned a total passenger count of 52,210,939 for 2024.

Finally, we combined the model request, response parsing, and SQL execution in sql_ai_agent(). The result is deliberately small: a question goes in, SQL is generated, DuckDB executes it, and a result comes back.

This first version uses the minimum schema information required for the example. In the next tutorial, Add Schema and SQL Dialect Context, we will turn that inline description into a more deliberate and reusable context layer. That improvement will help the model understand the database more reliably as our questions and schemas become more complex.

References

• Supporting notebook: 04_simple_sql_ai_agent.ipynb

• OpenAI text generation and Responses API guide

• DuckDB information schema documentation

• Ibis DuckDB backend documentation

• Open Source of the Week - Glyf

• New learning resources - system design, local inference, TabFM, decision models, n8n automation

• Book of the week - Financial AI in Practice: A playbook for credit, fraud, and investment systems by Taehun Kim

The newsletter is also available on LinkedIn and Substack.

Subscribe now

Open Source of the Week

This week’s focus is on the Glyf project.

Glyf is a new open source project from that adds a code-first visualization build step to a dbt project. Rather than defining charts in a browser-based BI interface, Glyf keeps chart definitions in SQL and dashboards in YAML alongside the data models that power them. It reads dbt artifacts, resolves model references, validates chart SQL, executes it against the warehouse, and renders charts with Altair. The build produces a static site and chart artifacts that can be reviewed, versioned, and published through an existing delivery workflow. The goal is to make visualizations a declared part of the data pipeline rather than a separate downstream asset.

Project repo: https://github.com/glyf-data/glyf

Key Features

• dbt-aware chart definitions — resolves ref() and source() references from the dbt manifest before rendering.

• GGSQL visualizations — combines SQL queries with chart instructions to define visualizations in source files.

• YAML dashboards and Python macros — composes charts, layout, labels, thresholds, and reusable components as reviewable project files.

• Build-time validation — checks dbt artifacts, chart files, and SQL execution before a complete build.

• Static publishing artifacts — produces rendered PNG and SVG charts plus a self-contained static site for deployment.

• Warehouse support — runs through the dbt profile with DuckDB, Trino, Snowflake, and BigQuery listed as supported adapters.

• Data-exposure controls — includes row-data modes, PII deny or redact rules, value scanning, per-audience builds, and build provenance.

More details are available in the project documentation.

License: Apache 2.0

New Learning Resources

Here are some new learning resources that I came across this week.

Design Software Systems Like a Senior Engineer – Complete Course

This complete course from freeCodeCamp.org focuses on designing software systems from a senior engineer’s perspective.

Qwen 3.8 27B Just Hit 200 Tokens/s on Mac M5

This video from Cloud Codes examines Lithos AI’s lithos-metal inference engine and its Qwen 3.8-27B benchmark on an M5 Max Mac. It contrasts Metal megakernels and speculative decoding with standard decoding, and distinguishes measured from projected token-per-second results.

Google’s GPT-3 Moment for Databases & Spreadsheets (TabFM)

The following video from Prompt Engineering introduces Google’s TabFM, a zero-shot foundation model for tabular classification and regression. It covers TabFM-Auto, a customer-churn workflow, and how data leakage can produce misleading predictions.

Ollama Decision Models: The End of Jev?!

This video by NeuralNine looks at using Ollama to run local decision models that return structured classifications, selections, or scores. It compares that approach with the hosted Jev product and considers tradeoffs around routing tasks, cost, privacy, and access to inference.

n8n Full Course – Architect Scalable AI Automations from Scratch

This full course from freeCodeCamp.org, taught by Rahul Joshi, builds n8n automation workflows ranging from personalized emails and CRM chatbots to document extraction and lead qualification. It covers AI agents, persistent memory, human approval, error handling, and operations monitoring.

Book of the Week

This week’s focus is on a new book — Financial AI in Practice: A playbook for credit, fraud, and investment systems by Taehun Kim. The book follows financial AI projects from data preparation and modeling through deployment, monitoring, and business decisions. Its four-layer framework connects data assets, modeling, strategy and monitoring, and application delivery. The examples span credit scoring, fraud detection, investment analysis, and customer personalization, with explainability and regulatory constraints considered throughout. The goal is to help readers understand how models fit into the wider systems that use their predictions.

Topics Covered

• Financial AI architecture — connecting data, models, decision rules, and application workflows

• Credit scoring — preparing data, engineering features, and deploying an XGBoost pipeline

• Model monitoring and explainability — tracking changes in data and explaining credit decisions

• Fraud detection — combining supervised classifiers with autoencoders for anomaly detection

• Graph analytics — using graph neural networks to identify patterns in transaction networks

• Investment analysis — pairing quantitative models with LLM-based news analysis

• Retrieval-augmented generation — grounding portfolio analysis in selected sources and recording decisions

• Personalization and agents — building customer segments and retention workflows with human approval

• Business strategy and governance — evaluating ROI, model risk, and operational tradeoffs

This book is ideal for data scientists, ML engineers, financial analysts, and technical leaders who want to build financial AI applications and understand the operational decisions surrounding them.

The book is available on the publisher's website and Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Regularized Recursive Self-Improvement

• New learning resources - MLA, Fledge Alpha, Jev decision models, and REST API design

• Book of the week - AI Agents with MCP by Kyle Stratis

The newsletter is also available on LinkedIn and Substack.

Subscribe now

Open Source of the Week

This week’s focus is on the RRSI project. RRSI, or Regularized Recursive Self-Improvement, is an open-source project from Google Research for evolving the harness around a frozen LLM agent. The harness includes the prompts, control flow, configuration, context management, tools, skills, memory, and subagents that shape how the model operates.

Rather than optimizing the harness against a fixed task set without constraints, RRSI regularizes the search process to reduce overfitting. It limits how many independent edits a candidate can bundle, uses the full edit history to avoid repeating failed hypotheses, and redirects stalled searches toward components that have not been explored. A critic screens candidates for task-specific logic before evaluation, while noise- and cost-aware selection rules reject gains that are too small or too expensive.

Project repo: https://github.com/google-research/rrsi

Key Features

• Open harness editing — modify prompts, control flow, tools, skills, memory, subagents, and context management

• Regularized proposals — control the number of bundled edits and use prior edit history to guide new candidates

• Leakage screening — review candidates for benchmark-specific logic before running evaluations

• Noise-aware selection — require improvements to exceed the measured evaluation variance

• Cost-aware decisions — accept additional inference tokens only when they produce a measured gain

• Three domain adapters — apply the same loop to terminal tasks, document work, and engineering design

• Auditable experiments — isolate candidates in Git worktrees and record hypotheses, score changes, costs, and verdicts

More details are available on the project website.

License: Apache 2.0

New Learning Resources

Here are some new learning resources that I came across this week.

Multi-Head Latent Attention Explained Visually: DeepSeek’s Secret to 93% Less GPU Memory

This visual explainer from Visual AI covers attention during prefill and decoding, the KV-cache memory bottleneck, and the progression from multi-head attention to MQA, GQA, and DeepSeek’s multi-head latent attention. It also explains low-rank latent compression and decoupled RoPE.

Fledge Alpha Is Here – And It’s Fast

This investigation by Bruno Vega and Kai examines the anonymously released Fledge Alpha model in OpenCode Zen. It reviews the model’s published specifications, the models.dev pull-request history, evidence that may connect it to Thinking Machines’ Inkling family, early coding tests, and the privacy tradeoff of using the free preview.

What Is Jev and How to Use It with an LLM? (Free Lab)

The following video from KodeKloud introduces Jev as a non-generative decision model that complements LLMs. It covers the state-and-questions API, binary, choice, and score outputs, and applications such as model routing, human-review thresholds, safety checks, and policy compliance, with a free hands-on lab.

Open-Source Jev Now Runs Locally

This walkthrough by Samuel Gregory explores running Jev-style decision models locally. It explains decision models, compares local options including Nimble and Tev1 with cloud-hosted Jev and Claude, and closes with a real-world example and benchmark comparison.

8 Must-Know REST API Design Principles for Developers

This video from Visualcoders presents eight patterns for APIs that need to support changing clients and larger workloads. It covers resource-oriented design, pagination, filtering and field selection, idempotency, backend-for-frontend, API gateways, asynchronous APIs, and versioning.

This is how to train Decision Language Models (like JEV) from scratch on custom tasks!

This long-form tutorial from Neural Breakdown with AVB builds a System One decision model inspired by Jev. It covers choice, score, and noul question types, bi-encoder versus joint-encoder design, a Qwen-based network and decision head, choice-order invariance, a live PyTorch implementation, dataset construction, training, and results.

Book of the Week

This week’s focus is on a new book — AI Agents with MCP by Kyle Stratis. The book explains how the Model Context Protocol connects AI applications with tools, data, prompts, and external services through a standardized interface. It moves from MCP’s architecture and connection lifecycle to hands-on client and server development with the Python SDK, then covers the operational concerns involved in using MCP beyond a local prototype.

Topics Covered

• Agentic AI and MCP — understanding agents, common use cases, and MCP’s role in agentic systems

• Protocol architecture — working with hosts, clients, servers, transports, and the connection lifecycle

• MCP clients — connecting over stdio and Streamable HTTP and consuming tools, resources, and prompts

• Advanced client capabilities — supporting sampling, roots, elicitations, OAuth, registries, and multiple servers

• MCP servers — building servers with the Python SDK and exposing tools, prompts, and resources

• Server utilities — adding logging, progress notifications, pagination, cancellation, and client-provided capabilities

• Testing and evaluation — debugging with MCP Inspector, automating tests, and evaluating server behavior

• Security — identifying MCP vulnerabilities and applying architectural and framework-level safeguards

• Transports and messaging — using JSON-RPC, session objects, stdio, Streamable HTTP, and custom transports

• Ecosystem and extensions — exploring the MCP Registry, MCP Apps, authentication extensions, MCP Tasks, and alternative frameworks

This book is ideal for intermediate to advanced Python developers, AI engineers, and software architects who want to build and operate MCP clients, servers, and agent workflows with a systems-level understanding of the protocol.

The book is available on the publisher’s website and on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - ShinyReact

• New learning resources - LangSmith, specialist agents, software factories, and building agents in Python

• Book of the week - RAG with Python Cookbook by Dominik Polzer

The newsletter is also available on LinkedIn and Substack.

Open Source of the Week

This week’s focus is on the ShinyReact project. ShinyReact is an open-source project from Posit that connects a React frontend to a Shiny server written in Python or R. It provides the communication layer but no UI components, leaving the application interface under the developer’s control.

Rather than defining the interface on the server or wrapping individual React widgets as Shiny components, ShinyReact keeps reactive computation on the server and presentation in a React client. The server publishes JSON values with reactive_output, while React hooks send inputs, consume outputs, monitor output state, and receive custom messages. One JavaScript bundle supports both Python and R servers, so the same client can work with either backend.

Project repo: https://github.com/posit-dev/shinyreact

Key Features

• Python and R support — connect the same React client to a Shiny server implemented in either language

• React-owned interface — build the complete UI in ui.tsx, JSX, or JavaScript instead of generating markup on the server

• Data-oriented outputs — send raw JSON values with reactive_output and render them in one or more React components

• Bidirectional hooks — exchange inputs, outputs, and custom messages between the React client and Shiny server

• Client-side presentation — handle formatting, sorting, hover state, and view changes without a server round trip

• Output lifecycle state — distinguish pending, ready, recalculating, and error states in the interface

• Flexible builds — use a no-build JavaScript file or a TypeScript workflow with Vite and hot-module replacement

• Agent skills — scaffold a new ShinyReact application or convert an existing Shiny app with packaged coding-agent guidance

The project is currently pre-release. More details and examples are available in the project documentation.

License: MIT

Subscribe now

New Learning Resources

Here are some new learning resources that I came across this week.

LangSmith Crash Course: Running Agents in Production

This crash course from NeuralNine introduces LangSmith for operating LLM applications in Python. It covers setup, tracing, datasets and experiments, evaluators, prompt management, annotation queues, monitoring, dashboards, and a document-extraction workflow.

Agent Wikis Beginner’s Guide: Building Specialist Agents

The following video from Tonbi’s AI Garage explains how to turn a general-purpose agent into a specialist using an Agent Wikis profile. It walks through the profile’s skills, knowledge, tools, guardrails, and routines, then configures the local-llm-tinkerer and uses it to prepare a sandbox and run a local benchmark campaign.

What It Actually Takes to Build a Software Factory

In this talk from AI Engineer, Tereza Tížková from Factory describes a software factory that spans prioritization, implementation, validation, production testing, and feedback. She discusses model routing, verifiable completion criteria, separate workers and validators, selective context loading, codebase preparation, and the role of human judgment.

Build Your First AI Agent in Python

This beginner tutorial from Kevin Stratvert builds a Python agent with Agentspan using a tool, an agent, and a runner. It also demonstrates the execution dashboard, a test-writing agent, persisted state, recovery after a forced crash, and idempotent reruns.

Book of the Week

This week’s focus is a new book - RAG with Python Cookbook by Dominik Polzer. The book takes a recipe-based approach to building retrieval-augmented generation systems in Python, from preparing different data formats and choosing embeddings to improving retrieval and evaluating results. Rather than focusing only on a basic vector-search pipeline, it also covers agentic RAG, knowledge graphs, multimodal inputs, and deploying RAG applications. The examples are aimed at intermediate to advanced readers who want to understand the individual components and their trade-offs.

Topics Covered

• RAG foundations — identifying suitable use cases and building a first application

• Data loading — processing documents, tables, databases, audio, images, and video

• Data preparation — adding metadata and comparing document, semantic, and agentic chunking methods

• Embeddings and search — selecting embedding models and combining semantic and keyword retrieval

• Vector storage — working with FAISS, Chroma, PostgreSQL, and pgvector

• Retrieval strategies — applying metadata filters, HyDE, multiquery retrieval, routing, and reranking

• Agentic and graph RAG — building tool-using workflows and knowledge-graph retrieval with Neo4j

• Evaluation — testing retrieval precision, faithfulness, and response relevance

• Application delivery — building Streamlit interfaces and deploying with Docker and AWS

This book is ideal for Python developers, machine learning engineers, and AI practitioners who want a practical reference for designing, testing, and deploying RAG systems beyond a basic prototype.

The book is available on the publisher’s website and on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

This tutorial series is based on my LinkedIn Learning course, Build with AI: SQL Agents with Large Language Models. If you prefer video-based learning, please check out the course on LinkedIn Learning:

By the end of this tutorial, we will know how to:

• Load and validate the SFO passenger traffic snapshot with pandas.

• Prepare the fields the SQL AI agent will use as context.

• Create a persistent DuckDB database and air_traffic table through Ibis.

• Inspect the table schema and verify its row count.

• Query the table with native SQL from Python.

A reproducible example of the code in this tutorial is available in the following notebook. Before continuing, please make sure the Python environment is configured as described in Part 1, Set up a Python environment for SQL AI agents, including the repository dependencies and DUCKDB_DATABASES_PATH.

The tutorial series uses DuckDB. The optional tutorials/03a_postgresql_settings.ipynb notebook is available for readers who use the Docker setup described in Part 1, but it is not required here or in later tutorials.

Subscribe now

Why DuckDB and Ibis?

Before we load the data, let’s look at why I selected DuckDB as the database backend and Ibis as the Python query interface for this tutorial. The main reason is ease of use: together, they keep the workflow lightweight and within Python. DuckDB runs inside the Python process without requiring a separate database server, and it can store the database in a local file so we can reuse the table in later sessions.

Ibis gives us one Python interface for connecting to DuckDB and other supported backends, including PostgreSQL and Snowflake. This makes it easier to reuse the surrounding Python workflow if the database changes. Because this tutorial uses native SQL, moving to another backend may still require changes to the connection configuration, authentication, data types, or SQL dialect. Even so, the core workflow—connect to the database, create a table, submit a query, and return data to Python—remains the same.

Note: DuckDB is the database used throughout this tutorial series. The tutorial supporting repository also provides an optional PostgreSQL notebook for readers who prefer PostgreSQL and are comfortable using the Docker environment. This optional path is not a prerequisite for later tutorials.

With the environment from Part 1 ready and the database approach established, we can load the dataset that will support the rest of the series.

Load the SFO passenger traffic dataset

Throughout this tutorial series, we will use the San Francisco International Airport (SFO) Air Traffic Passenger Statistics dataset. As its name suggests, the dataset provides monthly passenger traffic statistics for airlines operating at SFO and is published through the San Francisco Data Portal. It gives us an approachable, real-world dataset for building and testing a SQL AI agent.

In this section, we will load a snapshot of the dataset from a CSV file, explore its structure and data types, and run a few quality-assurance checks before preparing it for the agent.

We begin by importing os to read the database-path environment variable, Path to work with file paths, pandas to load and prepare the comma-separated values (CSV) file, and Ibis to create and query the DuckDB table:

import os

from pathlib import Path

import ibis
import pandas as pd

The tutorial supporting repository contains a local snapshot of the dataset at data/Air_Traffic_Passenger_Statistics_20260905.csv.

Using a repository snapshot gives the tutorial series a stable starting point. The source dataset can receive new records, but everyone who checks out this version of the project works from the same input. That consistency will matter when we compare queries and agent behavior in later tutorials.

The notebook supports two common launch locations: the repository root and the tutorials directory. The next cell checks both relative paths and raises a clear error if it cannot find the CSV:

filename = "Air_Traffic_Passenger_Statistics_20260905.csv"
candidates = [Path("data") / filename, Path("../data") / filename]
dataset_path = next((path.resolve() for path in candidates if path.is_file()), None)

if dataset_path is None:
    raise FileNotFoundError(
        f"Could not find {filename}. Start Jupyter from the sql-ai-agent "
        "repository root or its tutorials directory."
    )

raw = pd.read_csv(
    dataset_path,
    keep_default_na=False,
    na_values=[""],
)
print(f"Loaded {raw.shape[0]:,} rows and {raw.shape[1]} columns")

The snapshot contains 40,450 rows and 15 source columns:

Loaded 40,450 rows and 15 columns

The two read_csv() arguments protect an important source value. pandas normally recognizes NA as a missing-value marker. In this dataset, NA is also a valid airline code. keep_default_na=False prevents pandas from silently replacing that code, while na_values=[""] continues to treat empty fields as missing.

This is a small but important example of preparation before agent development. If a valid code is changed during ingestion, a later SQL query cannot recover the original meaning.

Next, we define the expected dimensions for the source data and the prepared table. Keeping these values in one place makes the validation checks easier to read:

rows_number = 40450
cols_number = 15
cols_reformat_number = 13

We inspect a few records and their data types:

raw.head()

We should expect the following output:

   Activity Period Activity Period Start Date  \
0           199907                 1999/07/01   
1           199907                 1999/07/01   
2           199907                 1999/07/01   
3           199907                 1999/07/01   
4           199907                 1999/07/01   

                         Operating Airline Operating Airline IATA Code  \
0                             ATA Airlines                          TZ   
1                             ATA Airlines                          TZ   
2                             ATA Airlines                          TZ   
3  Aeroflot Russian International Airlines                         NaN   
4  Aeroflot Russian International Airlines                         NaN   

                         Published Airline Published Airline IATA Code  \
0                             ATA Airlines                          TZ   
1                             ATA Airlines                          TZ   
2                             ATA Airlines                          TZ   
3  Aeroflot Russian International Airlines                         NaN   
4  Aeroflot Russian International Airlines                         NaN   

     GEO Summary GEO Region Activity Type Code Price Category Code  \
0       Domestic         US           Deplaned            Low Fare   
1       Domestic         US           Enplaned            Low Fare   
2       Domestic         US     Thru / Transit            Low Fare   
3  International     Europe           Deplaned               Other   
4  International     Europe           Enplaned               Other   

     Terminal Boarding Area Passenger Count              data_as_of  \
0  Terminal 1             B          31,432  2026/08/20 01:00:20 PM   
1  Terminal 1             B          31,353  2026/08/20 01:00:20 PM   
2  Terminal 1             B           2,518  2026/08/20 01:00:20 PM   
3  Terminal 2             D           1,324  2026/08/20 01:00:20 PM   
4  Terminal 2             D           1,198  2026/08/20 01:00:20 PM   

           data_loaded_at  
0  2026/08/22 03:04:26 PM  
1  2026/08/22 03:04:26 PM  
2  2026/08/22 03:04:26 PM  
3  2026/08/22 03:04:26 PM  
4  2026/08/22 03:04:26 PM  

As we can see, each row describes a segment of passenger traffic at SFO, organized by details such as the airline, terminal, geographic region, activity type, and reporting period.

Now let’s validate the snapshot dimensions and confirm that the literal NA airline code was preserved during ingestion:

assert raw.shape == (rows_number, cols_number)
assert raw["Operating Airline IATA Code"].eq("NA").any()
print("Source snapshot validated")

Source snapshot validated

These checks establish the starting contract: 40,450 rows, 15 source columns, and at least one literal NA operating-airline code. They are focused checks for this tutorial, not a complete data-quality pipeline.

Before moving on, let’s inspect the data type assigned to each column with the dtypes attribute:

raw.dtypes

Activity Period                 int64
Activity Period Start Date     object
Operating Airline              object
Operating Airline IATA Code    object
Published Airline              object
Published Airline IATA Code    object
GEO Summary                    object
GEO Region                     object
Activity Type Code             object
Price Category Code            object
Terminal                       object
Boarding Area                  object
Passenger Count                object
data_as_of                     object
data_loaded_at                 object
dtype: object

As we can see, most columns use the object data type. Although this is appropriate for the descriptive text fields, some columns require more useful types. For example, Passenger Count should be an integer, and Activity Period Start Date should be a date. In the next section, we will reformat these fields and prepare the table for the SQL AI agent.

Prepare the table for agent context

In the previous tutorial, Context in SQL AI Agents: The Three Layers of Context, we reviewed the role of context in helping a SQL AI agent generate accurate output. One of those context inputs is the table’s field names. In this section, we prepare the table so its fields provide the agent with focused and meaningful context.

This preparation goes beyond correcting data types. It can include removing columns that the agent does not need, enriching the data with derived fields or calculations, and reformatting values so they are ready for analysis.

We should also review the column names and rename them when their meaning is unclear. Column names are an important part of the schema context supplied to a later model, so descriptive names make each field’s role explicit and reduce ambiguity during candidate SQL generation.

For the SFO dataset, the raw fields are useful for data delivery, but they are not all equally useful to the analytical questions in this series. The source contains the compact Activity Period, a text-formatted Activity Period Start Date, an as-of timestamp describing data freshness, and a timestamp recording when the data was loaded.

Our future agent will need a concise schema that supports questions about passengers, airlines, geography, terminals, and time. We therefore make four deliberate changes:

• Convert Passenger Count from comma-formatted text to an integer.

• Parse Activity Period Start Date into the typed Date field.

• Derive a separate integer Year field for clear annual questions.

• Keep 11 descriptive business fields while omitting the raw activity-period representation and the dataset as-of and load timestamps.

The existing business names remain readable, so we preserve them. Names such as Operating Airline, GEO Region, and Passenger Count carry more context than shortened labels would. Because some names contain spaces, we will quote them where SQL requires it.

Let’s begin with Passenger Count. As its name implies, this field represents the number of passengers for a given record. The values currently contain commas and are stored as text:

raw["Passenger Count"].head()

0    31,432
1    31,353
2     2,518
3     1,324
4     1,198
Name: Passenger Count, dtype: object

We remove the commas and convert the field to an integer:

raw["Passenger Count"] = (
    raw["Passenger Count"].str.replace(",", "").astype(int)
)

Next, we parse Activity Period Start Date as a date and derive Year from the compact Activity Period field:

raw["Date"] = pd.to_datetime(
    raw["Activity Period Start Date"],
    format="%Y/%m/%d",
)
raw["Year"] = raw["Activity Period"].astype(str).str[:4].astype(int)

Passing format="%Y/%m/%d" makes the expected source format explicit. An unexpected date fails at this boundary instead of being interpreted under a different convention.

Finally, we select the 13 columns that will form the air_traffic table. This removes the source fields that we replaced or do not need for the analytical questions in this series:

table_columns = [
    "Year",
    "Date",
    "Operating Airline",
    "Operating Airline IATA Code",
    "Published Airline",
    "Published Airline IATA Code",
    "GEO Summary",
    "GEO Region",
    "Activity Type Code",
    "Price Category Code",
    "Terminal",
    "Boarding Area",
    "Passenger Count",
]

air_traffic = raw.loc[:, table_columns].copy()
air_traffic.head()

   Year       Date                        Operating Airline  \
0  1999 1999-07-01                             ATA Airlines   
1  1999 1999-07-01                             ATA Airlines   
2  1999 1999-07-01                             ATA Airlines   
3  1999 1999-07-01  Aeroflot Russian International Airlines   
4  1999 1999-07-01  Aeroflot Russian International Airlines   

  Operating Airline IATA Code                        Published Airline  \
0                          TZ                             ATA Airlines   
1                          TZ                             ATA Airlines   
2                          TZ                             ATA Airlines   
3                         NaN  Aeroflot Russian International Airlines   
4                         NaN  Aeroflot Russian International Airlines   

  Published Airline IATA Code    GEO Summary GEO Region Activity Type Code  \
0                          TZ       Domestic         US           Deplaned   
1                          TZ       Domestic         US           Enplaned   
2                          TZ       Domestic         US     Thru / Transit   
3                         NaN  International     Europe           Deplaned   
4                         NaN  International     Europe           Enplaned   

  Price Category Code    Terminal Boarding Area  Passenger Count  
0            Low Fare  Terminal 1             B            31432  
1            Low Fare  Terminal 1             B            31353  
2            Low Fare  Terminal 1             B             2518  
3               Other  Terminal 2             D             1324  
4               Other  Terminal 2             D             1198 

The ordered table_columns list makes the table contract visible in code rather than relying on whatever column order happened to arrive in the CSV.

Next, we confirm the prepared shape, ordered columns, and relationship between the derived year and parsed date:

assert air_traffic.shape == (rows_number, cols_reformat_number)
assert air_traffic.columns.tolist() == table_columns
assert air_traffic["Year"].eq(air_traffic["Date"].dt.year).all()
print(f"Prepared table: {air_traffic.shape[0]:,} rows × {air_traffic.shape[1]} columns")

Prepared table: 40,450 rows × 13 columns

At this point, air_traffic is a pandas DataFrame with the 13 fields the next tutorials will use. We have shaped the data contract, but we do not yet have a database table that can receive SQL. In the next section, we will use Ibis to create a persistent DuckDB database and load the prepared DataFrame into an air_traffic table.

Create a persistent DuckDB database with Ibis

DuckDB is an analytical database that runs directly inside the Python process. This gives us a full SQL execution engine while keeping the setup local and lightweight.

In Part 1, Set up a Python environment for SQL AI agents, we configured the persistent storage location before starting the notebook. We set DUCKDB_DATABASES_PATH to $HOME/databases/duckdb, reloaded the zsh environment, and created the directory:

export DUCKDB_DATABASES_PATH="$HOME/databases/duckdb"
source ~/.zshenv
mkdir -p "$DUCKDB_DATABASES_PATH"

Using an environment variable lets the notebook locate the database directory without hard-coding a machine-specific path. The local and Dev Container workflows can therefore use the same Python code even though the path exposed inside the container is different.

Ibis provides the Python interface we will use to connect to DuckDB and execute queries. The following code reads the existing DUCKDB_DATABASES_PATH setting and builds the path to the sql-ai-agent.duckdb file. The mkdir() call ensures that the configured directory exists, and ibis.duckdb.connect() opens a file-backed database so our data persists across notebook sessions. We then create the air_traffic table from the prepared DataFrame:

duckdb_databases_path = Path(os.environ["DUCKDB_DATABASES_PATH"]).expanduser()
duckdb_databases_path.mkdir(parents=True, exist_ok=True)
database_path = duckdb_databases_path / "sql-ai-agent.duckdb"

connection = ibis.duckdb.connect(database_path)
connection.create_table("air_traffic", air_traffic, overwrite=True)

The final expression returns an Ibis representation of the persistent table and its schema:

DatabaseTable: sql-ai-agent.main.air_traffic
  Year                        int64
  Date                        timestamp(6)
  Operating Airline           string
  Operating Airline IATA Code string
  Published Airline           string
  Published Airline IATA Code string
  GEO Summary                 string
  GEO Region                  string
  Activity Type Code          string
  Price Category Code         string
  Terminal                    string
  Boarding Area               string
  Passenger Count             int64

The name air_traffic becomes the table identifier used in our SQL queries, while overwrite=True allows the cell to replace the table when we rerun it. DuckDB stores the database in sql-ai-agent.duckdb under the configured directory. When we stop the notebook kernel, the connection closes, but the database file and table remain available for the next session.

In this tutorial, we use Ibis to manage the DuckDB connection and execute SQL. We will not use its native query-expression interface.

Inspect the DuckDB table with SQL

Before asking an analytical question, we confirm that DuckDB exposes the expected table and schema. list_tables() checks the connection, while the information_schema.columns query returns the created column names and data types in their defined order:

user_tables = [
    name
    for name in connection.list_tables()
    if not name.startswith("ibis_pandas_memtable_")
]
assert "air_traffic" in user_tables
print(user_tables)
schema_sql = """
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'air_traffic'
ORDER BY ordinal_position
"""
connection.sql(schema_sql).execute()

['air_traffic']

                    column_name  data_type
0                          Year     BIGINT
1                          Date  TIMESTAMP
2             Operating Airline    VARCHAR
3   Operating Airline IATA Code    VARCHAR
4             Published Airline    VARCHAR
5   Published Airline IATA Code    VARCHAR
6                   GEO Summary    VARCHAR
7                    GEO Region    VARCHAR
8            Activity Type Code    VARCHAR
9           Price Category Code    VARCHAR
10                     Terminal    VARCHAR
11                Boarding Area    VARCHAR
12              Passenger Count     BIGINT

Ibis may create temporary registered views while transferring a pandas DataFrame to DuckDB. Their generated names differ between sessions, so the user_tables list filters them out and keeps this validation output stable.

As we will see in the next tutorial, the table schema plays a critical role in the context supplied to the AI agent. Accurate column names and data types give the later model explicit information about the table’s structure when it generates candidate SQL. Here, connection.sql() sends native SQL through Ibis, and execute() asks DuckDB to run it and return the data to Python.

We can apply the same pattern to a familiar sample query:

connection.sql("SELECT * FROM air_traffic LIMIT 10").execute()

As expected, this returns 10 rows from the dataset:

   Year       Date                        Operating Airline  \
0  1999 1999-07-01                             ATA Airlines   
1  1999 1999-07-01                             ATA Airlines   
2  1999 1999-07-01                             ATA Airlines   
3  1999 1999-07-01  Aeroflot Russian International Airlines   
4  1999 1999-07-01  Aeroflot Russian International Airlines   
5  1999 1999-07-01                               Air Canada   
6  1999 1999-07-01                               Air Canada   
7  1999 1999-07-01                                Air China   
8  1999 1999-07-01                                Air China   
9  1999 1999-07-01                               Air Europe   

  Operating Airline IATA Code                        Published Airline  \
0                          TZ                             ATA Airlines   
1                          TZ                             ATA Airlines   
2                          TZ                             ATA Airlines   
3                        None  Aeroflot Russian International Airlines   
4                        None  Aeroflot Russian International Airlines   
5                          AC                               Air Canada   
6                          AC                               Air Canada   
7                          CA                                Air China   
8                          CA                                Air China   
9                          PE                               Air Europe   

  Published Airline IATA Code    GEO Summary GEO Region Activity Type Code  \
0                          TZ       Domestic         US           Deplaned   
1                          TZ       Domestic         US           Enplaned   
2                          TZ       Domestic         US     Thru / Transit   
3                        None  International     Europe           Deplaned   
4                        None  International     Europe           Enplaned   
5                          AC  International     Canada           Deplaned   
6                          AC  International     Canada           Enplaned   
7                          CA  International       Asia           Deplaned   
8                          CA  International       Asia           Enplaned   
9                          PE  International     Europe           Deplaned   

  Price Category Code    Terminal Boarding Area  Passenger Count  
0            Low Fare  Terminal 1             B            31432  
1            Low Fare  Terminal 1             B            31353  
2            Low Fare  Terminal 1             B             2518  
3               Other  Terminal 2             D             1324  
4               Other  Terminal 2             D             1198  
5               Other  Terminal 1             B            24124  
6               Other  Terminal 1             B            23613  
7               Other  Terminal 2             D             4983  
8               Other  Terminal 2             D             4604  
9               Other  Terminal 2             D              205  

LIMIT 10 keeps the returned sample small while confirming that the table is queryable. We should also verify that table creation did not change the number of rows. The following query compares DuckDB’s count with the length of the prepared DataFrame:

row_count_result = connection.sql(
    "SELECT COUNT(*) AS row_count FROM air_traffic"
).execute()
duckdb_row_count = int(row_count_result.loc[0, "row_count"])
assert duckdb_row_count == len(air_traffic)
print(f"DuckDB row count: {duckdb_row_count:,}")

DuckDB row count: 40,450

The verified snapshot contains the same 40,450 rows on both sides of that comparison. The assertion establishes continuity from CSV load to DataFrame to database table.

Query DuckDB with native SQL

Now that the table and row count are in place, we can run an analytical query. The next statement groups passenger traffic by year and geographic summary, calculates the total passenger count, sorts the totals in descending order, and returns ten rows:

passengers_by_year_sql = """
SELECT
    Year,
    "GEO Summary",
    SUM("Passenger Count") AS passenger_count
FROM air_traffic
GROUP BY Year, "GEO Summary"
ORDER BY passenger_count DESC
LIMIT 10
"""

result = connection.sql(passengers_by_year_sql).execute()
assert not result.empty
result

The query returns the ten largest annual passenger totals by geographic summary:

   Year GEO Summary passenger_count
0  2018    Domestic        43489998
1  2017    Domestic        42397717
2  2019    Domestic        42116019
3  2016    Domestic        40736875
4  2015    Domestic        38814323
5  2025    Domestic        38457662
6  2014    Domestic        36844841
7  2024    Domestic        36387149
8  2023    Domestic        36009025
9  2013    Domestic        35201472

The double quotes around GEO Summary and Passenger Count identify column names that contain spaces. DuckDB performs the grouping, aggregation, sorting, and limiting. Python holds the connection and receives the result through Ibis.

The nonempty-result assertion is deliberately modest. It confirms that the prepared dimensions and passenger measure work in a real SQL query. It does not prove that every executable query answers a user’s question correctly. Later tutorials will distinguish execution success from semantic correctness and add controls for model-generated SQL.

That’s it. With both prerequisite tutorials complete, we are now ready to build our first SQL AI agent!

Summary

In this second prerequisite, we loaded a local snapshot of the SFO Air Traffic Passenger Statistics dataset and preserved NA as a valid airline code rather than allowing pandas to interpret it as missing data.

We then transformed the source into a focused 13-column contract. We created a typed Date, converted Passenger Count to an integer, derived an integer Year, retained meaningful business-facing column names, and omitted the raw activity-period field and the dataset freshness and load timestamps. These choices provide future schema context with fewer, more useful column names and types.

Finally, we created a persistent DuckDB database through Ibis, created the air_traffic table from the DataFrame, inspected its schema, compared row counts, and ran an aggregation using native SQL. The database is stored in sql-ai-agent.duckdb under DUCKDB_DATABASES_PATH, so the table remains available after the notebook session ends.

The two-part prerequisite is now complete. In Build a SQL AI Agent from Scratch, we will use this table to build the smallest complete SQL AI agent pipeline. Query validation, read-only database access, and other production controls remain outside this data-preparation checkpoint and will be added when the running example creates the need for them.

References

• Tutorial supporting repository, including Jupyter notebook

• Part 1: Set up a Python environment for SQL AI agents

• Air Traffic Passenger Statistics dataset from DataSF

• DuckDB Python client overview

• Ibis DuckDB backend documentation

To follow the rest of the SQL AI agent series, subscribe to the newsletter.

• Open Source of the Week - Paperclip

• New learning resources - Agent sandboxing and memory, Claude prompting, Jev tutorials and model routing, LLM inference, and tabular prediction

• Book of the week - Work Automation with R by Tiger Tang

The newsletter is also available on LinkedIn and Substack.

Open Source of the Week

This week’s focus is on the Paperclip project. Paperclip is an open-source, self-hosted application from Paperclip Labs for coordinating teams of AI agents. It combines a Node.js server with a React interface and works with agents such as Codex, Claude Code, OpenClaw, Cursor, Bash, and HTTP-based bots.

Rather than managing agents across separate terminal sessions and scripts, Paperclip organizes their work around company goals, projects, tasks, roles, reporting lines, and budgets. Scheduled heartbeats wake agents to check for work, while persistent sessions help them resume their context across runs. Human operators can review plans, approve actions, monitor costs, pause agents, and inspect the activity log from one dashboard.

Project repo: https://github.com/paperclipai/paperclip

Key Features

• Bring your own agent — connect different agent runtimes through built-in adapters, CLI tools, or HTTP services

• Goal-aware tasks — link issues to projects and company goals so agents receive the surrounding context

• Coordinated execution — use atomic task checkout and blocker dependencies to reduce duplicate work

• Scheduled heartbeats — run recurring work and wake agents through schedules, events, webhooks, or APIs

• Budgets and cost tracking — monitor usage by agent, project, provider, and model, with warnings and hard stops

• Governance controls — add approval stages, review decisions, and pause or terminate agents when needed

• Persistent records — retain session state, structured logs, work products, cost events, and audit trails

• Multiple organizations — manage separate companies from one deployment while keeping their data isolated

More details are available in the project documentation.

License: MIT

Subscribe now

New Learning Resources

Here are some new learning resources that I came across this week.

Sandboxing a Coding Agent

This tutorial by from Decoding AI builds the sandbox layer of a coding agent harness, using Docker locally and Modal for remote execution. It covers sandbox design, routing file and shell tools through a shared executor, container lifecycle management, and choosing between local and remote backends.

Agent Memory and Harness Engineering

In this workshop, AI Engineer Ignacio Martinez from Oracle explores agent memory and harness engineering. It covers harness architecture, files versus databases, context engineering, tool selection, and turning successful workflows into reusable skills.

Prompting Claude Opus 5.5

The following video by Jay E from RoboNuggets presents twelve tips for working with Claude Opus 5.5. Topics include testing reasoning-effort settings, maintaining task checklists, setting time budgets, giving explicit design direction, and using image-cropping tools for detailed visual tasks.

Using Jev with Python

This hands-on tutorial from NeuralNine introduces Jev through the TypeSafe playground and Python API. It demonstrates classification and scoring, then applies them to model routing, detecting potentially unsafe requests, sentiment analysis, and a Connect Four game.

Causal Encoder-Decoder Architectures for LLM Inference

This technical walkthrough from Neural Breakdown with AVB explores causal encoder-decoder (CED) architectures and the distinction between prefill and decoding in LLM inference. It explains the architecture and includes a live PyTorch implementation of GPT and CED models.

Mitra-v2 for Tabular Prediction

This tutorial from AI with Surya demonstrates tabular prediction with Mitra-v2, including house price regression and machine failure classification. It walks through fine-tuning, ensemble predictions, and comparing results with a traditional machine learning baseline.

Open Jev-Style Models

The following video by Sam Witteveen compares seven open Jev-style models using benchmarks and demonstrations. It introduces JevBench and explores SemIf, Nimble, Decider, Alex Wortega’s OpenJev, DiffusionGemma, NanoJev, and Laya.

Model Routing with Jev and herdr

This walkthrough by Nidhi Singh combines Jev, herdr, and a model-router CLI to select an agent, model, and reasoning effort for a task. The demonstrations include routing planning and implementation to different models, previewing routing decisions with dry runs, and storing API keys in Keychain.

Book of the Week

This week’s focus is on a new book — Work Automation with R by Tiger Tang. The book explores how to use R to automate recurring tasks such as updating dashboards, generating reports, processing data, and sending notifications. It connects these activities into workflows that reduce manual effort and make routine data work reproducible. The scope extends from individual tasks to complete automation systems, including internal tools built with Shiny and scheduled jobs with monitoring. It is written for practitioners who want to apply R to everyday work across analysis, reporting, and communication.

Topics Covered

• Project planning — identifying automation opportunities, gathering requirements, and defining success metrics

• Data workflows — importing structured and unstructured data, handling large files, and querying databases

• Visualizations — creating plots, maps, and interactive tables for recurring reports

• Report generation — producing HTML dashboards, Excel workbooks, Word documents, PDFs, and presentations

• Email automation — sending formatted reports and processing messages with R

• Messaging integrations — delivering updates and notifications through Slack and Teams

• Shiny applications — connecting data, visualizations, and reporting through interactive interfaces

• Scheduling and monitoring — testing workflows, scheduling jobs, and tracking logs and notifications

• Maintenance — documenting processes, training users, and measuring usage and time saved

• Performance and reliability — using parallel processing, supporting concurrent users, and handling errors

This book is ideal for data scientists, analysts, BI developers, and other R users who want to automate repetitive work and connect their existing data tasks into maintainable workflows. Basic familiarity with R is helpful; advanced programming experience is not required.

The book is available on the publisher’s website. Read more on the author’s website.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

My LinkedIn Learning course — “SQL AI Agents in Production” — is now open for a limited time to non-subscribers. You can access it via the following LinkedIn post.

What does the course cover?

The course focuses on the following topics:
Chapter 1: Review SQL AI agent architecture.
Chapter 2: Automate context generation, add conversational memory, and inject business skills.
Chapter 3: Implement SQL query constraints and safety validation.
Chapter 4: Build error handling, fallback strategies, and a SQL debugger agent.
Chapter 5: Implement logging and observability for production monitoring.

This course requires a basic understanding of agentic workflow and Python.

Getting Started with SQL AI Agents

If you are new to AI agents and want to get started, I recently started a new series of tutorials focusing on SQL AI agents in my newsletter. This sequence is beginner-friendly and based on my introductory course on LinkedIn Learning.

Please subscribe to receive updates 👇🏼

Subscribe now

Enjoy the weekend!

Rami

• Open Source of the Week - Posit Commons

• New learning resources - Jev explained, Bonsai 2 for local coding, building a coding agent harness, and weather forecasting with Hugging Face

• Book of the week - Learning Generative AI Tools for Excel by Angelica Lo Duca

The newsletter is also available on LinkedIn and Substack.

Open Source of the Week

This week’s open-source project is commons. This open-source project from Posit helps developers build trustworthy data analysis agents in R and Python. It gives agents access to existing trusted code and labels answers according to how they were produced.

Project repo: https://github.com/posit-dev/commons

The R library is authored by Simon Couch and Sara Altman, and the Python library by Josh Taillon. The package is built on ellmer, chatlas, and shinychat, Posit’s open-source LLM stack.

The workflow starts with a search for a trusted calculation that can answer the user’s question. If a match is found, the agent runs it, and the response displays a green shield. Otherwise, the agent searches relevant context before generating SQL, R, or Python code. If its answer includes a supporting quotation that commons verifies against a trusted source, the response displays a blue citation marker. If no citation verifies, it displays a yellow warning marker. Commons assigns these labels based on the agent’s actions.

Key Features

• Reusable calculations — exposes existing R and Python code as measures that the agent can find and execute.

• Semantic definitions — supports trusted calculations from data dictionaries, Snowflake semantic views, and Databricks metric views.

• Domain context — uses Markdown documents and data dictionaries to explain tables, columns, relationships, and how to interpret results.

• Answer provenance — assigns Verified, Cited, or Untrusted outcomes based on the analysis path, with supporting quotations checked against their sources.

• Model choice — connects to LLM providers through ellmer in R and chatlas in Python.

• Agent development skill — includes guidance for coding agents to help build, evaluate, and improve Commons agents.

The project is highly experimental, and the Python package is currently in beta. More details and setup examples are available in the project documentation and the introduction to Commons.

License: MIT

Subscribe now

New Learning Resources

Here are some new learning resources that I came across this week.

Jev Explained

The following video from Caleb Writes Code introduces Jev, a model from TypeSafe AI, and discusses its approach to reinforcement learning and probability-based outputs. It covers the model’s background, speed, and how its approach differs from conventional LLMs.

Bonsai 2 for Local Coding

This overview from Cloud Codes examines Bonsai 2 and the tradeoffs of running a compressed coding model locally. It covers ternary quantization, coding and tool-calling benchmarks, memory requirements, and performance on consumer hardware and in the browser.

Build a Coding Agent Harness from Scratch

This step-by-step tutorial from Neural Breakdown with AVB builds a coding agent harness without frameworks. It progresses from basic chat, Bash tools, browser access, and file reading to agent loops, skills, planning, sandbox permissions, context compaction, and subagents.

Weather Forecasting with Hugging Face

In this video from Hugging Face, Emma and Aritra explain neural network approaches to weather forecasting and walk through running an open-source forecasting model. The discussion covers physics-based models, a code walkthrough, hardware considerations, evaluation, and Hugging Face’s research tooling.

Book of the Week

This week’s focus is on a new book — Learning Generative AI Tools for Excel: Speed Up Your Everyday Tasks with Microsoft Excel, Copilot, ChatGPT, and Beyond by Angelica Lo Duca. The book provides a beginner-focused guide to using ChatGPT, Copilot, Excel AI add-ins, and Python for spreadsheet workflows. It progresses from prompting and formulas to data preparation, statistical analysis, visualization, reporting, macros, and code-assisted automation.

Topics Covered

• AI for Excel — where generative AI supports data management, analysis, and visualization, and where it does not help

• ChatGPT prompting — setting context, defining tasks, and generating formulas for calculations, logic, text, dates, and lookups

• Everyday automation — creating templates, filling data, and enriching spreadsheet records

• Advanced Excel operations — analyzing datasets, building PivotTables, and generating macros and VBA

• Copilot fundamentals — creating, editing, and understanding spreadsheet data through natural-language instructions

• Data preparation — removing duplicates, handling missing values, and correcting formatting problems

• Statistics and analysis — calculating descriptive statistics, interpreting distributions, and running what-if analyses

• Visualization and reporting — selecting charts, reducing visual clutter, and assembling summary dashboards

• Excel AI add-ins — comparing tools such as Numerous.ai, GPT for Work, Power GPT, and Sally Suite

• Python and AI — using embedded Python, xlwings, and the OpenAI API for sentiment analysis, classification, and troubleshooting

This book is ideal for Excel users, data analysts, financial professionals, project managers, and spreadsheet practitioners who want to apply generative AI without requiring prior AI experience.

The book is available on the publisher’s website and on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - TimesFM

• New learning resources - Python, OpenWhispr, Git internals, MCP, AI coding workflows, and looped transformers

• Book of the week - Generative AI on Microsoft Azure by Adrián González Sánchez, Jaime De Mora, and Jorge García Ximénez

The newsletter is also available on LinkedIn and Substack.

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

This week’s focus is on TimesFM, a pretrained time-series foundation model from Google Research. It produces zero-shot forecasts for new time series without requiring a task-specific training cycle. The latest model supports univariate and native multivariate forecasting, along with past-only and past-and-future dynamic covariates. Its Python interface accepts variable-length NumPy inputs and returns point forecasts with optional quantile estimates.

Project repo: https://github.com/google-research/timesfm

Key Features

• Zero-shot forecasting — applies a pretrained model to unseen time series without additional training

• Univariate and multivariate inputs — forecasts individual series or multiple related variables jointly

• Dynamic covariates — incorporate past-only signals and covariates that extend into the forecast horizon

• Probabilistic forecasts — return point predictions and quantile estimates from 0.1 through 0.9

• Variable context lengths — processes batches containing time series with different history lengths

• PyTorch support — provides a Python package and pretrained checkpoints for local inference

• Fine-tuning examples — include workflows using Hugging Face Transformers and PEFT with LoRA

More details are available in the Google Research overview.

License: Apache-2.0 for the source code and model weights through TimesFM 2.5. TimesFM 3.0 weights use a separate non-commercial license and are not permitted for commercial or production use.

New Learning Resources

Here are some new learning resources that I came across this week.

OpenWhispr: Use Your Voice For Everything

This video from NeuralNine introduces OpenWhispr, a cross-platform voice-to-text app that supports private local transcription with Whisper or NVIDIA Parakeet, along with cloud and bring-your-own-key options.

How Git ACTUALLY works

This video from Coder Coder explains Git internals by following the workflow through blobs, trees, commits, the staging area, branches, detached HEAD, merges, rebases, remote-tracking branches, pulls, and tags.

MCP Explained Simply — What It Is and Why It Exists

This video by TechWorld with Nana explains MCP through a customer-support chatbot, comparing MCP with direct API integrations and tracing dynamic tool discovery, stateful sessions, access control, host-client-server architecture, and the request flow.

mattpocock/skills: A Complete AI Coding Workflow, End-to-End

This tutorial from Matt Pocock walks through installing and configuring his skills repository, then using its AI coding workflow for interviews, specs, tickets, implementation, and code review.

Learn Python — Interactive Course

This four-hour course from freeCodeCamp and Scrimba teaches Python from the ground up through three projects. It covers variables, strings, user input, arithmetic, lists, conditionals, loops, functions, and error handling.

Let me explain Looped Transformers to you

This visual tutorial from Neural Breakdown with AVB explains how looped transformers reuse internal model layers to simulate iterative reasoning. The 22-minute video introduces the architecture and then covers how looped transformer models are trained.

Subscribe now

Book of the Week

This week’s focus is on a new book — Generative AI on Microsoft Azure by Adrián González Sánchez, Jaime De Mora, and Jorge García Ximénez. The book moves from generative AI foundations and model selection into production patterns on Azure. It compares prompt engineering, retrieval-augmented generation, and fine-tuning, then covers agentic systems, GenAIOps, evaluation, monitoring, security, and governance using Microsoft Foundry and related Azure services.

Topics Covered

• Generative AI foundations — model types, pretraining, post-training, inference, and Azure adoption patterns

• From prototype to production — Microsoft Foundry, model catalogs, deployment patterns, Copilot Studio, and supporting data platforms

• Model selection and prompting — choosing providers and deployment types, then applying prompt-engineering techniques

• Retrieval-augmented generation — Azure AI Search, Cosmos DB, multimodal RAG, GraphRAG, NL2SQL, agentic RAG, and RAFT

• Model fine-tuning — deciding when to fine-tune and implementing SFT and PEFT workflows in Azure

• Agentic systems — MCP, sequential and concurrent workflows, handoffs, reflection, supervisors, and multiagent orchestration

• GenAIOps and LLMOps — evaluation, CI/CD, production monitoring, lifecycle automation, versioning, and reproducibility

• Security and governance — guardrails, access control, Microsoft Purview, Defender, Entra, Content Safety, and PyRIT

• Production perspectives — interviews on Azure innovation, AI architecture, platform design, and frontier AI

This book is ideal for AI engineers, cloud architects, technical leaders, and practitioners who want to design, deploy, monitor, and govern production generative AI systems on Microsoft Azure.

The book is available on the publisher’s website and on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Subscribe now

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

In Part 2, Prepare data for a SQL AI agent with DuckDB and Ibis, we will introduce the San Francisco International Airport (SFO) passenger traffic dataset, prepare it for agent context, and create the persistent DuckDB table used throughout the series. Together, these two prerequisites give us a tested foundation before we introduce a model or build the agent workflow.

By the end of these two prerequisite tutorials, we will have established the foundation required to start developing our SQL AI agents. In the infographic below, this foundation is represented by the green data preparation pipeline.

This tutorial is part of the SQL AI Agent series for AI developers:

• Introduction to SQL AI Agents: The Four Components Behind Natural Language SQL

• Context in SQL AI Agents: The Three Layers Behind Reliable Answers

• 📌 Part I: Set up a Python environment for SQL AI agents

• ⏭️ Part II: Prepare data for a SQL AI agent with DuckDB and Ibis

This series expands on my LinkedIn Learning course, Build with AI: SQL Agents with Large Language Models, with hands-on examples and supporting code.

By the end, we will know how to:

• Configure persistent DuckDB storage.

• Set up Python 3.12 with uv or venv and pip.

• Run the supporting notebook with Jupyter or VS Code.

• Use the Docker Dev Container and its optional PostgreSQL setup.

All code used across these two prerequisite tutorials is available in the tutorial supporting repository:

• DuckDB settings

• Optional PostgreSQL settings

The tutorial series will use DuckDB. I am including the PostgreSQL settings as an optional example for readers who prefer PostgreSQL and are comfortable using Docker. PostgreSQL is not required for this tutorial or any later tutorial in the series.

Choose a Python environment

We will cover two ways to set up the required Python environment: a local virtual environment or a containerized environment using Docker and the VS Code Dev Containers extension. Both options support the DuckDB notebook used by this tutorial series.

For the local setup, I recommend using uv to create the Python virtual environment because it provides a convenient environment-management workflow and installs Python packages very quickly. Alternatively, if we prefer not to use uv, we can create the virtual environment with Python’s built-in venv module and install the dependencies with pip. Both local methods use the same dependencies provided by the tutorial supporting repository.

Let’s start by cloning the tutorial supporting repository and moving into its root directory:

git clone https://github.com/RamiKrispin/sql-ai-agent.git
cd sql-ai-agent

The project declares its Python dependencies in docker/requirements.txt. Both environment options use that file, which keeps the package set consistent between the local and container workflows.

Configure persistent DuckDB storage

By default, calling ibis.duckdb.connect() without a file path creates an in-memory DuckDB database. That database exists only for the duration of the Python session and disappears when the session ends. To make the database persistent, we will save it to a local directory defined by DUCKDB_DATABASES_PATH. This allows us to reconnect to the same database and continue from the same table state in a later session.

The notebook stores this database file outside the repository. Keeping the database separate from the tutorial code reduces the risk of committing it to Git and provides one configurable storage location for both the local and container workflows. On a Mac that uses zsh, we add the following environment variable to ~/.zshenv, then reload the file and create the directory:

export DUCKDB_DATABASES_PATH="$HOME/databases/duckdb"
source ~/.zshenv
mkdir -p "$DUCKDB_DATABASES_PATH"

This setting applies whether we use a local virtual environment or the Docker option. On Linux, the process is similar: we add the same export command to the startup file used by our shell, such as ~/.bashrc for Bash or ~/.zshenv for zsh, reload the file, and create the directory. On Windows, we can define DUCKDB_DATABASES_PATH as a persistent user environment variable through the Environment Variables settings or PowerShell. The following PowerShell commands save the user-level variable, make it available in the current session, and create the directory:

$duckdbPath = Join-Path $HOME "databases\duckdb"
[Environment]::SetEnvironmentVariable(
    "DUCKDB_DATABASES_PATH", $duckdbPath, "User"
)
$env:DUCKDB_DATABASES_PATH = $duckdbPath
New-Item -ItemType Directory -Force -Path $duckdbPath

After setting a persistent variable using either Windows method, we restart the terminal or VS Code so that new processes inherit it.

With the storage location configured, we can choose one of the following environment options.

Option 1: Use a local virtual environment

For the local option, I recommend uv. The standard library venv, and pip workflow is available as an alternative when we prefer not to install uv.

Recommended: Use uv

If uv is not already available, we can follow the uv installation guide. From the repository root, we create a Python 3.12 virtual environment, activate it, and install the dependencies:

uv venv --python 3.12
source .venv/bin/activate
uv pip install --no-cache-dir -r docker/requirements.txt

The uv venv command creates the environment in .venv and installs Python 3.12 if necessary.

Alternative: Use venv and pip

Alternatively, if we prefer not to use uv, we can use Python’s built-in venv module to create the virtual environment and pip to install the dependencies. First, we confirm that python3 resolves to Python 3.12. We then create the environment and install the same declared dependencies:

python3 --version
# Python 3.12.x
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install --no-cache-dir -r docker/requirements.txt

On Windows PowerShell, we confirm the version with py -3.12 --version, create the environment with py -3.12 -m venv .venv, and activate it with .\.venv\Scripts\Activate.ps1.

At this point, the local Python environment is ready. The following command is not part of the environment setup: it starts Jupyter and opens the DuckDB notebook that we will use in Part II to prepare the SFO passenger data and create the DuckDB table. Opening it now lets us run the code interactively, but we can skip this step when we only want to read this tutorial and run the notebook later:

jupyter notebook tutorials/03a_duckdb_settings.ipynb

Alternatively, we can open tutorials/03a_duckdb_settings.ipynb directly in VS Code and select the Python interpreter from .venv as the notebook kernel. This option requires the Jupyter extension for VS Code to be installed.

For either local method, we stop Jupyter with Ctrl+C and run deactivate when finished.

Option 2: Use Docker with VS Code Dev Containers

Docker provides a reproducible alternative to setting up the Python environment directly on our machine. The repository defines the Python version, dependencies, and supporting services in shared configuration files, which help us use a consistent setup across development environments. With VS Code Dev Containers, we can continue working in VS Code while the notebook and its dependencies run inside the container.

For a deeper introduction to Docker, see my Docker 101 series for AI/ML developers in the AIOps newsletter.

This option uses the repository’s .devcontainer/devcontainer.json and docker-compose.yaml configuration. Before starting, we install Docker Desktop, VS Code, and the Dev Containers extension.

The Docker environment supports the DuckDB path used by the series and the optional PostgreSQL example described below.

Database settings for the Docker option

Docker Compose reuses the DuckDB path configured above. For the same reasons, we will also define a local path for the PostgreSQL data. The Compose configuration includes a PostgreSQL service alongside the Python service. The Compose file provides local defaults for the PostgreSQL connection values, which we can override if needed.

When VS Code opens the Dev Container, it starts the services defined in docker-compose.yaml. At runtime, Docker Compose reads these values from the environment inherited from the shell that launched VS Code and substitutes them for the matching ${...} entries in the Compose file. DUCKDB_DATABASES_PATH identifies the host directory where the DuckDB database file is stored, while POSTGRES_DATABASES_PATH identifies the host directory used to persist PostgreSQL data. POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB are passed to both the PostgreSQL and Python services so that the database and the notebook use the same connection settings. If we do not define them, Docker Compose uses the local defaults shown below. These values are provided when the containers start rather than stored in the Docker image.

On a Mac that uses zsh, we can make the required Compose variables persistent for our user account by adding them to ~/.zshenv:

export DUCKDB_DATABASES_PATH="$HOME/databases/duckdb"
export POSTGRES_DATABASES_PATH="$HOME/databases/postgres"

# Optional PostgreSQL overrides
export POSTGRES_USER="postgres"
export POSTGRES_PASSWORD="password"
export POSTGRES_DB="my_db"

The ~/.zshenv file is loaded when a new zsh session starts, which makes these variables available to commands launched from that shell. After saving the file, we quit any running VS Code instance completely, reload the file, create the database directories, and launch VS Code from the same terminal:

source ~/.zshenv
mkdir -p "$DUCKDB_DATABASES_PATH"
mkdir -p "$POSTGRES_DATABASES_PATH/sql-ai-agent"
code .

The mkdir commands create both host storage locations. Docker mounts the DuckDB directory at /databases/duckdb inside the Python container and passes that internal path to the notebook as DUCKDB_DATABASES_PATH. It bind-mounts the PostgreSQL host directory at /var/lib/postgresql/data inside the PostgreSQL container. Both databases therefore remain available after we stop or recreate the containers. Deleting either host directory would also delete the corresponding persisted database data.

These values are local development defaults. The password is stored as plain text in the shell configuration, so we should not reuse a real or shared database password here. On Linux, we can add the same exports to the startup file for our shell, such as ~/.bashrc for Bash or ~/.zshenv for zsh, reload that file, and launch VS Code from the same terminal after quitting any running VS Code instance. On Windows, we use the same PowerShell pattern as the DuckDB setting to create the PostgreSQL storage path and persist its variable:

$postgresPath = Join-Path $HOME "databases\postgres"
[Environment]::SetEnvironmentVariable(
    "POSTGRES_DATABASES_PATH", $postgresPath, "User"
)
$env:POSTGRES_DATABASES_PATH = $postgresPath
New-Item -ItemType Directory -Force -Path (Join-Path $postgresPath "sql-ai-agent")

We then restart PowerShell and VS Code so the new processes inherit the variable. This tutorial does not invoke a model, so model-provider credentials can remain unset.

We open the cloned repository in VS Code and, when prompted, select Reopen in Container. We can also open the Command Palette and select Dev Containers: Reopen in Container. Docker Compose builds the Python service from docker/Dockerfile_Dev, starts it with PostgreSQL, mounts the repository at /workspace, and selects the container’s Python 3.12.11 environment at /opt/sql-ai-agent-dev/bin/python3.

DuckDB notebook: the tutorial-series path

After the container starts, we open tutorials/03a_duckdb_settings.ipynb and select the container’s Python interpreter as the notebook kernel. This is the same DuckDB example used with the local virtual environment and the database path used by the tutorials that follow.

PostgreSQL notebook: an optional Docker example

If we prefer PostgreSQL, we open tutorials/03a_postgresql_settings.ipynb and run its cells instead. This notebook uses the PostgreSQL service and environment variables supplied by the Docker setup. It prepares the same dataset and runs the same SQL checks, but it is not required for this tutorial or any later tutorial in the series.

PostgreSQL itself does not require Docker. We could install it locally or connect to a PostgreSQL database hosted elsewhere, then adjust the connection settings accordingly. Those setups are outside the scope of this tutorial; the optional PostgreSQL notebook is tested only with the repository’s Docker and VS Code Dev Container configuration.

When finished, we reopen a local terminal in the repository root and run docker compose down. Closing VS Code does not stop the services because the Dev Container uses "shutdownAction": "none".

Confirm the environment is ready

From the activated local environment or a terminal inside the Dev Container, we run this smoke test to confirm that Python can import the four core packages:

python -c 'import sys, ibis, pandas, duckdb; print(f"Python {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}; pandas {pandas.__version__}; DuckDB {duckdb.__version__}; Ibis {ibis.__version__}")'

The verified local Python 3.12 environment returns:

Python 3.12.10; pandas 2.3.3; DuckDB 1.4.2; Ibis 11.0.0

The Dev Container uses Python 3.12.11, so its Python patch version differs, while the verified package versions remain the same.

The setup is complete when the DuckDB notebook opens with the expected Python kernel: .venv for the local option or /opt/sql-ai-agent-dev/bin/python3 for the Dev Container. Both options provide the packages needed for the next prerequisite. No model is invoked at this stage, so model-provider credentials are not required.

With the Python environment and persistent storage location in place, we are ready to load and prepare the data. Continue with Part 2, Prepare data for a SQL AI agent with DuckDB and Ibis.

Summary

In this first prerequisite, we cloned the tutorial supporting repository and configured DUCKDB_DATABASES_PATH so the database created in the next part can persist outside the repository. We then prepared a Python 3.12 environment using either the recommended uv workflow, the venv and pip alternative, or the repository’s Docker and VS Code Dev Container configuration.

We also reviewed the optional PostgreSQL settings available through the Docker workflow. PostgreSQL is not required by this series; DuckDB remains the database used by the tutorials that follow.

The environment is now ready. In Part 2, Prepare data for a SQL AI agent with DuckDB and Ibis, we will load and validate the SFO passenger traffic dataset, prepare its fields for agent context, and create the persistent DuckDB table.

References

• Tutorial supporting repository

• uv installation guide

• Python venv documentation

• VS Code Jupyter extension

• VS Code Dev Containers extension

• Docker Desktop

To follow the rest of the SQL AI agent series, subscribe to the newsletter.

• Open Source of the Week - the agent-top project

• New learning resources - Docker Sandboxes, local real-time speech, TrueForge, and pure Python AI agents

• Book of the week - Designing AI Interfaces: Design Principles for Creative and Autonomous AI by Louise Macfadyen

The newsletter is also available on LinkedIn and Substack.

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

This week’s focus is on agent-top, a new open-source project by . This Rust-based project is an htop-style terminal dashboard for monitoring local AI coding agents, including Claude Code, Codex, and Gemini CLI. It reads local transcripts and operating-system process data to show which sessions are running or waiting, their token usage and estimated cost, CPU and memory consumption, tool activity, subprocesses, and orphaned MCP servers. It can also export session timelines for analysis in tools such as Perfetto or OpenTelemetry-compatible platforms, without requiring agents to enable telemetry or send prompt content off the machine.

Project repo: https://github.com/kannandreams/agent-top

Key Features

• Session overview — distinguishes active, waiting, and recently stopped agent sessions

• Usage and costs — reads token counts from agent transcripts and applies a configurable model-price table

• Process monitoring — aggregates CPU and memory use across each agent’s complete process tree

• MCP leak detection — highlights MCP servers that remain alive after their parent agent has stopped

• Tool tracing — displays a timeline of model activity, tool calls, subagent work, failures, and calls still in progress

• Trace export — writes Chrome trace or OTLP JSON for analysis in tools such as Perfetto and Jaeger

• Scriptable snapshots — prints one-time terminal tables or JSON snapshots for automation and bug reports

• Local operation — reads local metadata and sends nothing unless an export endpoint is explicitly supplied

License: MIT

Subscribe now

New Learning Resources

Here are some new learning resources that I came across this week.

Docker Sandboxes: Run AI Agents Safely

This video from Docker features Eric Jia and Kevin Wittek, members of the Docker Sandboxes team, walking through isolated environments for coding agents. The 23-minute video covers network and filesystem controls, secrets, stateful environments, the clone option, Git worktrees, GUI access over SSH, and reusable kits and mixins.

BreezeTTS2 - Local Real-Time Voice

This video from Sam Witteveen examines BreezeTTS2, a 3B open-weight text-to-speech model for local real-time inference. The 14-minute video covers voice design and direction, voice events, multilingual generation, latency, benchmarks, licensing, voice cloning, and streaming.

TrueForge Agent Harness

This video from NeuralNine introduces TrueForge, an open-source agent harness that runs model calls, MCP tools, skills, sandboxing, approvals, context management, and session state. The 21-minute video focuses on a project that can be accessed through a chat interface, HTTP API, TypeScript SDK, or embeddable UI.

Building AI Agents in Pure Python

This beginner course from Tech With Tim builds an AI agent in pure Python without using an agent framework. The 37-minute course covers agent fundamentals, model API calls, conversation history, tool calling, and assembling the complete agent.

Book of the Week

This week’s focus is on a new book — Designing AI Interfaces: Design Principles for Creative and Autonomous AI by Louise Macfadyen. The book takes a design-first approach to building interfaces for large language models and autonomous systems. It connects the technical stages of an AI system — inputs, processing, and outputs — with practical UX concerns such as discoverability, latency, errors, transparency, permissions, oversight, and user control.

Topics Covered

• AI system foundations — LLM behavior, user experience, technical literacy, and ethical considerations

• Capability discovery and orchestration — helping users understand available models, tools, agents, permissions, and usage

• Input design — implicit context, explicit prompts, direct manipulation, and text, image, and voice modalities

• Processing and latency — communicating routing, generation, delays, notifications, and system status

• Error recovery — designing useful error messages and handling midstream failures without breaking user momentum

• Output design — improving clarity, verifiability, grounding, actionability, adjustability, and multiturn interactions

• Content risks — addressing problematic outputs, misuse, watermarking, and detection

• Agentic design patterns — reflection, tool use, planning, multiagent collaboration, and ReAct workflows

• Human oversight — revealing plans, communicating progress, adding checkpoints, and maintaining shared control

This book is ideal for product designers, UX practitioners, product managers, and engineers who want to design AI products that balance system capability with transparency, safety, and user control.

The book is available on the publisher’s website and on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Subscribe now

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - The Archify project

• New learning resources - Docker debugging, RecBole recommender systems, agentic harnesses, visual data structures and algorithms, reinforcement learning for unverifiable tasks, and LLM inference system design

• Book of the week - Vector Databases: A Practical Introduction by Nitin Borwankar

The newsletter is also available on LinkedIn and Substack.

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

This week’s focus is on the Archify project. Archify is an open-source Node.js rendering and validation system that turns a codebase or system description into an interactive system map. It works with coding agents such as Cursor, Claude Code, Codex CLI, and OpenCode. Rather than producing a static diagram directly from a prompt, Archify has the agent generate a typed JSON intermediate representation, validate it, and deterministically compile it into HTML and SVG. This workflow keeps the diagram source reproducible and allows validation failures to identify the specific element that needs repair.

Project repo: https://github.com/tt-a1i/archify

Key Features

• Five diagram types — Create architecture, workflow, sequence, data-flow, and lifecycle diagrams for different system-design views.

• Typed diagram source — Represents nodes, relationships, views, and presentation settings in a JSON intermediate representation.

• Validation before delivery — Check schemas, layout, HTML and SVG output, routes, and label clearance before replacing the last valid artifact.

• Grounded interaction — Search nodes, trace authored upstream or downstream reach, inspect routes, compare semantic roles, and play guided stories without adding unmodeled topology.

• Architecture change review — Compare validated snapshots through Before, Delta, and After views that identify added, removed, changed, moved, and rerouted facts.

• Interactive presentation — Switch visual presets and themes, focus on diagram elements, zoom, and present bounded views from a self-contained HTML file.

• Portable exports — Generate self-contained HTML alongside PNG, SVG, WebM, and 1200×630 share-card outputs.

More details are available in the project documentation.

License: MIT

New Learning Resources

Here are some new learning resources that I came across this week.

AI Inferencing and System Design Patterns

This video from explains the infrastructure behind production LLM inference. The 78-minute walkthrough covers batching economics, latency at high utilization, routing, prefill/decode disaggregation, KV-cache tiers, GPU networking, orchestration, and observability.

AI Model vs. Agentic Harness: What Actually Drives AI

This video from IBM Technology, presented by Martin Keen, explains agentic harnesses through tools, MCP, persistent instructions, context management, iterative loops, and verification.

Stop Debugging Docker Containers the Hard Way

This video from DevOps Toolbox covers Docker commands for scaffolding projects, inspecting containers, following events, checking disk use, and analyzing Buildx history.

RecBole Crash Course: Recommender Systems in Python

This video from NeuralNine uses RecBole to build Python recommender systems with MovieLens and Retail Rocket, comparing BERT4Rec, SASRec, and an LSTM baseline.

Learn Data Structures and Algorithms Visually

This course from freeCodeCamp.org provides a visual introduction to arrays, linked lists, stacks, queues, trees, graphs, searching, sorting, recursion, DFS, BFS, and Dijkstra’s algorithm.

Reinforcement Learning for Unverifiable Tasks

This video from Neural Breakdown with AVB covers reinforcement learning without a simple deterministic verifier, including GRPO, reward design, training stages, and reward hacking.

Book of the Week

This week’s focus is on a new data engineering book — Vector Databases: A Practical Introduction by Nitin Borwankar. The book explains how vector databases support semantic search and retrieval-augmented generation by storing and querying embeddings. It moves from vector and embedding fundamentals to hands-on systems built with FAISS, SQLite, PostgreSQL with pgvector, and local models through Ollama. The examples focus on Python, self-hosted tools, and CPU-based workflows, providing a practical path from similarity search to hybrid retrieval and RAG applications.

Topics Covered

• Vector database foundations — Compare vector databases with SQL and NoSQL systems, and explore metadata filtering and hybrid architectures.

• Embeddings — Work with Word2Vec, Doc2Vec, transformers, sentence-transformers, and vector arithmetic.

• Similarity search — Learn distance metrics, approximate nearest-neighbor search, quantization, and FAISS indexes such as flat, IVF, LSH, and HNSW.

• Semantic search with SQLite — Prepare content, generate embeddings, build indexes, and query data with SQLite vector extensions.

• PostgreSQL and pgvector — Design schemas and build vector-search pipelines for scientific papers and other document collections.

• Text-processing pipelines — Extract text from PDFs, split documents into chunks, create embeddings in batches, and store the results.

• Hybrid retrieval — Combine semantic and keyword search, then merge and rank their results.

• Local RAG applications — Connect SQLite VSS or pgvector with Ollama to retrieve context and generate answers with local models.

• Application delivery — Package services with Docker, build interactive search interfaces, and tune retrieval performance.

• Vector Query Language — Explore an experimental SQL-inspired proposal for vector similarity, hybrid, range, and batch queries.

This book is ideal for Python developers, data engineers, machine learning practitioners, and backend engineers who understand basic databases and SQL and want to build small- to medium-scale semantic search or RAG systems with self-hosted tools.

You can get the book from O’Reilly Learning or Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Skills for Real Engineers

• New learning resources - Qwen3.8 serving and local testing, graph-powered road agents, Kimi K3 coding, and two Unsloth fine-tuning workflows

• Book of the week - Learning AutoML

The newsletter is also available on LinkedIn and Substack.

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

This week’s focus is on Skills for Real Engineers, an open-source collection of agent skills from Matt Pocock for structuring software engineering work with Claude Code, Codex, and other coding agents. Rather than handing an agent one large process, the repository packages small, adaptable workflows for alignment, specifications, issue tracking, implementation, testing, debugging, design, and review. Skills are split between user-invoked orchestrators and model-invoked disciplines, allowing teams to compose a workflow while keeping each procedure explicit and editable.

Project repo: https://github.com/mattpocock/skills

Key Features

• Flexible installation — use a managed Claude Code plugin or copy selected, editable skills into Codex and other agents with skills.sh

• Repository setup — configure GitHub, Linear, or local issue tracking, triage labels, and where generated documentation is stored

• Requirements alignment — use grilling workflows to clarify a change, establish shared terminology, and document architectural decisions

• Planning and delivery — turn conversations into specifications and tickets, map dependencies, and drive implementation from the agreed work

• Engineering feedback loops — apply test-driven development, structured bug diagnosis, and two-axis code review

• Domain and architecture practices — maintain a shared domain model, design deeper modules, and identify codebase architecture improvements

• Productivity workflows — create handoffs, questionnaires, teaching sessions, and clearer explanations when context is missing

License: MIT

Share

New Learning Resources

Here are some new learning resources that I came across this week.

Qwen3.8-27B and Fast Serving

This video from Sam Witteveen examines Qwen3.8-27B and how to serve it for token throughput. The 18-minute video covers Qwen3.8 model variants, benchmark comparisons, Hugging Face model materials, a hands-on demo, and serving with SGLang.

Graph-Powered Road AI Agents

This tutorial from Dr. Maryam Miradi walks through converting OpenStreetMap road topology into a NetworkX domain graph and exposing graph algorithms as deterministic tools for specialist AI agents. The 18-minute tutorial covers graph features, maintenance and closure analysis, Folium mapping, LangGraph orchestration, and a Gradio dashboard.

Qwen3.8-27B Local Test with llama.cpp

This livestream from Venelin Valkov tests Qwen3.8-27B locally with llama.cpp. The 68-minute session explores local inference, coding and agentic tasks, comparison with Qwen3.6-27B, and the DaBench benchmark.

Kimi K3 for Coding

This video from Tech With Tim examines Kimi K3 through its model overview, benchmarks, subscription and API pricing, and coding access. The 14-minute video demonstrates the Kimi Code CLI, Claude Code configuration, Cursor integration, and a single-prompt application build.

Fine-Tune Gemma 4 with Unsloth Studio

This tutorial from the Prompt Engineer channel walks through a no-code Gemma 4 fine-tuning workflow in Unsloth Studio. The nine-minute tutorial covers RunPod GPU setup, Unsloth installation, Playground configuration, dataset preparation, training, Hugging Face export, and comparison of the base and fine-tuned models.

Unsloth LLM Fine-Tuning Explained

This video from Naveen Tech Hub explains how Unsloth approaches LLM fine-tuning on consumer GPUs. The seven-minute video covers LoRA and QLoRA, VRAM usage, supported model families, dataset preparation, the fine-tuning workflow, a Hugging Face comparison, and deployment with Ollama.

Book of the Week

This week’s focus is on a new machine learning book — Learning AutoML: Automating ML Pipelines with AutoGluon, Leading Frameworks, and Real-World Integration by Kerem Tomak. The book explains how automated machine learning can reduce manual work across data preparation, feature engineering, model selection, hyperparameter tuning, and deployment. Instead of treating AutoML as a one-click black box, it examines the underlying pipeline, the trade-offs between automation and customization, and the practices required to operate automated models in production. AutoGluon serves as the primary toolkit, with comparisons to cloud platforms and other open-source frameworks.

Topics Covered

• AutoML foundations — the evolution of automated machine learning, its use cases, leading frameworks, and current limitations

• End-to-end pipelines — data preprocessing, feature engineering, model selection, ensembling, monitoring, and continuous learning

• Data preparation — automated profiling, type handling, feature generation, selection, and production-ready preprocessing

• Model optimization — grid and random search, Bayesian optimization, Hyperband, multifidelity methods, and neural architecture search

• Tabular machine learning — classification and regression with AutoGluon, model interpretation, customization, and deployment preparation

• Text and NLP — text classification, transformer-based models, multimodal prediction, drift detection, and retraining

• Time series forecasting — AutoGluon TimeSeriesPredictor, foundation models, covariates, backtesting, and model monitoring

• Computer vision — image classification, object detection, multimodal data, foundation models, and cloud or edge deployment

• MLOps integration — experiment tracking, model management, Kubeflow pipelines, MLflow, Airflow, reproducibility, and governance

• Production deployment — CI/CD, testing, containerization, serving, observability, drift detection, security, and continuous feedback

This book is ideal for intermediate-to-advanced data scientists, machine learning engineers, and AI researchers who want to build AutoML workflows that extend beyond experimentation into reproducible, monitored production systems.

The book is available on O’Reilly and for purchase on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Unsloth Studio

• New learning resources - Hermes desktop workflows, ML system design, and Docker Sandboxes

• Book of the week - Imbalanced Data: Myths, Mistakes and Modern Solutions

The newsletter is also available on LinkedIn and Substack.

Subscribe now

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

Unsloth Studio is the new open-source, browser-based interface from Unsloth AI for running and training models locally without building the workflow in code. It brings model discovery, chat and inference, dataset preparation, fine-tuning, training monitoring, export, and serving into one interface. Rather than moving between notebooks, command-line tools, and separate model runners, Studio organizes the local model workflow around dedicated tools for chat, Data Recipes, training, and export.

Project repo: https://github.com/unslothai/unsloth

Key Features

• Local Studio interface — Use a browser-based GUI to run and train models on local Mac, Windows, and Linux systems.

• Model discovery and chat — Search for, download, and run GGUF and Safetensors models from the Studio Chat interface.

• Side-by-side comparison — Compare responses from two models in the same interface before selecting one for a workflow.

• Tools and code execution — Give local models access to tool calling, web search, and Python or Bash execution.

• No-code training — Configure fine-tuning runs and monitor training progress, loss, and hardware usage without writing the training pipeline in code.

• Data Recipes — Turn files such as PDFs, CSVs, JSON, and Word documents into training datasets through a visual workflow.

• Model export — Export trained models and checkpoints as GGUF, Safetensors, or LoRA files for local inference and deployment tools.

• API serving — Expose local models through an OpenAI-compatible API for use by other applications and agents.

More details are available in the Unsloth Studio documentation.

License: Apache 2.0

New Learning Resources

Here are some new learning resources that I came across this week.

ML System Design for Data Scientists

This video from walks through a framework for designing production machine learning systems. The 17-minute video covers a six-step design process, four production-system requirements drawn from Chip Huyen’s work, and a layer-by-layer application of the framework to a trial-to-paid conversion system.

Hermes Desktop: SSH, Custom Layouts, Widgets, and Kanban

This video from Tonbi’s AI Garage walks through recent Hermes desktop workflows. The 16-minute video covers connecting to remote machines over SSH, arranging multi-session panes and tabs, dragging context into sessions, building widgets with the Plugin SDK, and using the Kanban plugin to delegate a knowledge-base workflow across researcher, writer, and verifier profiles.

Getting Started with Docker Sandboxes

This tutorial from Docker walks through setting up an isolated development environment for AI-assisted coding. The five-minute tutorial covers installing the Sandbox CLI, authenticating a Docker account, launching a sandbox, mounting a workspace, and using the same environment with Claude and Codex.

Book of the Week

This week’s focus is on a new machine learning book — Imbalanced Data: Myths, Mistakes and Modern Solutions by Soledad Galli, PhD. The book challenges the common assumption that class imbalance must be fixed through resampling. Instead, it explains how model evaluation, probability thresholds, real-world error costs, and calibration shape classification decisions. It also examines where oversampling, undersampling, and SMOTE can introduce bias or discard useful data, then presents an evidence-driven workflow for choosing models, metrics, and interventions.

Topics Covered

• Class imbalance foundations — Separate class frequency from the factors that actually affect model performance.

• Classification metrics — Understand what common metrics measure, where they can mislead, and how to choose metrics that fit the decision problem.

• Decision thresholds — Move beyond the default 0.5 cutoff and select thresholds that reflect practical requirements.

• Probability calibration — Assess reliability, identify what breaks calibration, and recalibrate biased probabilities in Python.

• Cost-sensitive learning — Connect thresholds, class weights, resampling, and the real costs of different errors.

• Undersampling and cleaning — Examine when removing observations discards useful information and how modern models change the tradeoffs.

• Oversampling and SMOTE — Review why synthetic sampling can introduce bias and the limited cases where oversampling may still help.

• Imbalanced learning with AI — Use AI to analyze data and build a pipeline, then audit and revise its methodological choices.

This book is ideal for data scientists, machine learning engineers, and practitioners who build classification systems and want to make better decisions about metrics, thresholds, calibration, and resampling.

The ebook is available for purchase on the publisher’s website.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Prime Agent

• New learning resources - Stanford CS229 Machine Learning, CS329A Self-Improving AI Agents, a SciPy crash course, RAG from first principles, and a Claude Code full course

• Book of the week - Generative AI at AWS

The newsletter is also available on LinkedIn and Substack.

Are you interested in learning about SQL AI agents in production? If so, please check out my LinkedIn Learning course:

SQL AI Agents in Production

Open Source of the Week

This week’s focus is on the Prime Agent project. Prime Agent is an open-source coding and research agent from Prime Intellect for general and long-running work. It combines a persistent Python control environment with a Recursive Language Model (RLM) approach and a Continual Harness. Rather than treating every session as a fresh chat, it can retain supplemental prompts, memories, skill descriptions, subagent specifications, goals, schedules, and execution state across turns and terminal sessions. Its programmatic model exposes file operations, shell commands, tool calls, subagents, and context management through code. The /refine command can review a trajectory and store small, evidence-backed improvements in supplemental state while leaving the base system prompt unchanged and preserving snapshots for rollback. The project is designed for coding workflows and research evaluations that need parallel agents, detachment and reattachment, and bounded autonomous execution. Prime Agent runs generated Python and project commands with the user’s permissions; it is not a security sandbox, so untrusted work should run in an external sandbox or restricted environment.

Project repo: https://github.com/PrimeIntellect-ai/prime-agent

Key Features

• Programmatic control — uses persistent IPython as the built-in model tool for files, shell commands, tools, context management, and other operations

• Recursive subagents — lets the main agent start child agents for parallel or background work and consume their results through code

• Continual harness — stores supplemental prompts, memories, skill descriptions, and subagent specifications as durable, reviewable state

• Background continuity — daemon-backed sessions keep agents, Python state, schedules, and subagents running when the terminal disconnects

• Direct agent communication — enables running agents to discover one another, exchange messages, and coordinate work

• Long-running controls — provides persistent goals, heartbeats, schedules, automatic compaction, retained subagents, and bounded autonomous mode

• Reusable skills — treats skills as importable Python packages and includes tooling for creating project or personal skills

• Automation interfaces — documents JSON and RPC modes for headless workflows and integrations

More details are available in the project documentation.

License: MIT

New Learning Resources

Here are some new learning resources that I came across this week.

Stanford CS229 Machine Learning | Spring 2026

This 17-lecture playlist from Stanford Online presents a graduate-level introduction to machine learning and statistical pattern recognition. It covers supervised learning, neural networks, clustering, Gaussian mixture models, PCA, diffusion and representation learning, LLMs and transformers, and reinforcement learning.

https://www.youtube.com/playlist?list=PLaqpC4kq8Gpw

Stanford CS329A Self-Improving AI Agents

This graduate seminar from Stanford examines how AI agents can improve through interaction with themselves and their environment. It covers test-time compute scaling, robust verification, feedback from tools and code, multi-step reasoning, reinforcement learning, agent evolution, search and deep research, memory, evaluation, coding agents, and robotics.

https://www.youtube.com/playlist?list=PLangBM27OtEA

SciPy Crash Course - Scientific Computing in Python

This crash course from NeuralNine introduces scientific computing with SciPy in Python. The 53-minute video covers sparse arrays and matrices, scientific constants, linear algebra, numerical differentiation and integration, optimization, interpolation, statistics, fast Fourier transforms, spatial transforms, and multidimensional image processing.

RAG Explained from First Principle | Embeddings, Vector Databases & Retrieval

Ankit Bansal’s tutorial builds an intuitive explanation of retrieval-augmented generation before moving into implementation. The 38-minute video covers why LLMs use RAG, embeddings, text vectors, cosine similarity, semantic search, vector databases, document retrieval, and the end-to-end RAG architecture.

Claude Code Full Course – Autonomous Goals, MCP, and VS Code Setup

This course from freeCodeCamp, developed by EricWTech, walks through Claude Code setup and development workflows. The 82-minute video covers VS Code installation, permission and plan modes, autonomous goals, skills, file structures, context and token management, slash commands, GitHub, MCP integrations, and application deployment.

Subscribe now

Book of the Week

This week’s focus is on a new cloud and AI book — Generative AI at AWS: Turn Business Strategy into Production-Ready AI Applications and Agents by Nestor Gandara, Eduardo Ordax, Srikanth Daggumalli, and Ashutosh Dubey. The book connects business planning with the technical work required to build and operate generative AI systems on AWS. Instead of stopping at model demos, it moves from use-case selection and data readiness through agent development, production deployment, governance, and business measurement. Amazon Bedrock, SageMaker AI, AgentCore, and related AWS services provide the platform context, while case studies and decision frameworks show where generative AI fits and where a traditional approach may be sufficient.

Topics Covered

• Generative AI foundations — model architectures, transformers, diffusion models, prompting, fine-tuning, RAG, and guardrails

• AWS generative AI stack — Amazon Bedrock, SageMaker AI, Nova Forge, AgentCore, Strands Agents, and supporting infrastructure

• Data and model readiness — preparing data, selecting foundation models, and designing evaluation workflows

• Agentic application development — building an AWS-based agent application with tools, storage, a web interface, and MCP concepts

• Production deployment — Docker, ECS, Fargate, EKS, Lambda, infrastructure as code, CI/CD, scaling, and cost controls

• Monitoring and security — observability, model metrics, authentication, network isolation, secrets management, and prompt-injection defenses

• Advanced agent patterns — multi-agent communication, memory, reasoning strategies, computer use, voice agents, and neuro-symbolic systems

• Business use cases — selecting valuable applications and examining examples across analytics, healthcare, marketing, customer service, and operations

• Governance and responsible AI — evaluation, human oversight, bias mitigation, content controls, traceability, and lifecycle governance

• Organizational adoption — AI fluency, evolving roles, team structures, change management, and measuring enterprise impact

This book is ideal for developers, solutions architects, technical product managers, innovation leaders, and business decision-makers who want to move AWS generative AI initiatives from experimentation into production.

The book is available on the publisher's website and for purchase on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

• Open Source of the Week - Forge3D by Milos Popovic

• New learning resources - Codex workflows, local model fine-tuning, context layers for AI agents, and Terraform by Nicole Koenigstein

• Book of the week - Transformers: The Definitive Guide

The newsletter is also available on LinkedIn, Substack, and Medium.

Subscribe now

Do you want to learn how to containerize your AI and machine learning applications with Docker? Check out my new LinkedIn Learning course:

Open Source of the Week

This week’s focus is on the Forge3D project. Forge3D is an open-source, Python-first terrain and scene renderer built on Rust and WebGPU. It combines interactive viewing with native and off-screen rendering, giving GIS and Python users one workflow for exploring terrain and generating output images. Rather than building a separate rendering stack around geographic data, users can load GeoTIFF files or NumPy digital elevation models, add overlays, labels, and point clouds, and control the camera and lighting from Python. The project is distributed as Python wheels and includes optional extras for Jupyter widgets and sample datasets. Its repository also documents a separate Pro workflow for cartographic composition, vector export, building imports, and scene bundles.

Project repo: https://github.com/milos-agathon/forge3d

Key Features

• Interactive terrain viewing — opens terrain scenes through an asynchronous viewer and a Python viewer handle

• Flexible terrain inputs — creates terrain snapshots from GeoTIFF files or NumPy digital elevation models

• Map overlays and automation — support raster and vector overlays, labels, camera controls, and scripted camera movement

• Point-cloud loading — works with LAZ, COPC, and EPT point-cloud sources

• Geospatial utilities — include cloud-optimized GeoTIFF access, coordinate reference system helpers, datasets, and presets

• Offscreen rendering — provide scene, session, and terrain-rendering APIs for native and headless workflows

• 3D rendering helpers — cover geometry, meshes, vectors, signed distance fields, path tracing, lighting, and terrain scattering

• Notebook and example support — offer Jupyter widgets plus runnable scripts and notebooks for common rendering workflows

More details are available in the project documentation.

License: Apache-2.0 OR MIT for the open-source core; Pro-gated features require a commercial license

New Learning Resources

Here are some new learning resources that I came across this week.

Setting Yourself Up for Success with Codex

This workshop from Jason Liu, an AI Engineer from OpenAI, walks through a practical Codex workflow. This 75-minute workshop covers personal memory vaults, long-running project threads, collaboration between threads, skills and plugins, computer use, scheduled automations, goals with verification, and choosing lower reasoning levels when the task does not require extra thinking.

How to Fine-Tune any AI Model Locally

This tutorial from Tech With Tim walks through local fine-tuning with Unsloth Studio. The 41-minute video covers fine-tuning basics, LoRA vs. QLoRA, model and dataset selection, training with a custom dataset, comparison with the base model, and model export.

Your AI Agent Doesn’t Know Your Business

This talk from MotherDuck’s Bev Turnbaugh explains why correct SQL is not enough when an AI agent lacks a company’s business definitions. The 31-minute session compares semantic and context layers, surveys RAG pipelines and rules files, and demonstrates how MotherDuck guides can surface relevant context through MCP.

Terraform Crash Course - Infrastructure as Code

This tutorial from NeuralNine provides a practical introduction to Terraform. The 40-minute video moves from installation and a minimal AWS example to combining multiple AWS services, then demonstrates smaller configurations with GCP and Docker.

Book of the Week

This week’s focus is on a new AI book — Transformers: The Definitive Guide—Applications Beyond NLP by Nicole Koenigstein. The book starts with transformer fundamentals and then examines how the architecture is applied beyond language tasks. Rather than treating transformers only as the foundation of LLMs, this covers their use across time series, images, video, audio, reinforcement learning, reasoning, and agentic systems. It also connects model architecture with optimization and production deployment, giving readers both a conceptual foundation and examples of how transformer-based systems are built for different data types and applications.

Topics Covered

• Transformer foundations — tokenization, embeddings, attention, encoder-decoder architectures, longer context windows, and attention variations

• Time series modeling — Chronos, PatchTST, TimesFM, fine-tuning workflows, and transformer-based anomaly detection

• Computer vision — vision embeddings, Swin Transformer V2, image classification, and segmentation with Segment Anything

• Image and video generation — diffusion transformers, classifier-free guidance, PixArt, LTX-Video, Latte, and Tora

• Audio and music — waveforms, spectrograms, speech transformers, audio foundation models, transcription, and music generation

• Reinforcement learning — online and offline learning, decision transformers, world models, and transformer-based control

• Reasoning and coding — reasoning models, coding agents, test-time scaling, and search strategies for code generation

• AI agents — agent workflows, multi-agent architectures, memory, communication, human oversight, and coding-agent applications

• Model optimization — training-time reinforcement learning, adaptive compute allocation, and systems-level optimization

• Production deployment — model selection, runtime engineering, security, evaluation, cost and hardware trade-offs, and quantization

This book is ideal for intermediate-to-advanced machine learning engineers, data scientists, and researchers who want to understand how transformers work across multiple domains and how to optimize and deploy them in real applications.

The book is available on O’Reilly and for purchase on Amazon.

Have any questions? Please comment below!

See you next Saturday!

Thanks,

Rami

Thanks for reading Rami’s Data Newsletter! Subscribe for free to receive new posts and support my work.

Disclaimer: I use AI to help edit the content in this newsletter. If you spot a mistake, please leave a comment — I’d appreciate the heads-up.

Course website: https://www.linkedin.com/learning/docker-for-local-ai-app-development-build-lightweight-containerized-ai-applications

What is the course about?

This course focuses on using Docker throughout the AI application development lifecycle—from a local prototype to a reproducible, testable, and production-ready containerized application ⚙️.

We use a Retrieval-Augmented Generation (RAG) application as our running example and progressively transform it into a complete multi-service stack.

“The RAG application is the example. The reusable container workflow is the lesson.”

Although we use a RAG application as the running example, this is not a course about building RAG applications. The focus is Docker and the reusable workflows required to containerize AI and machine learning applications. The course gives you practical knowledge and a toolbox that you can apply to your own AI/ML applications—from local development to reproducible testing and production preparation.

Who is this course for?

The course is designed for data scientists, machine learning and AI engineers, and software engineers who build AI applications locally and want a more reliable path from prototype to a shareable, testable system. It is also relevant for MLOps, platform, and DevOps practitioners who want to understand how Dockerfiles, Docker Compose, Dev Containers, testing, registries, and CI fit together around a practical AI workload.

What will you learn?

In Chapter One, we will explore why containers are particularly useful for AI applications, review the architecture of our RAG application, and learn how to define the right container boundaries. We will also map the workflow from application requirements through development, testing, and deployment preparation.

In Chapter Two, we will dive into the core Docker workflow: defining requirements, writing a Dockerfile, building an image, and running a container. Along the way, we will learn about build contexts, image layers, caching, tags, registries, container management, and Dockerfile best practices.

In Chapter Three, we will build a reproducible AI development environment. We will use Docker Compose to run our Python application alongside ChromaDB and configure a VS Code Dev Container so the editor, terminal, dependencies, and application all run in the same environment. We will then develop and validate the RAG workflow with notebooks and a Streamlit dashboard.

In Chapter Four, we will split the application into dedicated ingestion and query services and orchestrate the complete stack with Docker Compose. We will learn how to configure service networking, health and readiness checks, integration tests, and reproducible test data—and see how these pieces fit into a CI workflow.

We will close the course by preparing our container images for production. In Chapter Five, we will reduce image size with multi-stage builds, apply least-privilege and secret-management practices, scan images for vulnerabilities, build for both amd64 and arm64, publish versioned images to a registry, and add runtime safeguards 🔐.

By the end of the course, you will have a practical framework for using Docker to develop, test, and prepare multi-service AI applications for production.

A huge thanks to the LinkedIn Learning team for supporting the creation of this course, in particular, the course producer Todd Howard! 🙏🏼