Teaching Graphify to Understand Databricks Notebooks

---

Orginally published on Medium on 8 July 2026

A side quest from my “Build Your Own LinkedIn Analytics” project

Status note: At the time of writing, this post describes three contributions: Issue #1497 and PR #1498 to Graphify core (both open, under review), and graphifyy-skills — a companion skills repo that works with stock Graphify today.

I. From LinkedIn analytics to Graphify

In late 2025 and early 2026 I published a 12‑part series on building a personal LinkedIn analytics platform using Databricks Free Edition.

Months later, I attended a talk at Google I/O Extended Singapore, where Melody Leong presented Graphify as an open‑source engine that turns code, schemas, docs, and even images into a queryable knowledge graph for AI assistants, in a way that saved greatly on tokens.

This got me thinking: what if I could implement this as an AI‑native way to navigate the whole LinkedIn project, not just by opening notebooks and YAML files manually, but by asking questions like:

• “Which jobs produce the tables that feed the dashboard?”
• “Where is fct_daily_post_statistics defined and how is it used?”
• “What’s the path from raw LinkedIn exports to a specific metric on the dashboard?”

So I pointed Graphify at the LinkedIn analytics repo and waited to see what it would learn.

That’s where the side quest began.

---

II. What happened when Graphify met Databricks

Running Graphify on the repo was immediately useful: it understood my Python modules, SQL files, and YAML configs, and built a graph that tied together jobs, pipelines, and resource definitions.

But two issues stood out very quickly.

a. Notebooks were invisible

Most of the critical pipeline logic in the LinkedIn project lives in Databricks notebooks: ingestion, cleaning, transformations from bronze to silver to gold, and the joins that power the final metrics.

Graphify, however, was blind to them.

Under the hood, Graphify uses a detect.py module to walk a directory and classify files by type: CODE, DOCUMENT, PAPER, IMAGE, and VIDEO, based on file extensions and some content heuristics.

When I looked at the extension sets in that file, here’s what I found:

• .py, .sql, .sh, .json, .tf, .hcl and many more were in CODE_EXTENSIONS.
• .md, .txt, .html, .yml, .yaml were in DOC_EXTENSIONS.
• .docx and .xlsx had a dedicated OFFICE_EXTENSIONS set and a special side‑car conversion path.
• .ipynb was nowhere to be seen.

That meant any .ipynb file would be classified as None and silently skipped by the detector. In practice, Graphify was constructing a rich graph of everything except the Databricks notebooks; unfortunately, these were exactly the files that described the pipeline.

b. Dashboards were treated as generic JSON

The open‑source LinkedIn repo includes a Databricks Lakeview dashboard exported as a .lvdash.json file. This JSON contains:

• Dataset definitions, with SQL queries referencing the gold tables.
• Pages and widgets (charts, tables, filters) that make up the dashboard.
• Ties back to the same tables used by notebooks and jobs.

Graphify was treating this file as generic JSON code, because .json sits in CODE_EXTENSIONS.

That’s reasonable in general (a lot of config files are JSON) but it meant the dashboard structure wasn’t being interpreted as a semantic object in the graph (a dashboard with pages, widgets, and datasets). For my use case, that was a missed opportunity.

---

III. Reading Graphify’s design before proposing anything

Rather than jumping straight into code, I spent time reading Graphify’s own architecture:

• The README and docs, to understand the core goal: “turn any folder into a queryable knowledge graph for your AI assistant.”
• The detect.py implementation, to see how file classification and side‑car conversion work, especially for Office files.
• The tests under tests/test_detect.py and related modules, to see how aggressively the project guards against things like secret leakage and zip bombs.

A few design principles stood out:

• File classification is conservative. Unknown or risky file types are skipped rather than guessed. Sensitive directories and filenames are excluded via explicit patterns and generic keyword heuristics.
• Heavy formats get converted to markdown sidecars. PDFs, .docx, .xlsx, and Google Docs files are converted to markdown before being handed off to the LLM.
• Incremental updates are a first‑class concern. Graphify stores md5 hashes and mtimes in a manifest to know which files have changed between runs.

Any change I proposed would need to respect those constraints rather than adding a one‑off hack for my specific project.

---

IV. Designing notebook support the “Graphify way”

I knew I wanted .ipynb files to be part of the graph, but not by pushing the raw JSON into the AST path. That would be noisy and brittle:

• The AST extractor would see the notebook as a JSON document, not as a sequence of code and markdown cells.
• The word count would be inflated with metadata keys like "cell_type", "outputs", "metadata".
• Any future notebook schema changes would break assumptions.

The solution was already in the codebase: copy what Graphify does for Office files.

a. Sidecar conversion: ipynb → markdown

Graphify uses functions like docx_to_markdown and xlsx_to_markdown to convert office files into markdown sidecars in graphify-out/converted/, then indexes those sidecars as DOCUMENTs.

I proposed an analogous ipynb_to_markdown converter:

• Read the notebook JSON.
• Iterate through cells in order.
• For markdown cells: emit the markdown as‑is.
• For code cells: wrap source in fenced code blocks (e.g. “`python).
• Ignore outputs and most metadata.

The result is a linear markdown representation of the notebook that preserves:

• The narrative structure (markdown cells).
• The code that defines the pipeline (code cells).

And crucially, it matches Graphify’s existing pattern: heavy or complex formats converted into a simpler, LLM‑friendly textual representation.

This is the conservative, universally-safe approach for the upstream PR — a single document representation any Graphify installation can handle. In graphifyy-skills (Section VII), I took this further by routing each cell type to a language-matched sidecar, so Python cells feed tree-sitter AST and SQL cells feed the SQL extractor directly.

b. Separate extension set, not “just add .ipynb to code”

In the pull request, I deliberately did not add .ipynb to CODE_EXTENSIONS. Instead, I introduced a dedicated NOTEBOOK_EXTENSIONS set and a routing branch in detect.py similar to OFFICE_EXTENSIONS.

The logic is:

• If a file has a notebook extension (.ipynb), convert it to markdown sidecar.
• Treat the sidecar as a DOCUMENT for extraction and word counting.
• Leave the original file untouched.

This keeps concerns clean:

• CODE_EXTENSIONS remains the set of files that are directly parsed as code (e.g. .py, .sql, .ts).
• OFFICE_EXTENSIONS and NOTEBOOK_EXTENSIONS are the “convert first” formats.
• The incremental detection code doesn’t need to change; it works on sidecar paths just as it does today.

---

V. Filing the issue: explaining the gap (Issue #1497)

With the design sketched out, I opened an issue on the Graphify repo: “feat: add Jupyter notebook (.ipynb) support via markdown sidecar extraction”. 

feat: add Jupyter notebook (.ipynb) support via markdown sidecar extraction · Issue #1497 ·…
Summary .ipynb files are currently silently dropped by classify_file() – the extension is absent from all sets in…github.com

In the issue, I tried to do three things clearly:

a. Describe the current behaviour

• .ipynb is unrecognized by classify_file() and therefore skipped.
• For a notebook‑heavy project like a Databricks pipeline, this leaves a large blind spot.

b. Describe the expected behaviour

• .ipynb should be read and converted into a markdown sidecar.
• The sidecar should be treated as a DOCUMENT so the LLM extraction path can “see” both code and markdown.

c. Outline a concrete, minimal implementation

• A small converter function.
• A new extension set and branch in detect.py.
• Tests in test_detect.py to validate classification and conversion.

The goal was to make it as easy as possible for maintainers and other contributors to assess whether this fit Graphify’s design and to propose improvements if needed.

---

VI. Proposing the implementation (PR #1498)

Issue #1497 framed the problem and proposed direction. PR #1498 implemented it. 

Feat/ipynb notebook support by KunojiLym · Pull Request #1498 · Graphify-Labs/graphify
AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and more). Turn any folder of code, SQL…github.com

At a high level, the PR does the following:

• Adds NOTEBOOK_EXTENSIONS = {".ipynb"} alongside the existing extension sets.
• Introduces ipynb_to_markdown(path: Path) -> str, mirroring the style and safety considerations of docx_to_markdown and xlsx_to_markdown.
• Wires notebook conversion into detect() so sidecars are created in graphify-out/converted/ and queued for extraction as DOCUMENTs.
• Adds tests to ensure that .ipynb files are no longer invisible; empty or malformed notebooks fail gracefully; and word counts come from the sidecar, not the JSON.

At the time of writing, this PR is still under review. That’s normal for a popular project: maintainers are juggling many issues, and they have a responsibility to guard performance, security, and maintainability as usage grows.

Regardless of the final shape of the change, the process itself was instructive.

VII. The skills layer: a working solution today

While PR #1498 waits in the review queue, I published a parallel solution that works with stock Graphify right now: graphifyy-skills.

The key design decision was to go further than a single markdown sidecar. Graphify already has purpose-built extractors for different file types: tree-sitter AST for Python, SQL AST for .sql, semantic extraction for markdown. Why collapse a notebook into one format when you can route each cell to the extractor it deserves?

ipynb-graphify does exactly that. It splits a Databricks notebook into per-language sidecars:

• %sql cells → .ipynb.sql : so table references are extracted by the SQL AST parser
• Code cells → .ipynb.py : so functions, classes, and calls go through tree-sitter
• %md and markdown cells → .ipynb.md : so narrative context feeds semantic extraction
• %sh, %scala, %r cells : each routed to their own sidecar

databricks-graphify adds a second converter, lvdash_split.py, which parses the Lakeview dashboard JSON, extracts dataset SQL queries with FROM/JOIN table references, and writes .lvdash.sql and .lvdash.md sidecars. The dashboard’s data dependencies become first-class graph nodes, connected to the same gold tables that the notebooks write to.

No fork of Graphify is needed for this implementation; the sidecars use file extensions Graphify already understands.

The relationship between the skills repo and PR #1498 is complementary, not competing. The PR fixes automatic detection: notebooks are ingested without any user action, for everyone. The skills fix comprehension quality: you get richer, per-language graph nodes instead of a single flattened document. Both matter; they solve different problems at different layers.

---

VIII. Why this side quest matters beyond one repo

For my LinkedIn analytics project, the immediate benefit is obvious: if Graphify learns to understand .ipynb, the graph for that repo becomes much more complete.

But the reasons I think this is worth writing about go beyond one project and one PR.

a. AI tooling is only as good as its domain fit

Generic AI tools are powerful, but they often under‑serve domain‑specific practices:

• Databricks and data teams use notebooks heavily.
• Analytics projects mix code, SQL, notebooks, YAML, dashboards, and Office exports.

Out‑of‑the‑box tools can miss large parts of that picture. It’s on us, as users and practitioners, to notice these gaps and propose how to close them.

b. Contributing beats complaining

I could have stopped at “Graphify doesn’t understand notebooks, that’s a shame.” Instead, I chose to:

• Read the code and tests.
• Designed solutions at two layers: a core detection fix (PR #1498) and a skill-layer extension (graphifyy-skills).
• Published a working implementation immediately, without waiting for upstream review.
• Opened an issue and PR to propose the core fix for the benefit of all users.

Whether or not the maintainers accept the exact implementation, the conversation helps move the tool forward, and it sets an example for others who hit similar edge cases.

c. Data engineers can (and should) shape AI tooling

There’s a lot of talk about AI assistants helping engineers. The reverse is also true: engineers can help AI tooling better reflect how real projects are built.

For data and analytics practitioners, that means:

• Asking: “Does this tool understand notebooks? Dashboards? Orchestrators? Lakehouse schemas?”
• When the answer is “not yet,” deciding whether to live with the gap or to co‑design the fix.

This side quest was my small contribution to that second path.

---

IX. If you want to try something similar

If you’re working on notebook‑heavy analytics projects and want to improve how your AI assistant understands them, here’s a lightweight way to start:

1. Pick a real project.
   Choose a repo that matters to you, whether it is a Databricks pipeline, a Jupyter‑based experimentation repo, or an analytics codebase.
2. Run Graphify (or a similar tool) on it.
   See which files show up in the graph and which don’t. Pay particular attention to notebooks, dashboards, and config artifacts.
3. Inspect the tool’s detection logic.
   Look for extension lists and classification heuristics. Ask: “What is this tool assuming about my project’s structure?”
4. If you’re on Databricks or Jupyter: install graphifyy-skills and run /databricks-graphify or /ipynb-graphify on your repo.
   You’ll get per-language sidecars that Graphify can extract AST nodes from immediately — no fork, no PR needed.
5. When you find a gap, write it down clearly.
   Capture current behaviour, expected behaviour, and a minimal proposed approach. This alone is valuable feedback to maintainers.
6. If you have time, turn it into a PR.
   Respect the project’s patterns, add tests, and be open to feedback.

And if you’re curious about the original LinkedIn analytics project that motivated this side quest, you can start with Part 1 of the series and work your way through to the open‑source repo and dashboard.