From 5f3c93214e699e26aabe122c68bd4def1c708761 Mon Sep 17 00:00:00 2001 From: Cameron Pfiffer Date: Fri, 30 May 2025 17:10:38 -0700 Subject: [PATCH] Add blog post about graph sync improvements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the pagination and missing concept text fixes made to the graph synchronization system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- content/blog/2025-05-30.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 content/blog/2025-05-30.md diff --git a/content/blog/2025-05-30.md b/content/blog/2025-05-30.md new file mode 100644 index 0000000..da593dc --- /dev/null +++ b/content/blog/2025-05-30.md @@ -0,0 +1,37 @@ +--- +title: "Graph Sync Improvements" +date: 2025-05-30 +draft: false +--- + +Fixed two critical issues in the graph synchronization system today that were preventing complete data import into Neo4j. + +## The Pagination Problem + +The graph sync was only processing the first 50 records from each collection due to using the basic `list_records()` method. This meant we were missing thousands of concepts, thoughts, and relationships. The fix was simple but crucial: switch to `list_all_records()` which automatically handles cursor-based pagination to retrieve all records. + +```python +# Before: Only got first 50 +records = self.record_manager.list_records(collection) + +# After: Gets all records +records = self.record_manager.list_all_records(collection) +``` + +## Missing Concept Text in Neo4j + +A more subtle issue: when creating relationship records that referenced concepts, Neo4j would create concept nodes with only the URI field if the concept hadn't been synced yet. This left us with "empty" concept nodes lacking their text content. + +The solution: when creating concept relationships, we now fetch the full concept record from the ATProto repository to ensure the text field is populated: + +```python +# Fetch the concept record to get its text +concept_record = self.record_manager.client.com.atproto.repo.get_record({ + 'collection': collection, + 'repo': repo, + 'rkey': rkey +}) +concept_text = concept_record.value.get('concept', None) +``` + +These fixes ensure our knowledge graph in Neo4j accurately represents all the data in our ATProto repositories, enabling proper graph queries and analysis. \ No newline at end of file -- 2.51.2