Paths, Entities, and Types in Spring Data for Neo4j (also, I'm back)

I know it's been awhile since my last post, but, I assure you--one and all--that I have not left you kind folks for good.  I can also assure you that the cause of my literary absence has everything to do with being loaded down with work and is not a result of my gallivanting around the world whilst trying strange and wonderful new beers.

(Boy, do I ever wish that was the case.)

Strange AND wonderful!
[Courtesy: FOX]

No, I've managed to find some time in which to write a bit about something I've been looking into concerning SDN: Paths with multiple entity types.

So, without further ado, let's begin.

Yes, quite.
[Courtesy: samuelrunge.com and Monty Python)

Disclaimer

While the topics I'm about to go into could very well have solutions to them that I have yet to uncover, I'm presenting my findings and questions in the hopes that not only will someone find the discussion interesting/useful, but, that it might also help lead me to better solutions.

So, please do read on and try to keep the caveat above in mind.

Retrieving Heterogeneous Paths

If you'll recall from several posts ago, I had been attempting to write a web application based around the concept of a simple recommendation engine and a software retailer (a la Babbage's from the last century).  The implementation was being done using a Spring-based Java stack with Neo4j as the data store.

I had gotten to a point where I was able to load data into Neo4j via a method not unlike the one from Cineasts' (which can be found as part of the Spring Data Neo4j Reference).  A quick recap of the domain model follows below.

The domain model in question.

One important task of any recommendation engine is the ability to suggest entities that are relevant to a given starting point via some sequence of relationships.  Implementing such an engine, while quite doable, is something that I will eventually get to.

Another, more simple, task is to be able to see how two entities are related, i.e. "how do I get from point A to point B?".  As an example, one such question might be, "How is game A related to game B, even though the difference in publication dates is large?"  And yes, this type of question is extremely similar to the "Six Degrees of Separation" question.

I bet he's a gamer.
[Couresty: Wikipedia]

Some Basic Code

You might also recall that I had implemented a GameRepository class with the following signature:

 
public interface GameRepository extends GraphRepository<Game>, RelationshipOperationsRepository<Game>

With the above repository extending the GraphRepository and RelationshipOperationsRepository interfaces, we are provided with a host of cool (and handy) methods out of the box (tip: make sure you're comfortable with the "convention over configuration" paradigm, as there is a bit of magic that goes on with those OOTB methods).

As you'd expect, we can also add additional methods to the interface.  One example of a method you might want to add could be a custom Cypher query that returns all Game nodes with a particular property (the implementation of this is outside the scope of this post, but, it's actually pretty simple; if people want to see it, just shoot me a note!).

However, today we're looking to address the "Six Degrees of Separation" question (minus the limit on the degrees of separation), i.e. "how are node A and node B related"?

So let's give this a (very simple) shot:

 
@Query("START n=node(1), x=node(97) MATCH p = shortestPath( n-[*]-x ) RETURN p")
Iterable<EntityPath<Game, Game> > getPath();

Given that the nodes with IDs 1 and 97 are "Game" nodes, the Cypher query above is essentially determining how the two nodes are related.

(For the sake of this post, I'm ignoring the fact that there could be multiple "shortest paths" between the two nodes as it has little bearing on the goal of this post.)

Quickly going over the return type, SDN allows us to return back an EntityPath given a starting node type and an ending node type which, in this case, is the Game type.  An EntityPath is capable of storing nodes and their relationships as part of a Cypher path.  The Iterable portion of the return type is necessary unless you want to use EndResult instead of Iterable.

We can then access the individual path(s) via the Iterable return type.

(NOTE: There is currently a bug with SDN that throws an exception when calling a single path's .nodes() or .nodeEntities().  This bug has been around since SDN 2.1.RC4.)

Traversing the Returned Path

There's a reason my explanation of the code used stops where it does.  Those of you with a keen eye and/or are familiar with OOP/OOD will identify a potentially big stumbling block: How do you iterate through a path of nodes and/or relationships with potentially wholly disparate, unrelated types?  Given that this is SDN and is geared towards integrating easily with Java and POJOs, the issue becomes apparent.

How do we solve this?

Solution 1

Make sure the nodes you are after all either implement a common interface or are derived from a common class.

Seem too good to be true?  You're right, because it is.  While there may be some scenarios in which this solution might work, it is often the case in a graph database the nodes/relationships are entirely unrelated concepts/types, e.g. Game and Customer, or, Game and Developer.  This separation would imply that there are likely methods and attributes that are specific to a given type that would have no business being in a shared superclass or interface.

Solution 2

Employ some form of reflection.

There it is: The "r" word.  Reflection is generally quite costly and so immediately handicaps its appeal.

A "poor man's" reflection might be to implement a series of "if/else" blocks to check types and perform some appropriate casting.  I think we can see that this could and would become very ugly and difficult to maintain.

What about full-on automated reflection?  Well, we run into a bit of a snag with that, too: In a typical assignment operation, we have a left-hand side (LHS) and a right-hand side (RHS), e.g. TheClass theClassInstance = new TheClass();.  The RHS of the assignment is fairly straight forward with reflection.  Since SDN persists a __type__ attribute/property into a given node/relationship, we can fetch it and use it for casting (since it's typically a fully-qualified type name).  It might look something like this:

// n = Node object in question
String theType = (String)n.getProperty("__type__", "");

// we could then make the RHS of the assignment something like: (LHS) = template.convert(n, class<theType>;

But what about the LHS?  Without an "if/else" block, how can we treat the returned string theType as a first-class citizen that would declare the type for the LHS?  As far as I'm aware, there is no way to do this (of course there might be ways I've just not seen, but, I have a feeling they'd be just as expensive as the rest of the assignment).  Java is a strongly-typed language, and so I'm sure most of us would expect this outcome.

So we see that this "solution" isn't really much of one.

Solution 3

Somehow modify the query to work with a @MapResult-annotated interface to deal with the results (note: At least as of SDN 2.3.1, @MapResult has been deprecated in favour of @QueryResult.  I haven't done too much with @QueryResult so your mileage may vary).  This obviously requires more knowledge ahead of time of the types of paths, nodes and relationships you're planning on returning which may limit the kinds of heterogeneous queries you can execute.

So What Else Can Be Done?

I recently attended GraphConnect 2013 in New York City where I had a chance to meet up with Michael Hunger (a name which should require little or no introduction in the graph database/Spring Data continuum).  We had a great conversation about the very subject of this post.

His overall insights into Spring Data and its purpose, merits and detractors were very helpful, especially from a conceptual standpoint.

The number one point to take away--and perhaps it's quite obvious but it's worth reiterating--is this: Spring Data is not a magic bullet.  Given the differences in concepts here (i.e. a strongly-typed, object-oriented language and a graph-based, schema-free data store), there is bound to be limitations.

Spring Data's strong point is ease of integration.  A typical use case for SDN is likely to be one where relatively few nodes/relationships are needed to be returned.  SDN is not meant to necessarily "explore" graphs.

In order to truly resolve such differences, it would seem to me to make more sense to either layer SDN on top of another layer of abstraction, or even to go direct to the Neo4j API.

Perhaps an even better approach would be to use a Domain-Specific Language (DSL) such as Groovy or JRuby; something that is much more loosely-typed, flexible, and still able to be integrated into the Java stack.

(Shameless plug: Check out Pacer, a powerful, JRuby-based graph traversal engine.)

Summary/Conclusion

In this post, we have seen that exploring subgraphs and paths with SDN is not as straightforward as we'd perhaps like; however, it is clear that SDN was not built to accomplish such features (at least not yet).

Spring Data for Neo4j's strong suit is ease of integration.  As should be evident from this post and others, it is easy and straightforward to get SDN into existing/legacy Java applications, and to quickly stand-up Java-based applications that rely more on "end results" than "exploration" per se.

Ok, folks; as always, I definitely welcome comments,feedback, and questions.  If you can think of a better way to approach this kind of problem space or even if I have something wrong here, please do let me know and I'll be sure to make good use of such feedback.

Thanks for reading, and we'll see you on the next post!

Persisting to Neo4j via Spring Data (or, "Aren't We Persistent?")

Hi gang!

Ok, I'm back with a new post, this time with a post about a couple quirks I ran into while implementing some test cases for Spring Data using Neo4j.  They're not bugs by any stretch; it's just new behaviour to get used to as you venture into the Spring Data world (which I'm loving, by the way).

During my copious amounts of downtime (that should be read while imagining me rolling my eyes so hard that I fall over backwards in my chair), I've been putting together a little playground for me to mess around with and play with Spring Data.

It's definitely evolving and changing as I change things up and try out new ideas, and I fully plan on sharing more about this in future posts.

For now, though, I'm just discussing a couple potential pitfalls newcomers to Spring Data might fall prey to (please pardon the prepositional phrase; I'm sure it won't be the last one).

Background
I've wanted to play with Spring Data a bit more seriously for some time now, so I started a few weeks ago and, I have to say, I'm loving every second of it.  (I'm already a huge Spring fan, and the annotations continue to make my life easier.)

My sandbox goes something like this: After having gone through the docs for Spring Data (especially "Good Relationships"), I thought I'd try out something similar for myself, borrowing the whole "store" concept (as it seems to me to be the best, first choice for implementing a graph database).

Instead of using the whole "movie store" concept, I switched to something a little different to avoid total code reuse (I do borrow some code from the link above but modify it an awful lot).

For any geeks around my age (or older), you will remember a certain computer software retailer called Babbage's.  I have fond memories of begging my parents to go into the store every time we passed one, which wasn't often (at least I don't think it was...).  GameStop Corporation went on to purchase Babbage's (and EB Games, and a bunch of other software retailers), so you're unlikely to see a Babbage's by that name.

With that short trip down memory lane finished (more like memory cul de sac), I decided to model my domain after the concept of a software retailer.  My store will cleverly enough be called Von Neumann's (any CS major and most geeks out there are currently groaning at that joke).

Domain Model
Currently, the domain model consists of the following:

Even looking at the UML diagram above, we can see that it's based on a graph model (can you pick out the entities and/or the relationship(s)?).

Test Cases' Setup
After setting up the relevant project (which I did as a Maven project), I set forth creating some tasks using JUnit.  Before creating the actual test cases, I needed to make sure I had my testing context setup.  I also needed away to ensure that any data being persisted was wiped clean after each run.

Fortunately, instead of having to create such functionality for my project, I learned that Neo4j already has a handy solution!  The ImpermanentGraphDatabase.  This little gem can be found in the Neo4j kernel.  Specifically, I added these lines to my POM (you can see the specific version I'm using, too):


<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-kernel</artifactId>
<version>1.8.M03</version>
</dependency>

...and then adding the following line to my testing context:

<bean id="graphDBService" class="org.neo4j.test.ImpermanentGraphDatabase" destroy-method="shutdown"/>

And presto!  A suitable testing graph database for my test cases!

(Warning: I am using the latest version that I found worked best for me and is compatible with all my other dependencies.  ImpermanentGraphDatabase as available in earlier versions of Neo4j, as well.)

It should also be noted that I make use of both the Neo4j repositories interfaces AND the Neo4jOperations class for persisting and retrieval.

Another note is that I've made the entire test class @Transactional.

Test Cases Proper
I'll list two of them below and a couple of the quirks I noticed.

Ensuring a Customer Can Make a Purchase
This test case consists of creating a Customer object, a couple Game objects, and making sure that Purchases can be created, persisted and retrieved (along with the associated entities). 

 
 @Test
 public void customerCanMakePurchases()
 {
  // setup our game constants
  final int QTY = 1;
  final String GAME_TITLE = "Space Weasel 3.5";
  final String GAME_TITLE_2 = "The Space Testing Game";
  final String GAME_DESC = "Rodent fun in space!";
  final String GAME_DESC_2 = "Tests in space!";
  final int STOCK_QTY = 10;
  final float PRICE = 59.99f;
  
  // setup our customer constants
  final String FIRST_NAME = "Edgar";
  final String LAST_NAME = "Neubauer";
  
  // create our customer for this test
  Customer customer1 = new Customer();
  
  // set the customer's properties (NOTE: "firstName" is an indexed property in the Customer entity, but "lastName" is not!)
  customer1.setFirstName(FIRST_NAME);
  customer1.setLastName(LAST_NAME);

  // create our games for this test
  Stock game1 = new Game(GAME_TITLE, GAME_DESC, STOCK_QTY, PRICE);
  Stock game2 = new Game(GAME_TITLE_2, GAME_DESC_2, STOCK_QTY + 5, PRICE + 5);

First, do the setup.  (And, for the sake of brevity, I'm leaving out the annotated entities.)

Nothing strange going on here--just creating two games and a single customer.  It is worth noting (for later on) that "firstName" is an indexed property of the Customer entity/node.  This means that it is searchable (also recall that Neo4j's default indexing engine is Lucene).

It had to be done.

The games we've chosen are clearly AAA-title games.  These tests should be interesting.

  // save entities BEFORE saving the relationships!
  template.save(game1);
  template.save(game2);
  template.save(customer1);

  // make those purchases! Support our test economy!
  // (NOTE: "makePurchase" actually uses the "template" parameter to persist the relationship, so no need to do it again)
  Purchase p1 = customer1.makePurchase(template, game1, QTY);  
  Purchase p2 = customer1.makePurchase(template, game2, QTY);

Above, we make sure to persist the 2 games and single customer.  We also do this prior to persisting any relationships.  This is necessary.  In this case, I make use of an instance variable called "template" which is actually an instance of  Neo4jOperations.  This is one way of accessing the necessary persistence/retrieval functionality we need.

We then create 2 Purchase objects/relationships (Purchase is actually a relationship entity).  It is also worth noting that, instead of using the "template" object to persist the relationships, I've followed the Neo4j tutorial book "Good Relationships" and attempted another way of doing persistence, i.e. by passing the "template" object into the necessary method and having the method (in this case makePurchase) actually do the persisting of the newly-created Purchase.

Again, both "game1" and "game2" need to be persisted prior to persisting any relationships between them.

Still with me?

  // retrieve the customer
  Customer customer1Found = this.customerRepository.findByPropertyValue("firstName", FIRST_NAME);
  
  //
  // Tests
  //
  
  // can we find/retrieve the customer?
  assertNotNull("Unable to find customer.", customer1Found);
  
  // can we find the specific customer for which we are looking?
  assertEquals("Returned customer but not the one searched for.", FIRST_NAME, customer1Found.getFirstName());
  
  // does the retrieved customer have its non-indexed properties returned, as well?
  assertEquals("Returned customer doesn't have non-indexed properties returned.", LAST_NAME, customer1Found.getLastName());

  // retrieve the customer's purchases
  // (NOTE: We case as a Collection just to make checking the number of puchases easier)  
  Iterable purchasesIt = customer1Found.getPurchases();
  Collection purchases = IteratorUtil.asCollection(purchasesIt);
  
  // do we have the correct number of purchases?
  assertEquals("Number of purchases do not match.", 2, purchases.size());

So now we get to some actual testing.

The tests above are all straightforward.  We ensure the following:

1. We can retrieve a persisted node, specifically via an indexed property.
2. We can retrieve the correct persisted node for which we are searching.
3. We can view non-indexed properties from the retrieved node.
4. We can retrieve the correct number of relationships of the retrieved node.

As noted in Section 9.3 of "Good Relationships", we use Iterable for those node properties that are collections and are to be left as read-only, and Collection or Set for those collections that can be modified.

  // go through the actual purchases...
  Iterator purchIt = purchasesIt.iterator();
  Purchase purchase1 = purchIt.next();
  
  // retrieving objects via Spring Data pulls lazily by default; for eager mapping, use @Fetch (but be forewarned!)
  // ...this means we have to use the fetch() method to finish loading related objects
  Stock s1 = template.fetch(purchase1.getItem());

What if we want to view a node's related nodes' data?

By default, Spring Data loads an entity's relationships lazily, which makes perfect sense (just picture how much memory would be needed if you had a very large, highly connected graph).  Also remember that there are implicit relationships between entities if an entity is contained as a property of another entity.

(Courtesy of Paramount Picture's Forrest Gump)
"Mama said eager loading is like a box of chocolates: You never know what you're gonna get."
Well, at least the chocolates had an easily-determined, finite number in the box...

It is possible to have an eager retrieval by using the @Fetch annotation (be warned, though, that it will currently only work, by default, on node entities and collections of relationships that are based on Collection, Set, or Iterable; Spring Data may expand that in later releases, but I believe you can extend the mappings to work with other classes, if you so desire).

So, with our lazily-loaded relationships, we can use "template"'s fetch method to finish loading in the missing data.  It's as simple as that!  Anyone familiar with ORM will get this immediately.

  // can we retrieve our first purchase successfully w/ its details?
  assertEquals("Purchased item not persisted properly.", GAME_TITLE, s1.getTitle());

  purchase1 = purchIt.next();  
  Stock s2 = template.fetch(purchase1.getItem());
  
  // can we retrieve our second purchase successfully w/ its details?
  assertEquals("Purchased item not persisted properly.", GAME_TITLE_2, s2.getTitle());
  
  // if we're here, then all test ran succesfully.  Hooray!
 }

Above, we run a couple more tests to ensure that we can, in fact, retrieve and view lazily-loaded objects from Neo4j.

Nothing to it!

Making Friends the Easy Way: By Creating Them!
For these tests, we're going to have a look at something a bit more social, i.e. customers befriending other customers (how Utopian!).  I suppose we could make them "rivals" or "enemies", but that's a bit too sinister for this blog (for now...).

Anyway, I have prior to this test method a setup method (annotated with the @Before JUnit annotation) that creates 5 customers (if you're interested, I persist them using a CustomerRespoitory I created by extending the GraphRepository and RelationshipOperationsRepository interfaces).

 @Test
 public void customerFriends()
 {
  // add friends
  c1.addFriend(c2);
  c1.addFriend(c3);
  c1.addFriend(c4);
  c1.addFriend(c5);

  // be careful! setting a "Direction.BOTH" relationship in one node entity will have the ENTIRE relationship saved (*including the adjoining node*) when saving just ONE of the two entities!
  // ...if you save both, Neo4j will remove the duplication (and you'll be left wondering why c1 is a friend of c2, but not vice versa)
  
  // save c1's friends
  customerRepository.save(c1);

In the code above, we have the customer "c1" make friends with the other customers (he's a social butterfly).

Now, perhaps the most important part of this whole blog is shown here (and below).  It has to do with relationships, specifically those that are annotated as being "Direction.BOTH".

As you can see from the comments in the code above, we need to be careful about how we create relationships between nodes and save them.  If we were to create the relationship between, say "c1" and "c2", and then persist each node (and therefore the relationships, which the customer repository will handle), we would notice that the relationships have gone awry, and that the duplicate relationship from "c2" has been removed.

So, what we're going to do is the following (keeping in mind that "c1" and "c2" have already been persisted in the setup method):

1. Persist "c1" (and thereby its friendship to "c2").
2. Retrieve "c2".
3. Add any other friends' relationships to the retrieved "c2" (while not befriending back to "c1").
4. Persist "c2" (and thereby its friendships to those added in Step 3).

Step 1 is done above.

  // we can't just continue to add friends to this.c2, as once we try to save this.c2, it'll remove the duplicate relationship between c1 and c2.
  // ...so, to get around this, we retrieve the persisted object from the DB
  Customer c2Found = customerRepository.findByPropertyValue("lastName", C2_LNAME);
  c2Found.addFriend(c3);
  c2Found.addFriend(c4);
  c2Found.addFriend(c5);

  // save c2's friends, which will preserve the existing relationship with c1! Old friends can remain friends!
  customerRepository.save(c2Found);

As you can see above, we finish the remaining steps (2 through 4).

Again, note that we DO NOT create a reciprocal relationship from "c2" to "c1".  (One would hope a friendship relationship would be reciprocal; unless you have stalkers or something...)

This would totally help.

All that's left now is to run some tests to ensure that our friends have remained friends throughout all this persisting!

  // retrieve c1 for some tests
  Customer c1Found = customerRepository.findByPropertyValue("lastName", C1_LNAME);
  
  Iterable c1Friends = c1Found.getFriends();
  Collection c1FriendsSet = IteratorUtil.asCollection(c1Friends);
  Iterator custIt = c1Friends.iterator();
  
  int numFriends = 0;
  
  // let's make sure all of c1's friends were retrieved
  assertTrue("Friend not found.", c1FriendsSet.containsAll(IteratorUtil.asCollection(c1.getFriends())));
  
  // let's also make sure that c1 and c2 are still buds specifically (these two are inseparable...you should see them at ComicCon!)
  assertTrue("Friend not found.", c1FriendsSet.contains(c2));
  
  // let's make sure the exact number of friends returned is correct 
  while (custIt.hasNext())
  {   
   custIt.next();
   numFriends++;
  } // while
  
  assertEquals("Number of friends returned incorrect.", 4, numFriends);

  // if we're here, all is well! Huzzah!
}

Above, as in the first test, we make sure that all of the friendships have been properly preserved, both from "c1"'s and "c2"'s perspective.

Conclusion
In this post, we have seen the basics of persisting with Spring Data and a couple of the quirks I ran into.  These are documented within the Spring Data documentation, but it never hurts to bring these little nuances out into the light even further.

We also saw that the ImpermanentGraphDatabase is available to us through the Neo4j kernel which is a wonderful tool for implementing test cases with quick setup and teardown--no needing to write initializers and cleaners for a Neo4j installation!

So there we have it!  A first pass through persisting with Spring Data and implementing some unit tests using Neo4j and JUnit.

If anyone has any questions or I've made a mistake, please feel free to leave feedback.

We'll see you on the next post!

Fun with Gremlin (not related to the movie or the car))

So you have all of this graph-tacular data in your graph database (for this post, I'm using neo4j).  It looks slick with its vertices and edges.  People stop by your desk to say, "Is that the new connect-the-dots app you're working on?"

After staring at them for a couple moments and repressing the urge to sell them the "app" for five bucks, you start thinking about how you're going to access and use this wonderful data.

"If only there was a way to query the data..." you wonder.

While SQL is the querying language of choice for relational databases, there is no real "standard" as far as NoSQL databases go (that's a subject for another time).

In the world of graph-driven databases, there are options:

• Gremlin; a Groovy-based querying language that can handle any type of query.  This language is perhaps geared more towards those with more of a math- or graph-based background as the syntax is nothing like SQL.
• SPARQL; a popular query language for RDF graphs.  This language is likely more easily picked up by those with an SQL background as the syntax is more SQL-like than Gremlin.
• Cypher; a neo4j-specific querying language.  This language currently only allows read-only queries of graphs (i.e. no inserts, updates or deletions).  Like SPARQL, Cypher also takes its cues from SQL.

Neo4j folks: At the time of this writing, neo4j comes pre-loaded with plugins for both Gremlin and Cypher.  (As I understand things, there is currently a ticket open in the neo4j community to develop a SPARQL plugin, but has not yet been completed; there are likely complications stemming from the fact that neo4j is not an RDF database at its core.)

In this article, I'm going to cover some basics in using Gremlin.  By using some concrete examples, I hope to demonstrate a bit of the power behind using a graph-based database!

For the purposes of this post, I'll be making use of a small graph I created that contains some people, who they know, and what they've purchased.  I expect this graph to grow and change as time goes on, but, that's where it stands for now (this is the beauty of schema-less data).

So far, Gremlin is the only querying language I've used with graph databases, hence this article making use of Gremlin.  The good thing about this Gremlin is that it won't require a new muffler and you don't have to worry about feeding it after midnight.

Getting started with Gremlin and Neo4j is easy enough.  It's a plugin that comes with Neo4j, so all you need to do to begin is to open up your Neo4j web admin instance, click the "Console" menu option at the top, and select the "Gremlin" option from the top-right of the console that appears.

At this point, you're faced with the currently-available variables and a gremlin overlooking them, like so:

We see that the variable g contains the current graph.  If we issue the query "g.V" (without the quotes, as always), we get a list of all the vertices (nodes) in the graph; however, this information is not incredibly useful as you're only given each node's ID.

Let's say some (but not all) of our nodes have been given the property "Name".  If we try using "g.V.Name", we'll again see a listing of the nodes; however, the value of the "Name" property for each node (if available) will appear (if "Name" isn't a property of a specific node, "null" will appear; also, note that Gremlin is case-sensitive).

We can also similarly view a list of the edges (relationships) by issuing "g.E" to the console; this time, however, in addition to the edge IDs, we also see the type of relationship and the adjoining vertices (nodes), e.g. 1-KNOWS->6.  Note that you can see that these edges are directed!  In this case, we see that the edge goes out from node 1 and goes in to node 6.  Useful stuff.

We can also (similarly) view a property of the edges (if it exists) as we did for the vertices, e.g. "g.E.Quantity".

Identifying individual vertices and edges is simply a matter of knowing each one's ID number.  Obtaining a reference to a node (which, yes, can be assigned to variables for easier reference) involves a call like "g.v(6)" or "g.e(3)" (note the casing).

You can examine the value of an individual node's/relationship's specific property by querying it much like we did above, e.g. "g.v(6).Name".

Want to know all of the edges coming out of a node?  "g.v(6).outE" will do the trick.  Similarly, if you want to know all of the edges coming in to a node, we can use "g.v(6).inE".

We can also go one step further and use the "inV" and "outV" steps to identify the nodes on the ends of edge.  "inV" will correspond to the node at the head of an edge (also known as the "incoming vertex"), whereas "outV" will correspond to the other side of an edge (an "outgoing vertex").

You can also use "bothV" and "bothE" to get both incoming and outoing vertices and edges (respectively).

So if you want to travel from one node to another, you might do something like: v(1).outE.inV.  You can shorten this by using: v(1).out.  There exist similar constructs for "in" and "both". (Note that for "in", it's a short-cut for "inE.outV".)

Have more than one relationship connecting a node?  No problem!  You can access specific ones via something like this: v(1).out('LIKES') (this will take you to the node on the other end of the 'LIKES' relationship for node v(1)).

(Gremlin's github page has a good basic tutorial about all this.)

I could go on at great length about the features of Gremlin, but I think this is a great starting point.  I'm going to include some concrete examples below.  If I use any constructs or syntax that doesn't make sense, I very much encourage you to visit the Gremlin wiki page to look up the answers; this is a good little exercise, especially for the "groupCount" and "cap" constructs.

The graph below assumes one that has nodes describing products and people, and has relationships showing purchases and who knows whom.

How many times has each product been purchased?

g.V.inE('PURCHASED').inV.ProductType.groupCount.cap

Which products have been purchased more than once?

g.V.filter{it.inE('PURCHASED').count() > 1}.ProductType

Who is known by more than one person in this graph (we define 'knowing' as sharing an edge/relationship--in or out--with another person)?

g.V.filter{it.bothE('KNOWS').count() > 1}.Name

Who knows the most people, and how many people do they know?

g.V.bothE('KNOWS').outV.Name.groupCount.cap

How well-known is each person?

g.V.both('KNOWS').Name.groupCount.cap

(By the way, if anyone notices anything wrong with anything above, please let me know as I'm always looking to evolve and develop my knowledge of, well, everything.)

So you begin to see the power of what we can extract out of a graph database!  Personally, I'm tempted to find out "who is your daddy and what does he do?"  Such a question would be relatively straight-forward to figure out!

Ok, I think that's enough for now.  Hopefully this is a decent (but brief) introduction to the world of Gremlin and querying graph databases.  I know writing this has forced me to examine in more depth exactly what exactly these queries actually do.

I'll see you next time!

Neo4j: First Blood

That's one serious-sounding title.

This post is about my first foray into the graph database world.  I chose as my first victim/offering Neo4j.  I'll likely end up writing more than a few articles about this particular database, but, I thought I'd start with the basics, including the following:

• Download
• Installation
• Configuration
• Poking around

(And yes, "poking around" is a sanctioned technical term.)

So, a bit about Neo4j to start!

• It's been around since 2007.
• NUMBER ONE SELLING POINT FOR ME: It's ACID compliant!  Not too many NoSQL engines that I've seen (yet) are, though for various reasons that are well outside the scope of this post.
• As you may have guessed, it's primarily meant for integration with Java.
• It can also be integrated with Spring (a big plus, if you ask me) via Spring Data (POJO development FTW!).
• Its API is REST-based, and so can be utilized by just about any platform (though you'll likely have to write your own wrapper, unless you can find one out there in the open source world).
• There are some ready-made wrappers for some platforms available, such as Python and Ruby.
• It's available for Windows, MacOS and Linux-based OSes.
• It's available in both 32-bit and 64-bit for Windows and Linux.
• It's available in 3 versions, including the Community version, the Advanced version, and the Enterprise version.  As you'd expect, the Community version is open source available under GPL (the other versions are covered under AGPL).
• It comes ready-to-run with a version of the web/app server Jetty.
• It comes with a built-in web admin console (hence the need for Jetty).
• Following in the NoSQL tradition, it scales very well for Big Data.\
• Its name lends itself well to any number of The Matrix jokes.

I strongly suggest going to their website (www.neo4j.org) to do a little research of your own.

Download

Given that I'm just looking to get my feet wet with Neo4j, I downloaded the Neo4j v1.6 Community Edition 64-bit Linux package from my ready-made VM (coincidentally named Morpheus) running CentOS (sorry Windows users).  Read: I can't be bothered downloading the source and compiling it.  Note that Java 1.6+ is required; a complete set of requirements can be found here.

The archive is only about 37MB (give or take) and so completed relatively quickly over my bonded DSL connection.

Installation

After un-tarring the package and moving it into an appropriate directory (I'm a sucker for /etc) and starting the Neo4j server from the command line via bin/neo4j start (don't worry; there's a README.txt in the root of the installation directory that has all the quickstart instructions in it), I was ready to rock!

(I should note that I did get a couple warnings, shown below, but it hasn't seemed to have affected anything just yet, likely given how small my current graph is.

WARNING: Detected a limit of 1024 for maximum open files, while a minimum value of 40000 is recommended.

WARNING: Problems with the operation of the server may occur. Please refer to the Neo4j manual regarding lifting this limitation.

)

Or so I thought.

Configuration

If there's one bone of contention I have with Neo4j, it's that finding the appropriate (and up-to-date) documentation for the config files takes a bit of digging (it's not impossible by any stretch, though).

As I quickly found out, trying to access the web admin console that comes with Neo4j (very handy, I must say) outside of localhost is a non-starter out of the box.

Did I pack it in for the day and go back to flipping through Steam for cheap games?  No!  I did some "research".

Here's the solution: In order to get Neo4j's web admin to work from somewhere outside of localhost, change the neo4j-server.properties in the install directory's /conf directory (go figure).

Commented out towards the top of the file is the property org.neo4j.server.webserver.adddress".  Uncomment it and change it to the IP you want to bind the server to (the property does note that there are security concerns to consider, so, you may want to consult the Neo4j documentation before doing this).  

You can also change other settings in this file, e.g. getting it to work over HTTPS, changing the default ports for each, etc.

(Note: The web admin defaults to running over HTTP on port 7474 and over HTTPS over 7473.)

So, after making the change to the appropriate IP and restarting the Neo4j server, I tried pointing my browser back at the Neo4j web admin.

Success!

Poking Around

Without going in to too much detail (I'll likely do that in subsequent posts), the Neo4j web admin has 5 distinct sections to help manage your installation:

1. Dashboard
2. Data browser
3. Console
4. Server info
5. Index manager

Each one is fairly self explanatory.

The dashboard provides at-a-glance information about your server over a specified time line, such as the total number of nodes, properties, relationships and relationship types.

The data browser allows you to perform basic CRUD operations via a GUI.  You can also perform look-ups (consult the Help icon immediately to the right of the search button for more details on exactly what you can search for).  In other words, you can create a graph right then and there.

You can also flip the view to a graphical representation of the current graph and manipulate it (via click-and-drag) directly.  This is perhaps the coolest part of the web admin console (hey, we all like cool features!).

That's some serious badassery right there.

Next we have the console.  This is a great way to get familiar with the languages used to query Neo4j, including HTTP (i.e. accessing the REST calls), Gremlin (a Groovy-based querying language becoming common across multiple graph databases; it seems to be mainly for those coming from a math/graph background), and Cypher (Neo4j's own querying language; it seems to be mainly for those coming more from an SQL background).  

A quick note: At the time of this writing, Cypher only allows for read-only queries, whereas Gremlin allows for both reading and writing.

Next up, server info.  This is just a way to view (read: read-only) the server's configuration information.  No biggie.

Finally, we have the index manager.  Now, this is something I'm sure I'll be getting into a lot more as time goes on.  It's worth noting that Neo4j is built using the Lucene project for indexing, so this is very promising (especially for those familiar with Lucene and/or Solr). This makes a great deal of sense given the concept of properties for each node (full text search is going to be very important).

Regardless, you can create and manage indices for both nodes and relationships here.

So there we have it: My first venture into graph databases.  I'll admit I picked Neo4j first based on my initial research into graph databases.  It does seem to be the most popular graph database at the moment, so I look forward to seeing what it can do.

In subsequent posts, I'll be monkeying around with the querying languages, creating and modifying graphs, messing around with indices, and all kinds of other good stuff.

I hope some of you out there found this somewhat useful/informative/cool.  Well, I know I found it cool, but then I always was kind of odd...

Until next time, when we'll go Graph to the Future!  (Sorry, couldn't go an entire post without making at least one movie-based pun.)

Basic Graphs

The purpose of this post is to give a common footing for those reading to understand what I mean when I talk about a "graph".  The field of graph theory is a very deep and well-explored one.  The real trick here is trying to give an introduction to graphs and graph theory without getting lost in too much detail.  I will endeavour to keep this post to the utter, simplified basics of graphs.

(My apologies to anyone who already knows all about graph theory; this is meant more for those who might not have been exposed to graphs before, as well as to establish some common terms and nomenclature I'll be using in this blog.  If some of the definitions seem too simplistic, please bear in mind I'm trying to just get the basics out there so anyone can understand the concepts.)

(And really, Wikipedia is a great resource for those wanting a slightly more in-depth and well-structured discussion on graphs and graph theory.)

When most people think "graphs", they think of a series of bars or lines depicting their Q4 growth projections.  (Ok, most marketing and sales people I talk to do, anyway.)

We also think of graphs like the following (which is one of my personal favourites from our good friend Jay-Z):

(Source: http://mammon.typepad.com/photos/uncategorized/2007/09/13/99problems_graph.jpg)

Alas, these are not the graphs I'm referring to in this blog.  The kind I'm talking about is a little bit different; though, at its core, perhaps we can take a graph to mean (at a very oversimplified level), "a graphical representation of data".

The kind of graph I'm referring to in this blog has to do with the representation of objects and their interconnections.  Because I'm terrible at drawing (and lazy), I'll show a simple graph that illustrates what I'm saying and then try to describe what's going on.

This, my graph-fiends (yes, I just made that horrible pun up), is one example of a graph.  What does this graph consist of?  Here's what we have:

• 6 vertices or nodes (the circles), and,
• 7 edges or relationships (the lines between the circles).

At a first glance, that's all there is to it.  In mathematics, we commonly use the terms "vertices" and "edges" (the vertices being the circles/points, the edges being the lines between); however, it's likely a bit more intuitive to most to use the terms "nodes" and "relationships" in place of "vertices" and "edges", respectively.  I'll try to stick to using "nodes" and "relationships", though I may occasionally use the other terms interchangeably.

Is a graph still a graph if there are no edges involved?  Absolutely.  This is one example of a "null graph".  (For those interested, at the other end of the spectrum, a "complete graph" is one whose vertices are all connected/adjacent to each other, for a total of n(n-1)/2 edges, where n is the number of vertices.)

Edges/relationships can be "undirected" (like the example above), or "directed" (like the example below).  Directed edges have a source and a destination (this is useful for showing relationships between two objects).  Undirected edges can also be used to show a bi-directional relationship (i.e. one that exists in both directions); for example, Tom likes Dana, and Dana likes Tom.

Still with me?

So, we know that a graph consists of a series of vertices or nodes that can be connected via edges or relationships.  These edges can be undirected or directed.

At its core, that is what a graph is.

By now, if you haven't already, you're seeing how this might be useful to represent highly connected data.  Social networks are a great example (think of how you could represent your "friendships" on Facebook or MySpace with a graph; this is actually called a "Likes" graph).  Try to think of other information you could represent in graph fashion.

What about a family tree?  For sure.  As another matter of fact, a tree is actually a type of graph that is directed and acyclical (meaning that there are no loops, i.e. if you start following the directions on a graph, you travel each node exactly once).

Weight is another concept we come across in graph theory and graph databases.  Think of weight as being how important or strong a relationship is between two nodes.  For example, a higher weight can imply a stronger relationship between two nodes representing, say, two friends.

This is an important concept as we can use it to determine a shortest path or least costly path between two nodes.  (See an explanation of paths further below.)

One slightly more complicated idea is that of degree, which represents the number of edges attached to a node.  So, a node with a high degree will be one that has a lot of relationships attached to it.  This can be very useful for determining just how popular, for example, something or someone is.  (Cue jokes for this blog.)

One last concept to get out there before wrapping this article up is the concept of a walk or traversal.  A traversal is taken to be the resultant path found by starting at a node X and following to some other node on the graph Y and the edges and nodes visited during the trip from X to Y.  This concept becomes important as we start to look in to graph-based databases as traversals are part of what we can use to extract useful information from a graph (e.g. "Who are all of my cousins?") and to even predict and recommend a product for someone based on their browsing history.

Ok, I think that's enough math-like stuff for one post.  The definitions given in this post don't even scratch the surface of what's out there in the world of graph theory.  While I'm no serious graph theorist academically, I'm sure there are some standard texts out there.  That said, as mentioned earlier, Wikipedia is really a great resource for exploring the world of graph theory a bit more.

I'll introduce other graph-related concepts as they become needed (e.g. searching or path-finding); however, if there are requests for discussing specific graphing concepts, I'd be happy to address them (provided I can speak intelligently to them, of course).

Now go weird your friends out with jokes that their family tree isn't really a tree as it has loops!

So, You Want to Be a Grapher?

I've managed to resist the urge to setup a blog, until now.  A good friend and colleague of mine convinced me to do this based on a discussion we had over a beer the other night (and, let's face it: that's always the best place to have such ideas).  So, Martin, thanks for that.  I think.

So, the point of this blog, you wonder anxiously?  What a great question for a segue into an introduction!

I've been working in the IT field as a software engineer for some time and am currently the VP, Technology for a downtown-Toronto based software development firm.  As such, it behooves me (what a great term) to at least try to keep up-to-date with emerging technologies, especially as they mature.

To that end, as of late, I've become fascinated with the whole NoSQL paradigm.  Having spent most of my professional career dealing with RDBMSes, I was curious as to how the whole Big Data notion fit into things.  Browsing through the myriad niches of NoSQL--from document-based to key-value based--and leaning more about the whole movement along the way, I came across one particular type of NoSQL database that really got me glued to the ceiling.

Graph-based databases.

Coming from a background in not just computer science and software engineering, but mathematics as well (I attended the University of Waterloo up here in Ontario, Canada), the graph paradigm spoke volumes to me.

Sure, my math as it pertains to graphs may be a bit rusty, but it's something I clearly remember being rather interested in (should have taken more graph theory courses...).

Even more exciting is the fact that such databases existed.  For those of us who understand (or at least know about) graphs, I think it's safe to say that we can all appreciate the representation of social networks, semantics, and other relationship-driven data, as graphs.

What hit me like a tonne of bricks (Lego or otherwise) was that graph-based databases have actually existed for some time.  How long, exactly, I'm not 100% sure yet (as an example, one such database is neo4j which has been around since 2007).

The fact that highly-connected data could be so easily and directly represented in technology (along with the above) is what drove me to start digging deeper.  Such data exists in abundance around the web (and elsewhere!).

So began my adventure into the realm of graph-based databases.

"What do you hope to accomplish with this blog?  What are your goals?"  Another astute question; one that I have anticipated to some degree:

• As much as I hope to inform and educate, this blog is as much to serve as a record of my journey into graph-based databases.  I'm hoping that as I continue to learn that others may hopefully glean some knowledge/insight from my posts (however little or much that may be).
• As stated above (I'll repeat it for the sake of being explicit), I hope to educate and inform people as the world of graph-based databases expands and matures.
• To explore graph-based databases and their related concepts.  This may include information on other aspects of NoSQL, or even deeper dives into graph theory.
• To give a base from which to derive a basic understanding of graphs and their databases.

I think that's about sufficient for now.  Should I need to revisit these goals, I will do so when the time comes.

If anyone actually reads these posts, I greatly welcome feedback and comments.  This blog may not be everyone's cup of tea, but, I'll take that chance.  Let's try to keep the comments at least somewhat constructive (I'm sure some out there will pick apart things that I may get wrong, but I fully expect and welcome such criticism and corrections).

For the time being, I'll link to any sources from the web that I use.  If I miss any or if people feel that I'm not citing enough, please let me know.  It is not my intention to plagiarize anyone's work as I know developing it comes with no small effort; rather, it is my intention to aggregate and disseminate knowledge wherever possible.

Ok, I think this post is long-winded enough.  The frequency with which I post remains to be seen.  While I'm sure most won't exactly be waiting with bated breath, maybe they will!

With that, I look forward to posting more soon!  Back to the grindstone!


(And Happy Valentine's Day to all of you out there!)