Hi, Somehow the Spark implementation crashes whenever the directed f

Reason I had met same error when running <code class="notranslate"

Setting Directed Flag Causes NullPointerException about node2vec HOT 1 OPEN

qantik commented on August 15, 2024

Setting Directed Flag Causes NullPointerException

from node2vec.

Comments (1)

datazhen commented on August 15, 2024 32

Reason

I had met same error when running val (j, q) = GraphOps.setupAlias(nodeAttr.neighbors) in GraphOps.initTransitionProb, This is due to the nodeAttr object being Null.

Detail

Some dst nodes should be included in the node2attr object but not in the following code:

    val node2attr = triplets.map { case (src, dst, weight) =>
      (src, Array((dst, weight))) 
    }.reduceByKey(_++_).map { case (srcId, neighbors: Array[(Long, Double)]) =>
      var neighbors_ : Array[(Long, Double)] = neighbors.groupBy(_._1).map { case (group, traversable) =>  
        traversable.head
      }.toArray
      if (neighbors_.length > bcMaxDegree.value) {
        neighbors_ = neighbors.sortWith{ case (left, right) => left._2 > right._2 }.slice(0, bcMaxDegree.value)
      }

Then when creating graph object by val graph = Graph(indexedNodes, indexedEdges), the dst nodes mentioned above, which are missing, will be created by default. And the format by default is [vertexId, Null] ,instead of [vertexId, NodeAttr]. So the error come.

Solutions

To solve the problem, some modules should be modified. The details is shown below:

GraphOps.initTransitionProb

    val graph = Graph(indexedNodes, indexedEdges).mapVertices[NodeAttr] { case (vertexId, nodeAttr) =>
      var path:Array[Long] = null
        if (nodeAttr != null) {  // add 
          val (j, q) = GraphOps.setupAlias(nodeAttr.neighbors)
          val nextNodeIndex = GraphOps.drawAlias(j, q)
          nodeAttr.path = Array(vertexId, nodeAttr.neighbors(nextNodeIndex)._1)
          nodeAttr
        }else{
          NodeAttr() // create a new object
        }
    }

Node2Vec.randomWalk

// add:.filter(x=>x._2.path.nonEmpty).
    val examples = g.vertices.filter(x=>x._2.path.nonEmpty).cache
...

// add the condition: attr.dstNeighbors != null && attr.dstNeighbors.nonEmpty
iter.map { case (edge, (attr, pathBuffer)) =>
            try {
              if (pathBuffer != null && pathBuffer.nonEmpty && attr.dstNeighbors != null && attr.dstNeighbors.nonEmpty) {
                val nextNodeIndex = GraphOps.drawAlias(attr.J, attr.q)
                val nextNodeId = attr.dstNeighbors(nextNodeIndex)
                s"$pathBuffer\t$nextNodeId" 
              } else {
                pathBuffer //add
              }
            } catch {
              case e: Exception => throw new RuntimeException(e.getMessage)
            }