Difference between revisions of "Networks Community Detection"
From OPOSSEM
Derek Ruths (talk | contribs) (→Introduction) |
Derek Ruths (talk | contribs) |
||
Line 23: | Line 23: | ||
Given an assignment of nodes to communities, modularity will return a value between -1 and 1 indicating whether there is less (< 0) or more (> 0) connectivity among the individuals belonging to the communities that would be expected at random. In community detection, we're seeking an assignment of nodes to communities that maximizes the modularity of the network - since an increase in modularity indicates that we have found labeling that increases the connectivity within the communities. | Given an assignment of nodes to communities, modularity will return a value between -1 and 1 indicating whether there is less (< 0) or more (> 0) connectivity among the individuals belonging to the communities that would be expected at random. In community detection, we're seeking an assignment of nodes to communities that maximizes the modularity of the network - since an increase in modularity indicates that we have found labeling that increases the connectivity within the communities. | ||
− | =Graph Partitioning= | + | =Searching Community Assignments= |
+ | As mentioned above, the number of possible community assignments for a network is too huge to permit us to test each one. This means that we have to only test promising community assignments. In order to enable the computer to do this work for us, we have to design algorithms that can quickly identify community assignments. Also mentioned above, these algorithms also need to know when to stop such that it's pretty sure that it's already seen and evaluated the best community assignment. In this section, we'll discuss a number of different methods for doing this bounded search. | ||
+ | |||
+ | ==Graph Partitioning== | ||
The most basic way to do community detection is to (1) tell the computer how many communities it should look for and then (2) for the computer to go find a way of assigning nodes to that number of communities that maximizes the modularity of the network. A number of methods exist that do this including spectral partitioning and the Kernighan-Lin algorithm. | The most basic way to do community detection is to (1) tell the computer how many communities it should look for and then (2) for the computer to go find a way of assigning nodes to that number of communities that maximizes the modularity of the network. A number of methods exist that do this including spectral partitioning and the Kernighan-Lin algorithm. | ||
However, this approach has a serious conceptual problem: you must know the number of communities you are looking for. In general, this will not be the case when working with a social network. Furthermore, even if one knows that there are, for example, two large groups present, it still may not be advisable to restrict the algorithm to searching for these two communities only, since more interesting phenomena and social structure might exist on another, finer level. | However, this approach has a serious conceptual problem: you must know the number of communities you are looking for. In general, this will not be the case when working with a social network. Furthermore, even if one knows that there are, for example, two large groups present, it still may not be advisable to restrict the algorithm to searching for these two communities only, since more interesting phenomena and social structure might exist on another, finer level. | ||
− | As a result, while partitioning algorithms exist, other algorithms that can discover arbitrarily many communities are typically more useful. We turn our attention to these | + | As a result, while partitioning algorithms exist, other algorithms that can discover arbitrarily many communities are typically more useful. We turn our attention to these in the next sections. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
+ | ==Edge Betweenness== | ||
One approach ... use of [[Networks_Centrality|betweenness centrality]]. | One approach ... use of [[Networks_Centrality|betweenness centrality]]. | ||
− | =Label Propagation= | + | ==Label Propagation== |
=Conclusion= | =Conclusion= |
Revision as of 07:38, 9 July 2011
Contents
Objectives
- Understand the difference between graph partitioning and community detection
- Survey the different methods available for community detection
Introduction
Algorithms for finding communities have many uses in network science, particularly in the social sciences. Since collective action is such a fundamental aspect of social and political behavior, identifying the groups to which individuals belong is an important task. Here we discuss a variety of methods for approaching this problem.
In network science, community detection amounts to assigning each node a community 'label'. For example, supposing that the network contains three communities, each node would be assigned a label of 1, 2, or 3 - indicating whether it belongs to community 1, 2, or 3. Community 1, then, consists of all the nodes that have been assigned the label '1', the same holds for communities 2 and 3.
Note that, given this setup, it's very easy to invent an arbitrary assignment of nodes to communities: for each node in the network, randomly select a community label (either 1, 2, or 3), and assign that label to the node. Of course, it's unlikely that any arbitrary assignment will be a good assignment, meaning that all the nodes labeled 1 actually correspond to a community in the real system. This means that we'll have to try a lot of different community assignments in order to find a good one - which is the crux of the community detection problem: generating different assignments of nodes to communities, evaluating how good the assignments are, and stopping when we find one we're happy with. As you can imagine, each of these steps is actually a non-trivial problem. Here we're going to look at ways of doing each.
Before delving into the different existing methods, let's first discuss why each of these problems is hard.
- Assessing the quality of a community assignment: despite much research in sociology, psychology, and computer science, we still don't have a definitive, quantitative answer to the question: what is a community. Nonetheless, we need some way of evaluating how good a community assignment is at grouping nodes into real communities. In network science, the focus has been on using the connectivity of the network to evaluate the quality of an assignment, operating on the assumption that in a real community nodes will be more connected than would be expected at random. This is what the modularity measure tries to capture - discussed below.
- Generating different assignments of nodes: as mentioned earlier, it's easy to generate a random assignment. However, we're looking for good assignments. The naive approach to generating different assignments is to simply look at every possible community assignment - if we do this, then we're guaranteed to find the right one. Unfortunately, this is computationally intractable. Consider a network of 100 nodes which we want to separate into 2 communities. There are exactly <math>2^{100}</math> different possible assignments - even if we can test 1000 assignments a second, it will still take more than <math>40,196,936</math> billion years to try all assignments. Clearly, we'll need to come up with a better way of identifying promising community assignments.
- Knowing when to stop looking: since evaluating all possible community assignments is clearly not an option, we not only need to know how to pick promising assignments, but we also need to know when to give up looking for a better assignment. Without a stopping condition, we could look for a long, long time, which would defeat the purpose of being selective about picking community assignments in the first place. Typically the way of generating community assignments and identifying a stopping point go hand-in-hand. They are typically aspects of the community assignment search problem, which is discussed in more detail below.
Modularity
Consider the patterns of connectivity within a community of people. What kinds of structural features might we expect to observe among these individuals? On plausible feature is that there is a higher degree of connectivity (more edges) among these individuals than we would observe for any random collection of people. This is the intuition that modularity captures.
Given an assignment of nodes to communities, modularity will return a value between -1 and 1 indicating whether there is less (< 0) or more (> 0) connectivity among the individuals belonging to the communities that would be expected at random. In community detection, we're seeking an assignment of nodes to communities that maximizes the modularity of the network - since an increase in modularity indicates that we have found labeling that increases the connectivity within the communities.
Searching Community Assignments
As mentioned above, the number of possible community assignments for a network is too huge to permit us to test each one. This means that we have to only test promising community assignments. In order to enable the computer to do this work for us, we have to design algorithms that can quickly identify community assignments. Also mentioned above, these algorithms also need to know when to stop such that it's pretty sure that it's already seen and evaluated the best community assignment. In this section, we'll discuss a number of different methods for doing this bounded search.
Graph Partitioning
The most basic way to do community detection is to (1) tell the computer how many communities it should look for and then (2) for the computer to go find a way of assigning nodes to that number of communities that maximizes the modularity of the network. A number of methods exist that do this including spectral partitioning and the Kernighan-Lin algorithm.
However, this approach has a serious conceptual problem: you must know the number of communities you are looking for. In general, this will not be the case when working with a social network. Furthermore, even if one knows that there are, for example, two large groups present, it still may not be advisable to restrict the algorithm to searching for these two communities only, since more interesting phenomena and social structure might exist on another, finer level.
As a result, while partitioning algorithms exist, other algorithms that can discover arbitrarily many communities are typically more useful. We turn our attention to these in the next sections.
Edge Betweenness
One approach ... use of betweenness centrality.
Label Propagation
Conclusion
References
Discussion questions
Problems
Glossary
- [[Def: ]]
- [[Def: ]]
- [[Def: ]]