performance issues on DU

[sweetheart]

As the forum will continue to grow, GD will only run faster and faster... to the point where the active page of discussions will be sooooo volatile, that no consistency exists.

Add to that a critical mass explosion of the user community that seldom do we see a name we recognize. DU, rather than becoming a tool of empowerment for the left becomes instead a place where liberal voices are marginalized by their sheer number.

Given these issues, the way the dialogues are indexed on DU is of serious concern now and ongoing for all of us. An example, is that were the servers powerful enough (and they are) the DU application could be set to NEVER archive. The ongoing dialogs would become recurrent trees of knowledge, some of them growing huge.

One of the problems is when dialog trees get more than 100 leaves. The performance degrades. This is largely as the leaf-index is composed instantly by an SQL query at the point where the page is rendered of the GD discussions. This SQL query deadlocks and hangs when the discussions are furious... as while some parties are trying to "lock and read" the index, others are trying to "lock-and-insert" new pages. It creates thrashing problems as users focus their locality one one thread and not another.

A pre-cache index system should re-hash all indexes every 5 seconds, and reroute all queries to this service. Then the leaf-trees will be fresh enough for real life... and the service will not deadlock on over-query as a leaf-tree gets hit. This way we can support many more active discussions.

And then... the next obvious question is how we index this knowledge so that a visitor to DU (us) is able to card catalogue their politics on the DU system. I would hope that the software designers are designing a system that actually allows for massive scaling.... as methinks the current system would scale into irrelevance with a 10fold trafic boost. Even if the servers scale, the application does not.

[Junkdrawer]

1. There's an "Ask the Administrators" forum...

I asked something similar, although not nearly as complicated...

http://www.democraticunderground.com/discuss/duboard.php?az=show_topic&forum=120&topic_id=1786

[newyawker99]

2. I am locking this thread.

Questions about Democratic Underground policies belong in the "Ask the Administrators" forum. All discussion topics relating to Democratic Underground policies, procedures, enforcement, etc., which are posted outside of the "Ask the Administrators" forum will be locked or deleted.


NYer99
DU Moderator