Great stories, or at least a certain kind of story, work by proceeding along the following path: from false order through disorder to true order.
Before I go any further, I’m by no means an expert; this is just my personal observations and ideas, for whatever those are worth, presented for consideration.
As I said, false order through disorder to true order. What do I mean by that?
A story starts in apparent peace: things seem good, or at least tolerable. The community is intact, its institutions standing firm. Yet there is a problem; the protagonist’s life is, in fact, built upon sand. The problem is not immediately before us, but sneaks in, as it were, from around the edges. The community is in fact threatened or undermined by secret or exterior forces.
Then comes disorder. The threat manifests itself, no longer being hidden, but open. The institutions crumble or prove impotent and the protagonist is thrown upon his own resources. It no longer becomes possible to restore the integrity of the community from within, but requires unusual effort and atypical behavior to solve the problem.
Finally, the problem is solved and true order is established. The threat that undermined the apparent order has been removed, the fault lines have been filled, and the institutions are reestablished, having been purged of their impurities.
To take an example: consider the Harry Potter series (Spoilers ahead). We begin with false order: the wizarding world seems a wonderful, magical place; a haven from the stultifying world of the Dursleys. It is where Harry’s friends and interests all reside. There are threats, but they are external: Voldemort is in hiding and his followers have to act by cunning and secrecy. The institutions stand firm: Hogwarts, the Ministry, and so on all seem strong and safe.
Yet even now there is hidden disorder, not only in the continuing threat of Voldemort, but also in the structure of the wizarding community itself, which we begin to understand is rife with corruption and bigotry.
Then comes disorder. Voldemort returns. The Ministry is torn by corruption. Hogwarts becomes, not the triumphant heart of the wizarding world, but a fortress under siege. Harry and his friends are increasingly forced to work outside the normal patterns of behavior and even outside the law. Finally the community collapses entirely with the fall of the Ministry and Harry becomes an outlaw, culminating in a full-on battle royale.
But, at the end, true order is restored. Voldemort and his followers are defeated, the wizarding community is rebuilt on more humane grounds, having been partially purged of its imperfections by the war, and Harry and his friends take their place as hardworking citizens. The wizarding world is what it was supposed to be from the beginning.
Another example: Pride and Prejudice (again, spoilers). We meet the Bennet family, things are in a semblance of order. The family is not especially happy (with an ill-matched parental pair and three “very silly” youngest sisters), but they are respectable, reasonably well-off, and the household bumps along well enough. They are an integral part of the community and are generally well-liked and respected among their neighbors. Yet, this pleasant existence is built on a foundation of sand, as Mr. Bennet’s estate is entailed on the male line, and the family has no son. Once he dies, the estate will pass to his obnoxious cousin and the fate of his daughters will be shaky indeed. Thus, one of them has to marry. But this brings us to the other faultline under the Bennet household: the fact that at least half of it is very stupid, thoughtless, and ill-bred. When Jane, the eldest and prettiest of the daughters, falls in love with a rich and easy-going man, the rudeness and stupidity of her mother and younger sisters cause the man’s friends to quietly remove him from her company.
So, the Bennet daughters have to marry, but the foolishness of their mother and of some of their sisters make it unlikely. At the same time, there’s the hidden danger of Mr. Wickham; a charming, but completely amoral young man who has lately become acquainted with the young ladies.
Then comes disorder: Lydia, the youngest daughter, elopes with Wickham and lives with him for a time without marrying him. This threatens to cast the Bennet’s out of all society forever, and certainly to dash their hopes of marrying well. In the face of this threat, ordinary concerns fall by the wayside. Mr. Bennet rushes to London in a wrath, and, unbeknownst to him, so does Mr. Darcy: the fabulously rich, but socially-awkward young man who has fallen in love with Elizabeth, the second eldest Bennet daughter, and who is Wickham’s old enemy. Darcy hunts up and down London, dealing with the lowest kind of scum, until he traps his enemy and, to save the Bennets, takes him under his financial protection under the condition that he marry Lydia.
Finally comes true order: Jane and Elizabeth, against all odds, marry the men who have always loved them, being thus set up for life and ensuring the family fortunes forever. The threat of the entail is destroyed, Wickham is effectively neutralized, and the insipidity of Mrs. Bennet no longer matters.
A final example: Toy Story (still Spoilers). At first, everything seems idyllic. Woody is Andy’s favorite toy and the de-facto leader of the bedroom. He’s respected by his friends, and, apart from some minor bickering, they all get along fine.
But there is a hidden fault line in the arrangement, and it comes from Woody himself: he’s become so used to his exalted position that he doesn’t really know how to live without it. Unfortunately for him, that question is forced on him when Andy become infatuated with Buzz Lightyear, a flashy new action figure.
For a while, the community holds, though in a strained form. Then Woody’s jealousy leads him to go too far, sparking a crisis that drags both him and Buzz far from the safety and familiarity of the bedroom and threatens them both with destruction. Woody’s personal failures have brought disorder into his life and the lives of all his friends.
The only way they can restore order turns out to involve not only facing their own flaws, but also breaking the rules of being a toy. At the beginning, part of the life of a toy means returning to your proper place and freezing whenever a human comes near. Now, however, they not only break that rule but actually show their ability to move and speak. The institutions of the toy community will no longer avail them and they have to look to their own resources to restore order.
Finally comes true order: Woody has learned his lesson, he and Buzz have reconciled, and the toy community is stronger than ever, with no more fear of being replaced. The faults that threatened the stability of the community have been identified and corrected.
This pattern obviously doesn’t play out everywhere, but I come across it more often than not. It’s very useful in terms of reader investment; the first phase gives us a chance to get to know the world and see that it is something worth preserving. It makes us care. We care about the wizarding world because, with Harry, we see it as a fun, glorious escape from the boring and humdrum world of the every day. We care about the Bennet family because we like Elizabeth and Jane and we sense the real love that exists between the sisters and their worthiness to come through in the end. We care about Woody because we see how much Andy loves him and vice-versa and because we sympathize with the fear of being displaced by someone else.
The second phase threatens the thing we’ve come to love. The wizarding world might be turned into a nightmare and all the colorful characters destroyed. The Bennet family may be ruined forever and the beautiful, worthy elder sisters condemned to a life of spinsterhood. Woody and Buzz might be lost forever, depriving a nice little boy of his two favorite toys. The threats have become real and require our protagonists to stretch themselves and grow in order to escape. It rests on them to avert the disaster and save the thing we love.
Finally is the satisfaction of true order: having passed through the storm, our heroes are permitted to come to rest in a world freed from the threats that destroyed their initial contentment. Harry and his friends grow up, marry, and raise their own families in a world freed from the dangers they had to face. Elizabeth and Jane each end up with a wonderful man who not only loves her passionately, but who is wealthy enough to support the rest of the family if need be. Woody and Buzz become friends and assume joint leadership over the other toys, who have learned from their experience and no longer fear being replaced. The thing we love has been preserved, and we are assured that it won’t be threatened again, at least not in the same way. The problem has been solved, the community restored. All is right with the world.