This is a genre that has been around for a long time. I can't even begin to go into its full history, but the basic formula is one that has endured at least since the 1980's: A police officer is assigned to his (it's almost always a guy) latest case, during which time he is also given a new partner (often but not always another cop or other law enforcement). Said partner is completely the opposite to him in terms of personality; usually one is more uptight and by the book while the other is more reckless and wild. Naturally the two can't stand each other but they're forced to work together anyway and by the time they've solved the case they are best friends.
It is a simple enough formula, and one that can be taken in plenty of directions storywise (exactly what the case is for example, varies widely between films). There are still patterns that are typically common. For instance, the two "buddy cops" are almost always men. Aside from a one or two exceptions, any female characters present in the story are usually a supporting role such as a love interest for one of them. I have yet to find a buddy cop film where the pairing is between a man and a woman.
That said, the combinations vary widely. Usually you have the dynamic of the uptight professional cop and the reckless wild one, but for a while it became popular with films such as Lethal Weapon to pair up a white guy and a black guy. With Hot Fuzz you have the cop who is so good at his job that he embarrasses everyone around him and the cop who at first has no idea what he's doing. The Canadian film Bon Cop, Bad Cop takes the basic formula but instead pairs together an English-speaking cop from Toronto with a French-speaking Quebecois officer. In a subversion of my earlier comment about how the "buddy cops" are usually men, we recently got The Heat, where the same formula is replicated but this time with both partners being women.
Then we get into weirder combinations. The infamous movie Cop and ½ sees poor Burt Reynolds forced to team up with a kid to track down the bad guys. Osmosis Jones does this kind of story but instead uses the setting of the human body, with a the "buddy cops" being a white blood cell and a cold pill who team up to track down a lethal virus. What I said earlier about having yet to find a pairing between a man and a woman might not be entirely true if you count Theodore Rex, which has the odd pairing of Whoopi Goldberg and a talking dinosaur. Yes, that actually happened.
So what is it that makes this genre so popular? Part of it might be just the creative combinations that different writers can come up (even if at times they get really weird or outright stupid), but a major part of the buddy cop film is in how the two main characters play off of each other. Naturally there's usually lots of action but it may just be that there is something emotional to be found about two conflicting characters who otherwise might have had nothing to do with each other bonding together over a crime, to the point where they often end with both partners getting together to solve another crime.
In this case, unlike perhaps a regular police-based narrative, such as the old TV series Columbo, the conflict and tension becomes more in how the two "buddy cops" have to figure out how to put aside their differences and work together to solve the crime moreso than the crime itself (although the crooks involved certainly don't help matters). After ;all while the investigation plays a major role in the story, the plots of both Hot Fuzz and The Heat were driven largely by the relationship between the two leads and how they gradually warm up to each other so they can finally take on the bad guys during the climax. The genre is a bit of a cliché now, but really the key to making a good "Buddy Cop" film is simply to have two leads who can really play off of each other.