From 1795 to 1904 the area was under Siamese administration with the local name of "Koh Kong". During the reign of King Mongkut the name Patchan Khiri Khet was given to the city as a counterpart to Prachuap Khiri Khan, a city on the same latitude which also had its name changed during the same year. In 1904, the region and the city of Trat was ceded to French Indochina in exchange of French troop evacuation from Chanthaburi. In 1907 Trat was returned to Siam in exchange for the Siamese province of Inner Cambodia while Koh Kong remained part of French Cambodia.

After Cambodia's liberation from the Khmer Rouge in 1979, Koh Kong province was quite under-populated. After the national government encouraged people to live in Koh Kong, there has been a net influx of people. It is estimated that the average annual growth rate in Koh Kong is 16%, which has put pressure on the mangrove resources in the province. Koh Kong's towns have developed rapidly partially in response to market pressures from Thailand and because of immigration from other parts of Cambodia.