increase it, 4GHz is kind of low for a 2500k, You should be able to get to 4.5GHz, but you might need to increase voltage.
If you want to OC your GPU, install MSI Afterburner. And Furmark for testing.
Also, about temps, 60C for CPU = Hot, only on full load (With Prime95)
80C for GPU = Hot, should be only when using Furmark, but can go higher.
HWMonitor for temps.
The CPU is 41-43 celsius on idle according to ASRock-software, with CPU-ratio jumping between 16.0x - 40.0x
And what voltage? If I remember right, there are a lot of Voltage-configs in BIOS so I wouldn't know which one to touch and how much more voltage. (My PSU is something along the lines of Corsair Golden ATX 750w Modular)
Also I had Prime95 but ended up deleting it. But will definitely check out MSI Afterburner, and maybe Furmark.
As for HWMonitor; GPU appears to be 49celsius on idle, with 1500 RPM fan speed.