It works pretty fast now even when removing multiple blocks on several chunks, one of the things I did was to change the way I was doing the merge, in the previous version I had to render both faces of each triangle because of situations like this:
When two voxels shared the same face but the face needed to look in both directions at the same time, the solution was simply to split the face in to two faces:
With this done now I can activate the culling and render each triangle only once.
I also Implemented the octrees to distribute the chunks and be able to do culling (still pending):
I hope to finish the frustum culling this week and maybe do some implementation for occlusion culling.