Normally, when we say "prove it", what we are really talking about is the process of strengthening the neurological pathways in our brain that allows us to short-circuit the expectation of certain outcomes. The more the expected outcome occurs, the stronger the connections become, the better we are able to react to the likely outcome.
The stronger our brain believes something to occur, the more freaked out it is when that thing *doesn't* occur. Our brains have a pretty good handle on how to deal with situations it has seen before. A brand new situation can be very dangerous or fatal. "Being surprised" and "being shocked" are synonyms. The shock of our brain freaking out from a non-standard occurrence that does not have well worn neuro pathways.
That is of course my armchair neuro-socio explanation for what most people seem to mean about proving something.
Scientific endeavors have a different meaning of that phrase. Proving something scientifically is about bayesian statistics. We run experiments to generate evidence to support that what we believe is correct, or more importantly generate evidence to support that what we believe is incorrect. The more evidence that we have for something being true the higher the expected probability that thing is actually true. The phrase, "extraordinary evidence for extraordinary claims," is about the bayesian nature of this process. If we have a bunch of very good experiments leading us to believe a certain thing is true and we have a so-so experiment leading us to believe that same thing is not true, then we will tend to believe that the large amount of good evidence trumps the small amount of so-so evidence. After all, it is more likely that the so-so small amount of evidence is wrong or indicating something other than the stated conclusions than for all of the large amount of good evidence to be wrong.
I have no idea of what to think about philosophical methods of 'proving' something. I should probably start reading the classic philosophical works when I get a chance. On the one hand it kind of seems like a bunch of hot air, but then again they do come up with some surprisingly good observations. For example, isn't it suspicious that induction can only be shown to work via a process of induction. And speaking of induction ...
Proving things in mathematics is much more certain than in other fields. The proofs have nothing to do with intuition or what is expected. The proofs have nothing to do with statistical extrapolation or gathering evidence. The proofs can be verified regardless of how your thought process works. The proofs start with a system of rules and follow them exactly, until they reach a useful conclusion. And within their system (assuming no mistakes were made ... and assuming that the system is valid) they are absolute.
... Of course outside of their system they are meaningless. So a good question is, "What's the point?" Isn't this the same question that's asked in every math class ever?
I think I have an answer to this question. I didn't come up with it because I was trying to answer the question. I came up with it because I was trying very hard to figure out how mathematics works. So if you have category theory and lambda calculus, why are they both classified as math. Surely this is not just someone's opinion. There's got to be some unifying characteristic that makes them both math. Additionally, if you have an arbitrary math, how do you know it works. Maybe you can prove that it's sound and complete and consistent, but how are we so sure that our proofs won't evaporate before our eyes. The rules don't have any special powers, we just made them up. Why should we be able to prove anything lasting about them.
My conclusion is that mathematics are abstract causality machines. Causality has the same philosophical problems that math does. We don't really know why effects have causes. And as far as the physical universe goes ... even if we are able to eventually prove (in the bayesian sense) that everything is in reality very small strings that are vibrating in 11 dimensions, we still won't know why those strings feel compelled to obey a certain set of rules.
So why is studying math useful? Well, we can't prove that it is useful (in the mathematical sense). But mathematics are just abstract causality machines. And as far we can tell, the vast majority of reality "runs" on causality. Understanding mathematics helps you to understand reality, in the general sense.