To begin with, both of them are abbreviations.
- tadaima is short for
tadaima modorimashita ただ今戻りました
- okaeri is short for
By the way, when a character is leaving home the words ittekimasu 行ってきます and itterasshai 行ってらっしゃい are used instead.
The phrase tadaima modorimashita, or tada ima modorimashita, can be divided into three parts:
- tada ただ
Just. Only. Merely. (adverb)
- ima 今
Now. (the time, noun and adverb)
- modorimashita 戻りました
Come back. (verb modoru conjugated to the polite past tense)
Therefore, tadaima is short for "I came back just now" in Japanese. Some time after someone comes back, he can say modotta 戻った or modorimashita 戻りました, but it won't be tada ima ただ今 anymore.
The word okaerinasai お帰りなさい comes from the verb kaeru 帰る which means "to go home."
Normally, a verb which is conjugated to its nasai form becomes an order, in fact, kaerinasai 帰りなさい is an order. If you say kaerinasai alone you're telling someone to go to his home, but you're not welcoming him, you're most likely just telling him off.
What changes that here is the prefix o お. This is the same prefix that appears in words like oniichan, oneechan, okaasan, otousan, etc. By using this prefix you're artificially making the word more respectful, which could be interpreted as "please go home." Though it can still be interpreted as telling someone to go away, it can also be interpreted as "please come home," that is, "welcome home," if home is where the speaker is at.