Learning Japanese to read manga in Japanese and watch anime in Japanese is in fact far, far easier than most people think. Of course it's still no small task, but it is indeed smaller.
The reason for this is that, when you ask someone: "how hard it is to learn Japanese?" They will think you want to... well, "learn Japanese." But what does learning Japanese means? It means learning a second language, or maybe a third, and maybe a very foreign one at that. So of course they will say it's basically the hardest thing to learn language-wise.
But the goal they imagine is (possibly) different from yours.
Think of it this way: why do you want to learn Japanese? Is it to read manga? Then it's easy. To watch anime? Also easy. Watch J-dramas or read light novels? Harder, but still kinda easy. Or is it to travel to Japan, pass the Japanese Language Proficiency Test I, get a job teaching English in some school, marry some hot asian person, have a family, and spend a few decades talking to people in foreign-ish?
That is hard.
Japanese Language Needs
There are two ways you can communicate in a language: written and spoken. And there are two directions of communication: receiving and transmitting. Thus we can divide the fluency of a language in four parts: reading, writing, listening and speaking.
In order to read manga in Japanese, you only need to know how to read Japanese, and know only a small part of the Japanese vocabulary. Better yet, you can actually suck terribly at reading Japanese and still be able to succeed. This is because you can just look up each and every unknown word in a dictionary all the fucking time.
Now if you were to write Japanese, say, in a chat... then you are screwed. Because you need a good grip of the Japanese grammar. Constructing sentences with proper grammar is much harder than interpreting sentences written by someone else with proper grammar. Not to mention that writing a word you know in English but not in Japanese is also incredibly difficult and the dictionary hardly helps in that case.
Next we have listening. This is easy. Listen enough and you get used to the Japanese fast-paced pronunciation. You are able to tell syllables apart, words apart, and so on. If you are watching anime and you don't know a word, just replay the same bit a few times and look it up in a dictionary. So easy!
Now, speaking, on the other hand... you get the idea.
I should note that, specially because of kanji, we could add reading handwriting and handwriting to the list. Because writing on a computer is very easy and writing on hand is not very easy. Reading handwriting is exceptionally hard in any language, too.
The point I'm trying to make here is: you don't need to learn the whole language, you can study only the part you care about. Only the part you need.
If you consider all the stuff you need to become a full-fledged Japanese speaker, there is a lot of it. There is knowing all of the grammar, all the particles, all the verb conjugations, knowing how to use polite words, knowing when to use polite words, knowing the kansai dialect, knowing a much greater vocabulary, etc. etc. etc.
It is a long road to master the Japanese language. A road you don't necessarily need to take. Because to read manga or watch anime, the amount of skill you need is far less than that. It's a feat a lot more achievable.
Yes, yes, this is incomplete learning. You won't be able to say you can speak Japanese if you can't actually speak Japanese, that is, if all you can really do is read manga in Japanese or watch anime in Japanese. But do you really want to speak Japanese? What are you going to speak Japanese for?
Goal Reached Journey Ended
To give you a concrete example: me. I can read and write English pretty fine. I'm Brazilian, and for almost ten years I have used English to write posts on the internet, to read posts on the internet, to chat with people, to study, little by little I got better at English to the point I'm probably better at writing it than your average American.
However... I can't actually speak English. Worse yet, I can't even listen to English. I can't understand what vloggers say on Youtube, or what the lyrics of a music are like. I need English subtitles to watch English movies, series and cartoons because I can't make out what the actors are saying.
Is it possible for someone to go almost 10 years without figuring out how to listen to English despite using the language almost every day? Well, yes. And the main reason is: I don't need to be able to listen to English, since I can just get the English subtitles instead. If I made an effort, I could learn it, but I'm lazy so I don't.
The same thing applies to Japanese. If you spend 10 years only reading manga in Japanese, you will be a star at reading manga in Japanese... but you won't be able to hold a conversation for one single minute. The question is: is reading the manga all you want? Or do you want to become a perfectly fluent Japanese speaker?
If all you want is to read manga, you can basically get to a good spot after studying for a mere year or two. Not five or ten years. A year or two. Then you can actually stop studying, because by that point you already manage to read manga. If a manga has too many kanji and is hard to read, you can still manage to read it very, very slowly, with the help of a dictionary. So you can basically read any and whatever manga you want at this level.
And the best of all, if you learn manga-reading level of Japanese in your 20's after a year or so of studying and stop there, so long as you keep reading manga and not forget the Japanese you learned, you can still change your mind and become fully fluent when you are in your 30's or 40's or whatever. So you aren't actually ditching away your chances of becoming fluent by half-assing learning the language at first.
Anyway, this is what I wanted to say with this post: it is easier to learn Japanese to read manga than what people make it to be. It's not a task that daunting, that impossible. It doesn't require studying every day for hours for years. Anyone can do it. So if you want to do it, just do it!