Imagine you had a friend named Bob. Bob is a great guy, nice, interesting, and you like having conversations with him. He's also a non-active pedophile. What do you think of Bob? Do you disregard all his good points because of this TERRIBLE thing? Or do you continue associating with Bob because those good points outweigh his disgusting side?
This is what I struggle with when thinking about Kodomo no Jinkan, there is a reason why this manga is controversial (And the controversy is what made me want to read it). However, at the same time, it's actually quite well written, well thought out, and brings up some interesting questions. So join me as I walk you through this... thing. Let's break it down:

~Story~
Okay, before I get into story content, I'll give credit where credit is due: KnJ is pretty decently written. The majority of the time the pacing is very balanced, and the progression of the story makes sense. However, there was one major section that was very emotionally heavy and seemed like it was going to lead into some interesting story developments... until it is just glossed over, as if the mangaka thought to herself "Oh hey, this kinda makes the story hit a wall, let's just pretend it never happened". So it was disappointing to see that story thread kinda just fizzle out. Also the final major 'thing' that happens lacks any sort of foreshadowing, so its insertion felt a bit rushed and awkward. Other than that, yeah, for a manga that is quite sexual towards young girls, good job.
BUT SPEAKING OF SEXUALISING YOUNG GIRLS (And this is a big but) - I am conflicted.
So here's some trivia - The Mangaka Kaworu Watashiya is a woman. The sex of the author never matters to me, but for a series that is quite sexual, I have to make an exception. Was this manga intended to be highly sexualised, or was it its subject matter that made it so? Perhaps it was the choice of the editor to make the manga more sexual? These questions (of which I can't find a clear answer) would give much better context to better judge this manga with. Here is why: First, let me make it clear that I love it when manga and anime address controversial topics - however, how they handle these topics is key. For a manga to criticize the sexual assault of children (And it certainly does), and also have a theme of 'if you love the person, have self control' - but then show overtly sexual images of young girls masturbating, chest fondling and a girl who literally says (though this may be a translation thing) "As long as it had been with you I would have been fine with you raping me" (She's 11 or 12); IT COMES OFF AS INSINCERE.
And this is what I have as an issue with the manga in general - it comes off as hypocritical, at least in this regard.
But enough of that, instead I'll briefly mention the OTHER things it tackles; and for that I give it praise. There are a lot of heavy topics covered in KnJ, sexual assault, child abuse, the confusion around sexuality and growing up, as well as many more. For the most part, it tackles these well - bringing a view that it seems can only be accessed when taken from the experiences of children.

~Art~
It's a pleasant looking 'round and smooth' style of art. Y'know, apart from all that sexualisation of children thing.

~Characters~
It's a mixed bag, being completely honest.Some characters are quite well developed and sympathetic like Kokonoe and her brother. Other characters seem to make it halfway like Mimi and Shirai, but just don't get enough exploration to fully flesh out their characters. The others are like Oyajima, sincere and good for a laugh - but not really explored too deeply.
THEN THERE IS OUR PROTAGONIST. Geez, why does the protagonist have to be the weakest character. He's just a standard sincere, nice, innocent, virgin dude; finding himself pulled by his emotions and his situation. Honestly, by the end of the manga, apart from his sexual preferences I don't know anything about him other than that he has a sister. No idea what drives him, any sort of hint why he'd be attracted to grade schoolers, no deep exploration into his history or what makes him 'him'. He's defined by what happens TO him, not what causes him to react the way he does in these situations. That is not character.

~Enjoyment~
I guess I enjoyed it. I mean, all the sexualisation of young girls (Still can't stress that enough) made me feel a bit uncomfortable - and I can't help that they could've done the same manga with less overt sexual imagery. But I guess they wanted it to sell, and the only way was to... sexualize little girls... *sigh*
But still, yeah, I guess I enjoyed it.

~Overall~
Depending on your answer to the opening question about Bob will pretty much answer how you approach this anime. There is some good stuff here, I won't lie about that, but if you can't shake the slight feeling of distaste due to the subject matter I won't blame you. I will, however, read Watashiya's other works; hopefully they cover just as many controversial and taboo topics - with less preteen chest-baring.

Toodles~