If you are learning the language but wont use it on regular basis I think you wont have any side effects but if you are using both languages regularly like Jess and I then this is true, your native language definitely changes.
I have 2 kids. Their native language is English mine is Spanish. In my house we speak mostly English because my husband doesn't speaks Spanish. My kids will remember the words I choose to teach them in Spanish as first words (mostly because I didn't know how to say them in English lol).
What I noticed is that when they began to go to school and to learn the alphabet, how to write, etc it was easier for them to understand another language. Before they were kind of confused and didn't get the stuff. So as their brain matures their ability to learn, memorize and understand gets better.
I believe there are two aspects of this discussion like if you introduce two languages at the same time when they start to talk it would be easier to them but if they are learning a language as a second language I believe is easier and they grow up.